Experiencing patient-experience surveys: a qualitative study of the accounts of GPs

Abstract

Background

Despite policy and financial support for patient-experience surveys as feedback to modify clinical practice, their benefits and other effects remain unclear.

Aim

This study aimed to examine the perspective of GPs who have engaged with such tools.

Method

Qualitative semi-structured interviews with 30 GPs across south Wales, UK. The transcripts were analysed using a constant comparative method.

Results

The participants regarded patient feedback as highly important but felt that patient-experience surveys, as currently implemented, were hard to interpret. Reflecting on the doctor–patient interaction, participants perceived ambiguity about whether results were evaluating the individual or the GP practice, and whether they were directed towards the ‘practitioner-as-person’ or towards specific behaviours. A potential threat was posed in both interpretations, with the risk of damaging consequences for the practitioner and the nature of general practice. The tension between satisfying patients and performing good health care was a clearly voiced dilemma. Doctors did not feel confident in their ability to change in response to feedback, particularly outside a formal training environment.

Conclusion

Patient evaluation is seen as a key part of multisource feedback for practising doctors. Clarity is needed about the purpose of evaluation – whether for organisational or personal ‘feedback’, and whether formative or summative – and there is a need for attention to the process, with trained facilitators, to ensure that doctors gain from structured reflection, and can minimise the potential negative effects of evaluation.

INTRODUCTION

There is increasing enthusiasm for using the results of ‘patient-experience surveys’ as an indicator of service quality, or a lever for change in health care.¹ Patient-experience surveys have been adapted from the world of consumer surveys and applied to the healthcare context internationally, mostly in primary care settings.² However, closer inspection suggests that a range of constructs – clinical skills such as empathy,³ or patient satisfaction or patient ratings of the ‘experience’ from the consultation – are emphasised by the various available tools,² and it may not be clear which is being assessed.⁴ This suggests that the purpose and utility of patient-experience surveys, or the use of the results as feedback to modify clinical practice, remains unclear. A further source of variation concerns whether tools are intended to evaluate at the level of the individual practitioner or at an organisational level (for example, the general practice).

Much development work on healthcare survey tools derives from North America, although there has also been a steady stream of work in the UK and elsewhere. The tools and the concept of measuring patient experience appear to sit comfortably in the ‘personalised’ healthcare domain,⁵ particularly so in family medicine where the patient-centred method was developed and promoted so that it is now an accepted paradigm across all healthcare disciplines. There seems to be an implicit assumption that using the results of survey tools would lead to an improvement in patient-centredness, although the mechanism is unspecified.

In the UK, after early work on several validated tools, two patient survey instruments, the Improving Practice Questionnaire (IPQ),⁶ and the General Practice Assessment Questionnaire (GPAQ),⁷ were selected and integrated into the Quality and Outcomes Framework of the 2003 General Medical Contract.⁸ Using one of these surveys was required for an individual general practice to achieve its maximum financial incentives in the first few years of the contract (up to 2009),⁸ ensuring feedback at either organisational or individual practitioner levels.^6,7 These instruments have currently been dropped from the required performance indicators, the contract now rewarding practices that achieve high ratings on an externally administered survey of patient experience with access and availability of appointments. This may threaten continuity and good interpersonal care.⁹ However, the use of equivalent patient-experience survey tools is likely to return as part of the multisource feedback that will be a key part of revalidation of doctors for their licence to practice in the UK from 2011. Multisource feedback integrates feedback from different parties; that is, patients, professional colleagues, possibly those responsible for health service provision, such as directors of the primary care organisation, and also self-rating and reflection.

How this fits in

Patient experience surveys have been adapted from the world of consumer surveys and applied to the healthcare context internationally, mostly in primary care settings. However, there are uncertainties about the purpose and utility of patient-experience surveys, whether tools are intended to evaluate at the level of the individual practitioner or at an organisational level (for example, the general practice), and how the results should be used in feedback to modify clinical practice. The participating GPs regarded patient feedback as highly important and monitored it constantly during their daily interactions, but they felt that patient experience surveys, as currently implemented, lacked a clear purpose, and were hard to interpret and respond to in terms of training or changing their consultation behaviours. Clarity is needed about the purpose of evaluation, whether for organisational or personal ‘feedback’ and formative or summative, and there is a need for attention to the process, with trained facilitators, probably as part of the annual appraisal process, to ensure that doctors gain from structured reflection and can minimise the potential negative effects of evaluation.

However the aims, use, and effects of survey results when applied to individual doctors, when used in UK or US to date, remain unclear.² The connection to a summative assessment such as required for revalidation makes confidence in the validity of the tools an essential pre-requisite.¹⁰ Such confidence in the validity of tools, particularly for peer appraisal, cannot be taken for granted.¹¹ Crucially though, while implementation of patient surveys has occurred in many countries, few studies have examined the impacts of such tools on practitioners. The effects of feedback, how it is to be provided and managed, and whether it can achieve its formative development purposes, require evaluation. Existing evidence, although sparse, suggests that the benefits of feedback in a service context are limited.^12–15 Some authors have noted the potential for adverse consequences, and questioned whether the tools may lead doctors to modify their interactions with patients away from the essential holistic elements of care.¹⁶ The variable impact of this sort of ‘feedback’ has been noted as a neglected issue,¹⁷ because of potential unintended negative effects.¹⁸

Given the lack of explicit aims and clear understanding of the effects of patient survey results, this study aimed to examine the perspective of GPs who have engaged with such tools. While located within the UK, the principles underpinning the use of feedback have an international relevance. The aim of this study was to understand how GPs respond to the survey results. It sought to crystallise those features of the process that are regarded as helpful or problematic, which could inform the implementation of survey feedback in its new UK setting as a part of multisource feedback.

METHOD

Participants

Following ethical approval, potential participants were identified and stratified to include GPs from a range of practice sizes (single-handed, small practices, to large group practices with four or more partners) and freelance doctors from publicly available lists of four primary care organisations in the UK (Swansea, Neath-Port Talbot, Bridgend, and Cardiff) and the regional postgraduate departments. They were approached by randomly selecting from these strata of the sample, making initial invitations by telephone, and providing study information ahead of interviews. Doctors' scores in the patient-experience surveys were not known to the researchers.

Data collection

All interviews were conducted by a GP or social scientist, audiorecorded, and transcribed. The interviews were conducted at a location convenient to the doctors. Because relatively little was known about the impact such tools have in practice, a flexible approach that allowed revision and progressive focusing was required,^19,20 that accessed the inductive accounts of participants. While an interview schedule guided initial data collection, prompts were only used to redirect the interview at certain points. Interviews were initiated by a single broad query ‘what are your thoughts about patient surveys?’, followed by secondary prompts to explore issues as they were raised by the participants. Notes were taken during the interview, from which clarification of ‘naturally occurring’ issues was raised and demographic details of the practice noted. When issues related to existing themes, probing questions were used to decipher the circumstances or perceived accomplishments of patient evaluations. All interviews took place from June 2008 to March 2009.

Analysis

Interviews were then transcribed and cleaned, allowing for overlapping speech, inaudible words, pauses, and external noises that impacted upon the interviews.²¹ Following immediate analysis as part of transcription, the constant comparative technique enabled iterative modification of subsequent interviews,^19,20,22 and was supported by qualitative analysis software, ATLAS.ti. (Berlin, 2005). Transcripts were coded line by line and thematically. Independent open coding and comparison led to constant revisions and analytical development. This method allowed for emergent themes to be probed in more depth during subsequent interviews.^19,20,22 The data were further analysed in relation to the emergence of core categories, which were summarised at three key stages (after the ninth, sixteenth, and twenty-third interviews) and then fed back to the research steering committee to develop the analytical frame as the study progressed. Contextual maps were developed to reveal contradictions in and between transcribed interviews.

RESULTS

Forty-three physicians were approached and 30 (70%) were recruited, lack of physicians' time being the most usually cited reason precluding involvement (n = 11). The sample consisted of 21 males and 9 females, with experience ranging from 1 to 34 years in practice. Thirteen participants were based in south-east Wales, five were freelance locums, and the remaining 12 were from different parts of south-west Wales. Ten of the sample held, or previously held, teaching, or appraisal roles. The interviews were mainly conducted within the participants' practice or home (n = 26), lasting between 38 and 87 minutes (mean 51 minutes). The physicians were very frank about their use and views of patient evaluations, and were keen to discuss their experiences. Twenty-five had used the GPAQ survey, three had used the IPQ, and two freelance physicians were unsure which survey had been used. It was found that receptionists dispensed the survey as patients attended, and the results were generally treated collectively, as will be shown.

The emergent themes in relation to the key results that will be presented were:

1. Feedback: a fine idea in principle; and informal concurrent feedback.

2. Contrast between organisational and individual-level patient feedback.

3. Inability to interpret individual feedback: the need for context; problems with numeric scores; and ambiguous focus on the ‘person’ or on ‘performance’.

4. Potential consequences.

5. Tensions between satisfying patients and performing medical care.

6. Context-dependent practitioner response: established practice versus training.

Data will be presented under these themes, with identification of each participant's ID number, sex, and years of experience.

Feedback: a fine idea in principle

Without exception, participants reported that the broad concept of ‘patient feedback’ was a good idea. Reflecting and acting upon patient feedback was seen as good practice and integral to the relationships that GPs have with their patients, the patients' families, and the local community, and not simply during episodic consultations.

Doctors who rejected the idea of feedback were regarded as depressed, alcoholic, burned-out, or just ‘bad doctors’. The positive view of feedback was tied into ideas of a good consultation and held in high regard. The accounts confirm from the participants' perspective that patients and doctors ‘read’ each other throughout the consultation. It is not simply that the doctor interprets patients' accounts, but they are concurrently reading and ‘feeding back’ to each other. The participants suggested that when patients feel at ease with the doctor, this gets to the heart of what it means to be a GP, in an interpersonal encounter:

‘I think when you're consulting it, you, you're your continuously assessing the patient um and listening to different things and doing ah going through the process of the consultation the patient as well is assessing you at the same time and the patient will only tell you what they want to tell you and if they feel at ease and if they feel that they can come out with … if they think the doctor is not listening or is not interested or I don't feel really confident enough I don't feel at ease with this person I'm not gonna go down that very deep problem which is perhaps impotence or something.’ (P16, male, 12 years' experience)

Informal concurrent feedback

Participants generally placed more value on informal concurrent feedback, as part of dialogues in consultations, and viewed this form of interaction as having an immediacy, legitimacy, and context that was crucially seen to be missing from the patient-experience surveys:

‘Well [patient feedback] is an excellent concept but it's how you get it. I mean I get all mine from the patient and from talking to the patient. Maybe I'm getting all the wrong feedback I don't know.’ (P25, male, 18 years' experience)

‘I tend to try and engage the patients and try to get them to feed back informally … because I understand their context a bit better I'm actually able to assess the validity of the concerns a little bit better because I'll say look this guy this is not a guy who normally complains and if he's really dissatisfied then there is obviously an issue.’ (P13, male, 18 years' experience)

Contrast between organisational and individual-level patient feedback

While wholeheartedly supporting the principle of patient feedback and its potential benefits, the participants emphasised the need to differentiate organisational from individual feedback in terms of interpretation and use:

‘I think that patient feedback is actually really important if it was done properly, and I don't think this is the way to get patient feedback on individual doctors, I think it's actually very good for practice feedback … it's not really the way for individual doctors to improve their practice is it?’ (P02, female, 14 years' experience)

Organisational or practice feedback was viewed as understandable and useable, especially when concrete examples of suggested improvements were provided. The participants, however, viewed survey feedback at the individual level as an entirely different phenomenon, one that brought with it significant difficulties of interpretation and use:

‘I think a lot of patients tend to hold their individual [physicians] in quite high regard and a lot of the negative comments seem to come out about the practice. People vent their problems on the practice as a whole rather than on the individual [doctor] … they are happy to say bad things about an organisation, but when it comes to individuals are less so.’ (P06, male, 3 years' experience)

Inability to interpret individual feedback

The overriding concern of participants, emerging as the predominant theme, was their inability to ‘make sense’ of the survey results – how to interpret the ‘feedback’. As each interview developed, the concept of individual feedback became viewed as increasingly problematic in terms of what it meant and how it could be used. On reflection, many participants found it difficult to describe any potential helpful impact of individual feedback. Some became sceptical as their reflections deepened during the interview.

The need for context

The first problem the participants faced is placing the feedback in a context that enables meaningful reflection:

‘… so people can just tick whatever they feel but why they're ticking that we don't know so we don't get the proper background information so it is left for us to think why they've scored that.’ (P10, male, 16 years' experience)

‘… what exactly that means because you don't really have a context in the whole thing … you don't know who was dissatisfied, whether there were personal circumstances for them at that point and that's why they were sort of taking it out on you … I find it a bit frustrating.’ (P13, male, 18 years' experience)

‘… you should never dismiss things, but you should also be able to contextualise and see what weight, what value something has, and why it's being done.’ (P28, male, 26 years' experience)

Problems with numeric scores

Individual comments were seen as potentially useful. However, aggregate anonymous numeric scores led to puzzlement and frustration because of an inability to formulate specific action in response:

‘What does this feedback figure of 73 mean? This is the national benchmark, what is the national benchmark? You find yourself something that is a couple points below it and other points a few points above it and what does that mean?’ (P22, male, 14 years' experience)

‘… feeling of disappointment that you have this information and then have difficulty using it in any way … you know arbitrary numbers are more difficult to translate into what I'm actually doing right or wrong.’ (P11, female, 2 years' experience)

The production of such a numerical score was viewed as crude and a simplistic reduction from a complex range of factors that participants felt contributed to the doctor–patient relationship. The appearance of scientific accuracy was seen as spurious, undermined credibility, and confounded the potential usefulness of the results:

‘… it's very difficult to interpret because if someone's circled a three on something there's a huge patient thought process that goes into that and there are multiple factors which will result in that score, which are not reflected in that number, whereas the next patient comes along and circles the same number for an equal number of different reasons but again you can't see that in the actual score.’ (P11, female, 2 years' experience)

‘I don't think really gosh I got 4.8 this year I must get 4.9 next year because I think life's not like that … there are all sorts of factors come in I mean we all do you know reasonable consultations we do some that go swimmingly well and we do others that we know don't go well and that's life isn't it and I'm not sure that just averaging everything out is really terribly useful.’ (P17, female, 25 years' experience)

‘… after that it's a guessing game which particular bit of that question are you getting wrong or right.’ (P12, male, 12 years' experience)

Feedback above the average was viewed as potential affirmation of practice, which could lead to inaction, complacency, or, more sceptically, was regarded as patients being unrealistically positive. Feedback around the ‘average’ or ‘moderately below’ was not seen as specific enough to be meaningful in terms of affirming behaviours or to motivate action. Sub-average feedback was seen as likely to lead to potentially damaging self-doubt, but without being linked to any actionable specifics; what did I do wrong? What do I do now?

Ambiguous focus on the ‘person’ or on ‘performance’

Many participants felt that the survey results reflected more than an appraisal of a set of competencies displayed in a preceding consultation. The participants felt that many patients were not completing the survey on the basis of the practitioner's performance but were reacting to their general views about continuity and trust over time as well as their feelings about the practitioner as a person. Patients, they said, were giving scores on the basis of what they ‘think of you’ as a person:

‘… as time has gone on you have built a rapport with patients, they gain trust in you and obviously that takes time. I think if you've seen a patient once then they will probably be just specific to that that consultation but obviously I have been here 10 years and most of the patients who do surveys around me have probably known me for 5 or 10 years as well so they will survey me on what they felt about you, think of you over a period of years rather than that day.’ (P20, male, 10 years' experience)

‘… for me the patient is not just that individual. I know their parents, I know their grandparents, I know their brothers their sisters. So they'll come to see me with just a simple cough or cold and they'll say “oh did you know my mum has been in hospital for such and such a length of time” so therefore I would discuss that as well and for me it's not just that individual there's a huge you know surrounding extended family there.’ (P27, female, 13 years' experience)

Potential consequences

Physicians, when describing survey results directed to them as a person, whether positive or negative compared to a peer average, pointed to their significance as a potential threat. It became evident that practitioners took away very little from positive patient evaluations, but identified a propensity for healthy reflection to be too easily transformed into destructive self-doubt:

‘… we had one fairly new partner who felt incredibly threatened [by] the personalised GPAQ information … so I think there's a danger there that you've got to be very careful with that.’ (P17, female, 25 years' experience)

‘… it depends on the individual doctor what their personality is like, they might be pretty horrified and it might knock their confidence, particularly if it's an older doctor they might think “well you know I've been doing this all my life and now they're saying it's a problem, is it time for me to retire?”.’ (P14, male, 1 year's experience)

Interviewer (I): ‘… so if it's just an isolated comment it's easier to kind of dismiss?’

Participant (P): ‘It's easier to live with; I don't know whether you really dismiss it to be quite honest. You think it's sort of a bit cutting and a bit hard, you know it bugs a bit. You go home and you talk about it and you go “hmm someone said this about me today,” and you know you mull it over and you think it about for a while and eventually you say yes it will get dismissed and pushed back because other things take over. But it gets in there there's no doubt about that yeah.’ (P26, male, 14 years' experience)

Tensions between satisfying patients and performing medical care

A sense of conflicting agendas emerged as a theme in most interviews (n = 23), with many responders re-emphasising it as they reflected on the patient surveys. Examples were frequently given where the doctor felt a responsibility to negotiate between competing objectives, such as a reluctance to meet what they regarded as inappropriate requests for sick notes, ‘lifestyle drugs’, or branded versus generic drugs, which was seen to lead to decreased patient satisfaction:

‘Sometimes people come with expectations of getting diazepam … even though we try to explain, some will not accept it, and they can put something adverse in the patient survey.’ (P10, male, 16 years' experience)

Participants commonly voiced a conflict between ‘doing the best thing’ in their eyes versus keeping the patients happy. A heartfelt concern was the fact that it was possible to have very satisfied patients while delivering very low-quality care:

‘… you've got the problem of course that underperforming doctors may be thought the best thing since sliced bread by the patients … they can make the patients like them by doing what they think what seems to the patients to be best, where it might not be best in the long term … rave reviews yes.’ (P19, male, 27 years' experience)

‘… at the end of the day the patient's health is most important not whether your personality is a thing that shines out.’ (P27, female, 13 years' experience)

Context-dependent practitioner response: established practice versus training

Participants were unsure whether the aim of patient-experience surveys was to provide feedback to them so that they would modify their ways of consulting with patients. They did not appear confident in their ability to change even if it were indicated.

Established physicians felt that their interactional styles had been in development over many years and would be difficult to modify unless the feedback was specific and linked to active interventions in a dedicated learning context.

There was a consistent view that feedback about communication skills should be located in learning environments, preferably during formal training, where it was specific and directed to change:

‘Am I empathic as a doctor or do I listen? You think you do in your own mind but it may not come across, but then if it was picked up I'm not sure after 15 years of doing it how you could actually change your style. You can read as many books as you like about consultation theories but you know you're doing 20 a day and it's just the way you are I suppose.’ (P03, male, 13 years' experience)

‘I don't know if being told you don't listen very well or you don't explain things very well I don't think it would have a major change on people … coming back to this idea of changing your manner I was quite surprised how back into the same mould I got into after 5 years out. I found myself sitting down with the same approach, the same conversations, the same explanations. After this gap I am coming out with exactly the same words because you're not quite moulded but it's like a routine and breaking that pattern I think is quite hard especially if you have been doing it for x years. I mean you can add things on and sort of make sort of subtle changes but to sort of completely revamp yourself, no I don't think you can, but whether the subtle changes are enough … ’ (P18, female, 15 years' experience)

‘… it's much easier when you're in a training position you know if your consultation skills for instance are not particularly good, it's very good, easy; when you aren't in a training situation, to make some more videos or get somebody to sit or whatever once you're a partner perhaps it's not quite so easy to do that. You can do it, sure you can make a video of yourself and watch it, but actually that requires quite a lot of motivation.’ (P17, female, 25 years' experience)

Participants considered communication skills to be closely aligned with routines, moulded by workloads, their own personalities, and practice location. There were no objections to further skills training, but there were doubts about how useful such training might be across the range of participants. This doubt was heightened by the belief that communicative styles were inextricably linked to personality, workload, and practice population. Negative feedback was therefore seen as a slight on the individual, rather than on aspects of their communication skill.

Accordingly, the credibility of the patient-experience survey was called into question; it was seen as an ambiguous set of aggregated numbers that lacked clarity, could not legitimise action, and at times was dismissed outright:

I: ‘do you have any view on the purpose or use of individual feedback?’

P: ‘I don't know what the purpose of it is.’ (P02, female, 14 years' experience)

Synthesis

In summary, the participants supported feedback in principle as valuable, but felt that survey results did not meet their tacit construct of effective feedback. A consensus about the characteristics of effective feedback is summarised in Table 1, and contrasted with the views of participants about the results of the surveys.

Table 1.

Characteristics of effective feedback compared to practitioner views about the results of patient experience surveys.

Characteristics of effective feedback17,23–24	Practitioner views on patient experience surveys
Specific clear goals	Confusion over aims
Feedback has a clear purpose	Survey ‘feedback’ reflects competing agendas and purposes
Feedback is meaningful	Difficult to interpret. What do the results mean to me?
Low task complexity	Doctor–patient communication viewed as highly complex. Survey ‘feedback’ viewed as simplistic
Little threat to self	Perceived as threat to self

DISCUSSION

Summary of main findings

The participating GPs regarded patient feedback as highly important, and monitored it constantly during their daily interactions, but they felt that patient-experience surveys, as currently implemented, lacked a clear purpose, and were simplistic and hard to interpret.

Furthermore, organisational and personal ‘feedback’ were regarded as different entities by the participants but were felt to have been conflated in the surveys in use at the time of the study.

The ability of participants to interpret the survey results as helpful individual-level feedback was further limited by a lack of meaningful context for reflection, owing to aggregated scores or anonymous comments, and they were not confident about their ability to modify practice if a need was indicated.

Reflecting on the doctor–patient interaction, the participants perceived an ambiguity about whether results are directed towards the practitioner-as-person or towards specific behaviours. A potential threat is posed in both interpretations.

The tension between satisfying patients and performing good health care was a clearly voiced dilemma marking the boundary between a perceived role as a ‘responsible’ physician providing medical care, and satisfying patients.

A contrast was also drawn between training environments, where ‘feedback’ about observed skills can be focused on rehearsing specific tasks, and established physicians where they regard their communication styles, for better or worse, as relatively fixed.

Strengths and limitations of the study

There was a good response rate from those approached, and the sample included GPs from a range of practice types, seniority, and positions of leadership, and all had received feedback from patient-experience surveys as part of their ‘normal practice’. The interviewers and primary analysts had clinical and social science experience, and were complemented by wider discussion with the project-management team. The method enabled development of the later interviews to explore concerns, issues, and themes arising from the analysis of early interviews. The study is limited to the physician's perspective and was not complemented by exploration of the patients' views as recorded in the completed surveys. Furthermore, the doctors' scores in the surveys were not known to the researchers, so it is not known whether the sample included a range of high to low scorers, which may affect their reflections and evaluation of the process. The study lacked an ethnographic perspective that would interrogate how the surveys are enacted and implemented in practice. The research itself was only conducted in South Wales (UK), yet the results have implications for how such tools are taken up elsewhere. Finally, the study primarily examined the impact of the GPAQ in the UK,⁷ as this was the most common tool used by practitioners, and other tools in other settings may have different implications for practice.

Comparison with existing literature

The participating doctors supported feedback in principle, as has been identified elsewhere, and it has been noted that doctors may be more responsive to feedback from patients than their colleagues.¹⁸ However, doctors may not be inclined to use feedback for improvement,²⁵ with particularly few changes being made in response to positive feedback.¹⁸ This study revealed difficulties in the interpretation of survey results because of the perceived low credibility and simplicity of numeric scores. Similar findings have been reported in relation to the use of multisource feedback for physicians, where factors such as low credibility and the importance of opportunities for constructive reflection affect whether feedback has positive impacts.²⁶ Here, the doctors identified more salient and different forms of feedback, particularly from relationships with patients. Others have noted the problem that patient views about the ‘doctor-as-a-person’ are based on a network of relationships with families, often spanning years and generations, rather than their performance on any given day, and that doctors and patients are concurrently reading and ‘feeding back’ to each other as part of performing the consultation.²⁷ Consistent with Kluger and DeNisi's feedback intervention theory,¹⁷ most physicians view feedback as being aligned to their self-concept and professional persona. This brings with it potentially destructive effects of feedback if it is felt to be poorly focused, or directed to individual doctors, where it might be dismissed, or, worse, may lead to reduced performance.²⁸ The potential for negative outcomes of feedback was prominent in the present study data. This can be in terms of ruminating on one's ‘self-concept’,²⁶ with potentially strong and long-lasting negative effects.²⁹ It may bring with it influences on doctors to focus more on the ‘tasks’ of consultations rather than interpersonal care and continuity.⁹

There are views from educationalists, GP registrars, and trainers that assessing patient experience may be among the potentially most valuable ways of addressing ‘relationships with patients’ (alongside colleague feedback).^30,31 Indeed, these may be more objective views, and potentially more able to bring about change and improvement than the feedback received during consultations that may reinforce doctors' self-concepts and professional persona (as above). The data here, though, would suggest that the current methods need improvement. The implementation of an assessment system (whether formative or summative) may be facilitated by stakeholder agreement that appropriate qualities are being assessed.^30,32 There needs to be agreement that the tools themselves are reliable, but the literature shows that this is contested.^4,33–35 Burford et al identified stronger support for credibility in text-oriented tools, for feedback on communication and attitude and to identify doctors in difficulty.³⁶

Greater attention is also needed to the ‘consequential validity’ of patient feedback – the potential negative effects noted above – and how to minimise them.^18,32 Reflection needs to be facilitated if it is to be effective and positive in this domain characterised by analysis of ‘self-concept’.^23,26 Negative comments can be especially potent in stimulating change, but a facilitator is necessary to ensure these changes are creative and not destructive. The facilitator can guide an individual's reflective process, help to assimilate reactions and emotions, interpret, and identify learning and performance needs.²⁶

A further area for debate concerns the balance – and potential trade-off – between patient satisfaction and ‘effective practice’. In short, this concerns a tension described by these participants between patients' wants and needs. When evaluative tools for patient experience are used to lead practice, akin to consumers as surrogate managers,^37–38 there is potential for patient wants to be reproduced in practice over and above the doctor's clinical definition of patient need. This is an issue that has not been adequately addressed in the literature relating to patient-evaluation tools previously.^{6,18,29,39–40}

Implications for clinical practice and future research

While there are a few cautionary voices surrounding patient service evaluations,^15,17,23 the appeal to policy-makers grows. The patient-experience survey is an integral part of the multisource feedback now piloted by the General Medical Council. The potential pitfalls need to be appreciated, and steps taken to minimise them. Without this there is a concern not only that family physicians are under increasing scrutiny, but perhaps that the scrutineers by proxy (the patient) will have a knock-on effect on the longer-term relationships that patients have with their family physician.

While newer tools have been validated,⁴¹ care is still needed about matters of process that may affect the outcomes. There needs to be clarity about whether the tool is assessing the individual or the organisation within which they work. Some measures may be more designed for the individual,⁶ and others may be more designed to evaluate organisations.^7,42 The latter may also be divided into those that assess patients' experiences of the healthcare encounter itself, and those that collect data on organisational aspects of care, such as ease of accessing appointments, or obtaining test results.⁴³ Whichever emphasis was intended for each instrument, and with apparently clear instructions to the survey responder, it also appears from the accounts of these doctors receiving the feedback that they felt the patients' evaluations were unclear or inconsistent in terms of whether they related to individual or organisational performance. There needs to be even greater clarity as these tools are used in practice.

Previous research on patient-evaluation surveys indicates that the potential negative effects they can have on practitioners may be downplayed.^18,29 Sargeant et al noted a participant who had been reflecting upon negative evaluations for 2 years;¹⁹ this could be seen as unhelpful for family practice as well as the individual concerned. To guard against this, patient-experience evaluations should be used primarily for development (formative) purposes, and not for performance management (summative assessments).⁴⁴ A continuing debate is required about the balance between summative or formative approaches, and how this will integrate with multisource feedback for revalidation.

The view of this study, derived from the accounts of these doctors, is that patient-experience feedback is more likely to achieve its purpose of improving the patient experience if it is used only in formative appraisal contexts, and is removed from the process and requirements for revalidation. Implementation in this way will allow doctors to be more challenged about their views on the value of feedback data, and to promote receptivity to making changes.

When these doctors were interviewed, the patient-experience survey existed (in the Quality and Outcomes Framework up to 2009) without a formal process for formative feedback. Even as it resurfaces as part of the multisource feedback for revalidation in 2011, greater attention is required to the process of feedback. The General Medical Council recommends trained feedback and a feedback session.⁴⁵ This is not currently achieved by the ‘senior medical colleague’ who a doctor names for voluntary discussion about their feedback.⁴² It may be more consistently achieved if done through the annual appraisal processes in which GPs participate. The British Medical Association recommends training for appraisers and appraisees about how to make the most of the process, and that this should be operational before the feedback process is implemented.⁴⁴

The means through which doctors use and interpret patient surveys remain poorly understood. What patients are actually writing about, who they are writing to, and how much time they spend thinking through the boxes that are ticked are not known by doctors or researchers. A more in-depth ethnographic study of how feedback is used and understood in practice is necessary, in particular including a range of doctors with higher and lower scores on patient surveys. This would be crucial to enhance understanding of such tools and the consequences they have for patients and physicians on one level, and the means through which a consumer tool influences practice on another.

In summary, patient evaluation is seen as a key part of multisource feedback for practising doctors. The participating GPs regarded patient feedback as highly important, and monitored it constantly during their daily interactions, but they felt that patient-experience surveys, as currently implemented, lacked a clear purpose, and were hard to interpret and respond to in terms of training or changing their consultation behaviours. Clarity is needed about the purpose of evaluation – whether for organisational or personal ‘feedback’, and whether formative or summative – and there is a need for attention to the process, with trained facilitators, probably as part of the annual appraisal process, to ensure that doctors gain from structured reflection, and can minimise the potential negative effects of evaluation.

Funding body

The project was funded by the Wales Office for Research and Development, Welsh Assembly Government, project grant number F06/2/204.

Ethical approval

The study was approved by South West Wales REC 07/WMW02/8.

Competing interests

The authors have stated that there are none.

Discuss this article

Contribute and read comments about this article on the Discussion Forum: http://www.rcgp.org.uk/bjgp-discuss