‘Give me something meaningful’: GPs perspectives on how to improve an audit and feedback report provided by health insurers – an exploratory qualitative study

Abstract

Background

Audit and feedback (A&F) is a valuable quality improvement strategy, which can contribute to de-implementation of low-value care. In the Netherlands, all health insurers collaboratively provide A&F to general practitioners (GPs), the ‘Primary Care Practice Report’ (PCPR). Unfortunately, the use of this report by GPs is limited. This study examined the thoughts of GPs on the usability of the PCPR and GPs recommendations for improving the PCPR.

Method

We used an interpretative qualitative design, with think-aloud tasks to uncover thoughts of GPs on the usability of the PCPR and semistructured interview questions to ask GPs’ recommendations for improvement of the PCPR. Interviews were audiorecorded and transcribed ad verbatim. Data were analysed using thematic content analysis.

Results

We identified two main themes: ‘poor usability of the PCPR’, and ‘minimal motivation to change based on the PCPR’. The GPs found the usability of the PCPR poor due to the feedback not being clinically meaningful, the data not being recent, individual and reliable, the performance comparators offer insufficient guidance to assess clinical performance, the results are not discussed with peers and the definitions and visuals are unclear. The GPs recommended improving these issues. The GPs motivation to change based on the PCPR was minimal.

Conclusions

The GPs evaluated the PCPR as poorly usable and were minimally motivated to change. The PCPR seems developed from the perspective of the reports’ commissioners, health insurers, and does not meet known criteria for effective A&F design and user-centred design. Importantly, the GPs did state that well-designed feedback could contribute to their motivation to improve clinical performance.

Furthermore, the GPs stated that they receive a multitude of A&F reports, which they hardly use. Thus, we see a need for policy makers to invest in less, but more usable A&F reports.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Audit and feedback (A&F) is an often-used quality improvement strategy, with varying effectiveness.

WHAT THIS STUDY ADDS

• This study strengthens findings on how to design A&F that professionals deem usable and add the perspective of general practitioners on claims data as the source for A&F.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• A call for action to make use of existing knowledge on A&F and user-centred design and invest in less, but better A&F.

Background

Audit and feedback (A&F) is one of the most frequently used quality improvement strategies across healthcare settings, including primary care.¹ A&F is defined as ‘a summary of clinical performance over a specific period of time, and the provision of that summary to individual practitioners, teams or healthcare organisations’.¹ Theory behind A&F is that, by providing clinicians feedback on clinical performance, they are stimulated to change their clinical behaviour and thereby improve clinical practice and patient outcomes.^2–4 While numerous trials show A&F is effective, the effectiveness of A&F interventions varies widely.¹

There remains uncertainty regarding how to design A&F in such a way it is maximally effective in contributing to quality improvement.^{5 6} Research shows that the effectiveness of A&F depends on the design of the intervention itself, recipient characteristics, and the context in which the intervention is performed.^{1 7–9} Regarding the design of the intervention evidence shows that A&F is most effective when the ‘frequency of feedback is more than once, the format is both written and verbal, feedback is given by a respected peer or colleague and feedback is accompanied by specific goals and an action plan’.^{1 10} Furthermore, A&F may be more effective if it is ‘actionable’, meaning the feedback is timely, individual and non-punitive.^11–13 Also, research suggests that performance comparators in A&F ideally include trends and specific targets and are compared with similar peers.¹⁴

Quality improvement strategies are necessary in many Western countries, as they struggle with the provision of low-value care (LVC), that is, care is not effective or not cost-effective.^{15 16} The provision of LVC reduces the overall quality of care, potentially causes patient harm and is ineffective use of scarce healthcare resources. In primary care, LVC entails prescribing of ineffective medications and unnecessary referrals, diagnostics and interventions.^17–19 A&F has been shown to be effective in de-implementing LVC in primary care.^20–22

In the Netherlands, health insurers collectively commission the development of an A&F report, the ‘Primary Care Practice Report’ (PCPR)^[1, to stimulate general practitioners (GPs) to improve their clinical behaviour. The PCPR presents benchmark information on consultations, diagnostics, interventions and prescriptions in primary care; and numbers of patients using medical specialist care and mental care. Unfortunately, the use of the PCPR by Dutch GPs is limited: only 60% of the GPs have ever downloaded the PCPR between 2015 and 2019 and repeated use of the PCPR is not customary.² While literature gives some theoretical guidance regarding how to design A&F, large uncertainty remains on how to design A&F. The limited use of the PCPR clearly illustrates a gap between theory and how an A&F report is designed in practice. Therefore, we examined the thoughts of GPs on the usability of the PCPR and second, we examined GPs recommendations for improving the PCPR.

Method

For reporting of this study, we followed the COnsolidated criteria for REporting Qualitative research, the COREQ-checklist can be found in online supplemental file A.²³

Design

We used an interpretative qualitative design to explore the perspectives of GPs on the usability of the PCPR.

We used think-aloud tasks to examine the thoughts of GPs on the usability of the PCPR and we used a semi-structured interview to ask GPs for their recommendations on how to improve the usability of the PCPR (online supplemental file B).

In think-aloud methodology, participants are asked to perform a task and verbalise any occurring thoughts while doing so.^{24 25} This methodology is valuable to get a complete and realistic view of users’ perspective on tools, such as the PCPR.²⁶ In each interview, we showed GPs a representative selection of tables and graphs (online supplemental file C) from the PCPR and asked the participating GPs to ‘think aloud’ while interpreting these tables and graphs.

Patient and public involvement

Due to the nature of the study, patients and public were not involved in the design of this study.

The PCPR

The PCPR is a nationwide benchmark tool for GPs developed since 2015 by VEKTIS in collaboration with health insurers and GPs. VEKTIS is a business intelligence centre that is established and financed by the Dutch Association of Health insurers,³ in which all Dutch health insurers are represented.

The PCPR provides GPs insight in the care use of their patients and aims to contribute to GPs' insights in the quality and cost-effectiveness of the care they provide. The PCPR is based on claims data, received by VEKTIS from all Dutch health insurers on insured citizens which covers about 99% of the Dutch population. The feedback includes costs and/or volume indicators on seven main topics: (1) the practice patient population, which includes figures on the volume of the patient population, presented by age, gender and socioeconomic status (SES); (2) the practice total care costs, (3) the GP care provided, which includes cost and volume indicators on consultations, visits and minor surgical procedures performed in primary care; (4) costs of prescribed drugs; (5) referrals to mental healthcare; (6) referrals to medical specialist care and (7) primary care diagnostics. Excluding general descriptive information, the PCPR consists of 24 pages of feedback, with 86 figures and 22 tables.

Feedback is presented for the total GP practice, thus if multiple GPs work in one practice the feedback is on group level. Only one figure in the PCPR presents feedback on individual level, this is for the number of diagnostic requests per year. Furthermore, the feedback in the PCPR is given for a specific year, about 1.5 years prior to the release of the report, the PCPR contains some figures with historical trends, and the performance results of a specific practice are in many figures compared to an ‘expected value’. This expected value is based on the national average for a practice with a similar patient population which is case-mix corrected for age, gender, income, pharmaceutical use, and history of Chronic Obstructive Pulmonary Disease (COPD), diabetes, and cardiovascular diseases. This expected value is given as a benchmark to enable GPs to compare their own clinical behaviour. The PCPR is updated once or twice annually, after which GPs are invited via email to freely download the report from the VEKTIS website. Since 2015, seven versions of the PCPR were made available for download.

Participant recruitment strategy

We purposively recruited GPs that were diverse in respondent characteristics, such as gender, type of practice (health centre, group practice and duo practice) and geographical distribution of the Netherlands. We received contact details of 80 GPs from the medical faculty and via our personal network. GPs related to the medical faculty offer internships to medical students from the ‘Vrije Universiteit, in Amsterdam, the Netherlands’ and represent a diverse group that work in varying regions of the Netherlands (widely dispersed in the North and Middle region of the Netherlands) and work in various settings (solo practices, duo practices, group practices and health centres). Besides one GP, of whom the interviewer was enlisted in the GP practice, none of the contacted GPs were directly related to the personal network of the authors. Of the 80 GPs that were invited by email or phone for participation in this study, 32 responded to the invitation of which 18 GPs were interested in participation. Fourteen GPs declined; reasons for decline were high work pressure, retirement and acting as a locum. The study aimed to enrol 10–20 participants. This sample size was based on existing think-aloud studies, the likelihood of drop-out and the expected number required to meet data saturation and congruency.²⁷ Data saturation was reached after 10 interviews, a further two interviews were conducted to increase the rigour of the results. GPs participated voluntarily in the study and did not receive any kind of compensation.

Data collection

All interviews were conducted by a master student (ABvdK, male), with no clinical affiliation. The interviewer knew one GP prior to the interview, as he was enrolled at this GPs practice. He had no personal relationship with this GP or relation to any of the other GPs.

Interviews were held from May to July 2018 in the GP practice of the participant. ABvdK was supervised by two of the authors (PJGMdB, MD), with no clinical affiliations or existing relationships with the participants. All participants provided written informed consent.

Prior to the think-aloud, we asked GPs for descriptive information regarding whether and which types of A&F they used. Then, we used think-aloud tasks to examine the thoughts of GPs on the usability of the PCPR. At the start of each think-aloud session the interviewer briefly explained think-aloud as a method to uncover participants thoughts. The interviewer explained to GPs they would be shown different pages of the PCPR and asked them to reflect on anything they saw or thought when reviewing each page. We asked the GPs to perform the think-aloud task six times, for six different pages of the PCPR, covering 18 out of 86 graphs and 3 out of 22 tables. Since the PCPR does not require an active task but an interpretation and reflection on clinical performance, we used guiding questions to stimulate GPs to think-aloud. Thus, for each page of the PCPR shown, the GPs were asked to answer three subsequent questions (1) What do you see and do you understand what you see? (2) What are your thoughts on what you see? and (3) How does what you see influence you and/or your practice? The six pages for the think-aloud tasks were selected in collaboration with VEKTIS based on diversity in topic of the PCPR, type of healthcare presented and design of graphs. We selected pages that are representative of the total PCPR. The selection included three out of seven topics in the PCPR, namely: (1) patient population: showing the volume of the patient population, presented by age, gender and SES, (2) general practice care: showing volumes of use of care, volumes of various types of consultations and visits, and volumes of minor surgical procedures performed in the GPs practice and (3) general practice diagnostics: showing costs and volumes of diagnostic requests. To assess accuracy of the interpretation of the participant, two researchers discussed the transcripts of the think-aloud tasks and compared results to the legend of the PCPR.

Second, we asked the GPs semistructured interview questions to share their recommendations on how to improve the usability of the PCPR. Lastly, participants were given the opportunity to make additional statements or ask questions.

After the first interview, the wording of some questions was altered to more concrete and accurate wording.

All interviews were audiorecorded. Field notes were made during and immediately after the interviews to aid analysis. We sent a summary of each interview to the respective participant for a member check. None of the participants responded. Interviews lasted 30–55 min. No repeat interviews were carried out.

Data analysis

The interview recordings were transcribed ad verbatim. We used thematic content analysis to analyse the interviews.²⁷ The analysis was done in two phases. First, a deductive analysis was performed in the context of a master thesis internship, for which two coders independently coded a transcript after each interview was held. The resulting code book was discussed within the wider research team, but since this scheme closely reflected the topic guide it did not fully answer the research questions. To gain deeper understanding in the interviews the research team decided to recode the transcripts inductively, for which initially one reviewer recoded all transcripts. The resulting coding scheme of the inductive approach was briefly discussed in the wider research team to assess the added value of inductive coding. After the research team decided inductive coding was valuable, another researcher (who previously coded deductively) recoded all manuscripts inductively. Subsequently, the two coders discussed the codes per transcript, to ensure consistency in coding. Lastly, the coding scheme was discussed within a wider research team to identify recurrent themes and formulate the final themes. In the themes, we grouped results from the think-aloud tasks and recommendations, since these two approaches identified similar themes. Think-aloud identified negatively framed items, while recommendations identified similar items framed positively.

All coding was performed using qualitative computer analysis software package, Atlas.ti (V.9.1.3) software. The quotes that best illustrated the different themes and subthemes were selected and translated from Dutch to English. All quotes were then evaluated by the remaining authors and changes were made to reach agreement among all authors.

Results

Participant characteristics

We interviewed 12 GPs, from 10 different practices, aged 35–64 years. The respondents varied in years of work experience, active years in current practice and digital experience, type of practice and work setting (table 1). All GPs were familiar with A&F reports in some form, although most GPs did not regularly and systematically used A&F to review their own performance. Respondents receive many types of A&F reports, such as reports from their own GP information system, accreditation programmes, a recognised health services research institute, health groups or regional networks, individual health insurers, pharmacotherapy networks and diagnostic laboratories. Eight GPs (67%) had heard of the PCPR before the invitation, but only three GPs (25%) had tried to use the PCPR for quality improvement purposes.

Table 1.

Respondent characteristics

GP	Sex	Age	Years of clinical experience	Years active in current practice	Type of practice	Work setting
R1	F	52	22	21	Health centre	Urban
R2	M	59	23	22	Group	Urban
R3	F	61	34	32	Health centre	Urban
R4	F	64	36	23	Group	Urban
R5*	M	59	28	27	Group	Rural
R6*	F	59	29	28	Duo	Rural
R7	F	50	17	17	Duo	Urban
R8	M	53	19	19	Duo	Urban
R9	M	62	30	10	Group	Urban
R10	F	35	6	6	Group	Urban
R11†	F	46	16	12	Duo	Rural
R12†	F	40	12	7	Duo	Rural

Type of practice=health centre (centre in which multiple types of healthcare providers reside, such as a physiotherapist, pharmacy, midwife, social worker, dietician, psychologist), group practice (GP practice owned by >2 GPs) or duo practice (practice jointly owned by two GPs).

Work setting=urban (GP practice in a city); rural (GP practice outside a city).

*R5 and R6 worked in the same practice.

†R11 and R12 worked in the same practice.

GP, general practitioner.

Themes

We identified two main themes from the interviews: ‘poor usability of the PCPR’, and ‘minimal motivation to change based on the PCPR’. In the theme ‘poor usability of the PCPR’, we identified the subthemes ‘desire for clinically meaningful feedback’; ‘limitations of claims data for audit’; ‘performance comparators are difficult to interpret’; ‘desire to discuss results with peers’ and ‘unclear definitions and figures’ (online supplemental file D). The second theme did not have subthemes. Recommendations for improvement of the PCPR, if provided from the interviews, are described per theme.

Poor usability of the PCPR

Desire for clinically meaningful feedback

The GPs stated that they did not find the feedback clinically meaningful, since feedback on quality of care is lacking, cost indicators reflect prices which GPs cannot influence and detail on conditions is lacking. The GPs did not see how the feedback shown coud help them to improve their clinical practice.

R7: "It is unclear to me how these figures might help the patient sitting at the other side of my desk."

R11: “We strive for high quality healthcare, but feedback usually reflects euros or percentages, instead of health gains. For how many people suffering from high blood pressure did my prescription result in a lower blood pressure? And how many of my patients have not had a stroke as a consequence? How do I relate to peers?”

The GPs found the cost indicators uninteresting and frustrating, since these include variation in prices or costs made in secondary care, which both cannot be influenced by the GP. Also, GPs found the feedback on high level, in which all types of conditions are combined, not clinically meaningful.

GPs recommended including quality indicators on either clinical outcomes regarding their treatment role, such as the blood pressure of their patients, or indicators on the effectiveness of their gatekeeper role, such as whether longer GP consultations lead to less referrals to secondary care. Furthermore, GPs recommended excluding feedback outside their control, such as costs influenced by prices. Lastly, GPs recommended including feedback with lower detail level, such as for common disorders.

R1: "I would like to see more figures on my referrals for separate conditions. And this information is mostly cost-related, so price increases have an effect. However, I can’t control those costs."

Limitations of claims data as a source for A&F

The GPs found that the claims data on which the PCPR is based has several important limitations. First, half of the GPs indicated that the timeliness of the data was insufficient, with data dating back to 2 years prior to the report release date.

R11: “Drawback is that it is of course much too late, in the sense that I’m looking at numbers of 2016 (in 2018). Thus, it is not exactly management information. I can only see that I am at the bottom of the canyon, because I missed an exit in 2016”

R4: "Anyway this feedback is of over two years ago, I find that difficult. The longer ago, the more difficult it is to apply in practice. I would appreciate feedback on my figures from last year somewhere within the first three to four months of the current year. That would really help me."

Second, the GPs stated that the group level feedback makes it unclear whether deviations in performance are attributable to themselves or to a colleague. Even for the only figure with individual feedback, GPs stated that this may not be attributable to themselves, since diagnostic requests have been made in their name by physician assistants, residents GPs and part-time GPs.

Third, the GPs questioned the reliability and validity of the data.

R1: "Then my thoughts are: is it correct? Is it correct for our population?"

R5: "This seems to be untrue; I do not know how the benchmark is developed; this difference is too big. So, I have my reservations."

Furthermore, two GPs expressed a lack of trust in the source of the PCPR, they were sceptical about the role of health insurance companies as the supplier of data.

GPs recommended using recent, individual, reliable, and valid data as the source for feedback.

Performance comparators offer insufficient guidance to assess clinical performance

GPs found that the historical trends, the ‘expected value’ as the benchmark and the lack of targets in the PCPR offered insufficient guidance to assess their clinical performance.

Futhermore, GPs noticed unstable trends in their performance, with high outliers in one year and low outliers in the next. This caused confusion and GPs did not know how to interpret this or how to act on these unstable trends. Also, GPs indicated that policy changes in healthcare organisation or claim codes can make data incomparable over time, hindering interpretation of historical trends.

R5: "If one year I do too much and the other year I am below expected, then yeah, I think yeah, I really won’t do my work differently."

Last, GPs found it difficult to compare their clinical performance to the ‘expected value’, since this value is derived from national data and thus includes incomparable practices. Eight of the 12 GPs attributed deviations in their clinical performance to casemix differences, while the figures were already case-mix corrected. In addition, several GPs missed specific targets in the PCPR.

R2: "Are a hundred colleagues providing too many treatments or am I providing too few? Hmmm, who knows?"

GPs recommended comparing performance to similar peers, rather than national data. GPs would like to compare to peers within their practice as well as peers from other practices. GPs found it important that the comparator is similar in setting (rural versus urban) and type of care services (whether an ultrasound is available, a Doppler, etc). Also, GPs recommended including regional comparisons and to account for integrated networks. Furthermore, GPs recommended including an advice or an action plan in the feedback, which helps them assess their clinical performance.

Desire to discuss results with peers

Based on the feedback, multiple GPs expressed they would like to discuss this feedback with peers from their own practice or peers from other practices. GPs wanted to discuss the causes of deviations in audit results with peers, to gain insights in other ways of organising and delivering care and thus on if and how to change their clinical behaviour.

Unclear definitions and figures

Every GP found one or more tables/graphs in the PCPR hard to interpret. The GPs stated that several definitions were unclear, such as ‘primary care diagnostics’, ‘request for diagnostics’. GPs did not know how these indicators were defined and what classified under such categories.

R12: "I also don’t really know what primary care diagnostics entails, is that only blood tests or does it include more?"

R11: "I don’t even know about what kind of diagnostics this is now: special diagnostics, scopies, no clue what this is about.”

GPs recommended including more explanations on definitions and case-mix correction in the tables/graphs instead of only in the preface of the report. Furthermore, GPs found there was an unclear presentation of numbers with varying scales in similar graphs, unclear units and indistinctive colours.

Minimal motivation for change based on the PCPR

Most GPs found that the PCPR did not provide the correct information to change, due to the content of the feedback, the type of audit data that was used or the lack of guidance of performance comparators.

R5: "If you are higher than expected at one year and lower at the year after, then I think: ‘the average seems good, there’s no reason to work differently’. If I have a consultation with a patient who is coughing for half a year, I do not think ‘oh, I’ve applied too much photos so far, let’s not refer this patient.’ You still look at the patient and his needs."

There were only a few cases in which GPs stated they were motivated to change their behaviour based on the PCPR. They were either motivated to change clinical behaviour, such as increasing intensive visits and increasing Dopplers, or motivated to change claims behaviour.

R4: "This number strikes me. I think this is due to under registration on our part."

R11: "We perform less 24hours blood pressure measurements than average. That is good to know, because then we will perform those a bit more, because they earn well. For Dopplers the same. Those we can nicely increase a bit."

R6: "We should look at our administrative process. Probably we sometimes forget to submit declarations. That is no medical difference, but purely administrative."

Interestingly, most small deviations of performance did not give GPs motivation to change, while large deviations resulted in GPs questioning the data thus also did not evoke motivation to change. Also, many deviations in performance were explained by GPs as conscious practice management.

R1: "ECG we indeed don’t perform often, we only do this for rhythmic disorders, and we do that on purpose, we have them done in the hospitals because we believe our quality in assessing ECGs is inadequate."

R4: "Well, as I said we perform less phone consultations, we agreed not to perform triage, that is conscious policy and we are very happy with that."

Importantly, the majority of the GPs stated that well designed A&F could potentially motivate them to change behaviour and thus have a meaningful impact on quality of care. Two GPs stated that A&F in general could not change their behaviour. Both GPs stated that this was because aggregated data could not influence how they provide care for an individual patient, one GP states this was because he applies critical thinking continuously and that he would find it difficult to weigh aggregated data more than his own critical thinking and the other that he provides care one patient at a time and that you do want your actions to be ‘medically responsible’, thus that aggregated data does not influence patient care.

Discussion

This study examined the thoughts of GPs on the usability of the PCPR and GPs recommendations for improving the PCPR. GPs found the usability of the PCPR poor due to (1) the feedback not being clinically meaningful; (2) the audit data not being recent, individual and reliable; (3) the performance comparators offered insufficient guidance to assess clinical performance; and (4) the results are not discussed with peers and the definitions and visuals are unclear. GPs recommend improving these issues. The poor usability of the report led to minimal motivation to change based on the PCPR.

Our results are consistent with previous literature on effective A&F and user-centred design; and indicate that policy makers still do not draw sufficiently from existing evidence when developing A&F reports.^{13 28–31} The PCPR contains information on process and cost indicators aggregated across all types of conditions, based on the healthcare procurement perspective of health insurers. However, this does not match the clinical perspective of the GPs. In our study GPs requested information on clinical outcomes regarding their treatment role and effectiveness of their gatekeeper role, which can be developed based on claims data. Also, the PCPR could present feedback for common disorders and common interventions specifically instead of aggregated data. Aligning feedback with recipient priorities is considered key to improve the effectiveness of A&F.³² Furthermore, the PCPR does not specifically focus on LVC and lacks targets, which makes it difficult for GPs to interpret whether and how they can improve their quality of care. Since claims are not identifiable to one GP and claim processing has a lag-time, it is difficult to meet requirements of recent and individual A&F based on claims data.

Our study also adds to the knowledge on how to design effective A&F. First, GPs specifically asked for feedback on clinical outcomes and the effectiveness of their gatekeeper role, such as whether more long consultations prevent referrals or whether more email consultations in primary care prevent face-to-face consultations in primary care. It is interesting that GPs in our study requested clinical outcomes as feedback, even though outcomes are not a direct reflection of clinical performance.¹¹ GPs in our study stated that outcome indicators, such as irregulated blood pressures, are important indicators for them to review their clinical behaviour, such as whether they should change their prescriptions or referral behaviour. Second, although using trends as a performance comparator is recommended since trends can create positive motivation towards change, we found that unstable trends could cause confusion in interpreting the feedback.¹⁴ To help clinicians interpret trends we advise to include descriptions of relevant policy changes or possible changes of claims code definitions in the caption of feedback. Third, we found that using claims data as the source for A&F in some cases motivated GPs to change claims behaviour instead of clinical behaviour. Since it is the aim of A&F to stimulate reflection on clinical behaviour, it is questionable whether using claims data as the source for feedback distracts from this purpose and whether patient record data or medical registries are better suited data sources. However, it is unknown whether the type of data source used for A&F influences the effectiveness of A&F interventions.³³ Lastly, some GPs were sceptical about the role of health insurance companies as the supplier of the data. Health insurers in the Netherlands have the legal task of guarding affordability, accessibility, and efficiency of healthcare. Therefore, professionals express a fear that health insurers will use audit data to reduce healthcare reimbursements. Using A&F in a punitive manner might lower professionals’ motivation to change and thus result in less effective A&F interventions.¹¹

A comparable qualitative study, which examined how the PCPR and group discussions contribute to GPs motivation to change, found many similar results, such as that feedback should be reliable, recent, individual and concern the own influence sphere.³⁴ Furthermore, this study found that group discussions on the PCPR contributed to GPs motivation to change. It could be that not the quality of the feedback itself, but feedback combined with peer discussion is a vital strategy in A&F effectiveness.^{34 35} However, studies examining the effectiveness of peer-discussion in A&F interventions and quality improvement show varying results.^{1 36–38}

Theoretical underpinning of using A&F for quality improvement purposes is that A&F motivates professionals to change.^{2 4} In our study, most GPs stated they were a priori motivated to change to improve quality and that A&F could contribute to their motivation to change. However, the GPs stated that they find the quality of the PCPR insufficient, thus it did not motivate them to change. In A&F literature lack of quality of feedback is frequently mentioned as a barrier for motivation to change.³⁹ It is possible that lack of quality is mentioned as a barrier for change, because of cognitive dissonance or external attribution, in which professionals search for an external explanation for underperformance.^{14 40} Experts state that the optimal design of A&F likely depends on recipient factors and context, but it is yet unclear which thresholds exists for quality of feedback to be ‘sufficient’ to contribute to motivation to change⁶ Furthermore, Desveaux et al³⁵ describe that ‘A&F may not be translated into actual change, due to incapability of participants to interpret A&F in an actionable way’. Thus, even when the quality of the feedback itself is sufficient, recipients may still need education on how to interpret and translate A&F into change.⁴¹

Multiple studies throughout the years describe that GPs find A&F reports unusable, while these same studies do find that GPs are motivated to use A&F for quality improvement.^41–43 In addition in our study, GPs state that they receive a multitude of A&F reports, many of which they hardly use. Using low-quality A&F creates waste of resources and leads to ineffective use of clinician’s time. While the PCPR has important strengths, such as support from national policy makers, national coverage, it covers all healthcare provided and it is based on routinely collected data, not all concerns of GPs regarding usability of the PCPR can be solved, such as the timeliness of the data or individual feedback. Thus, it might be more effective to invest in another type of A&F for quality improvement. For more effective quality improvement, we see a need for policy makers to invest in less, but more usable A&F reports.⁴⁴

Strengths and limitations

An important strength of this study was the think-aloud methodology, which enabled us to capture direct reactions to specific issues of the PCPR, instead of capturing general reflections on the PCPR or A&F. Hereby, we were able to identify items that explain the limited uptake and influence the usage of the PCPR. Another strength is that we examined an existing A&F report which is routinely available.

This study also had several limitations. First, we had a relatively limited recruitment strategy. While the group of GPs was diverse in several characteristics, they were relatively old which may have caused bias. Second, the interviewer was enlisted at the GP practice of one respondent, but since this prior relationship was limited, we do not believe this relationship has altered our results. Third, some of the respondents were familiar with the PCPR prior to the interview, which may have affected their responses to the think-aloud tasks. Lastly, since we did not work iteratively and completely independent during the second analysis phase this might have caused bias in the interpretation of our results. However, we tried to overcome this bias by discussing codes per transcript and discussing themes within the wider research team and since another study regarding the PCPR found similar results, we believe that our results are reliable.

Conclusions

The GPs in our study evaluated the PCPR as poorly usable and were minimally motivated to change behaviour based on the PCPR. The PCPR seems developed from the perspective of the commissioner, the health insurer, and does not meet known criteria for effective A&F design and user-centred design. Importantly, the GPs did state that well-designed feedback would contribute to their motivation to improve clinical performance.

Furthermore, GPs in our study stated that they receive a multitude of A&F reports, which they hardly use. For more effective quality improvement, we see a need for policy makers to invest in less, but more usable A&F reports.

Data availability statement

Data are available on reasonable request.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

According to the regulations of the BETHCIE, this study did not require full ethical review. Participants gave informed consent to participate in the study before taking part.