Using a dark logic model to explore adverse effects in audit and feedback: a qualitative study of gaming in colonoscopy

Abstract

Background

Audit and feedback (A&F) interventions improve patient care but may result in unintended consequences. To evaluate plausible harms and maximise benefits, theorisation using logic models can be useful. We aimed to explore the adverse effects of colonoscopy A&F using a feedback intervention theory (FIT) dark logic model before the National Endoscopy Database Automated Performance Reports to Improve Quality Outcomes Trial study.

Methods

We undertook a qualitative study exploring A&F practices in colonoscopy. Interviews were undertaken with endoscopists from six English National Health Service endoscopy centres, purposively sampled for professional background and experience. A thematic framework analysis was performed, mapping paradoxical effects and harms using FIT and the theory of planned behaviour.

Results

Data saturation was achieved on the 19th participant, with participants from nursing, surgical and medical backgrounds and a median of 7 years’ experience.

When performance was below aspirational targets participants were falsely reassured by social comparisons. Participants described confidence as a requirement for colonoscopy. Negative feedback without a plan to improve risked reducing confidence and impeding performance (cognitive interference). Unmet targets increased anxiety and prompted participants to question messages’ motives and consider gaming.

Participants described inaccurate documentation of subjective measures, including patient comfort, to achieve targets perceived as important. Participants described causing harm from persevering to complete procedures despite patient discomfort and removing insignificant polyps to improve detection rates without benefiting the patient.

Conclusion

Our dark logic model highlighted that A&F interventions may create both desired and adverse effects. Without a priori theorisation evaluations may disregard potential harms. In colonoscopy, improved patient experience measures may reduce harm. To address cognitive interference the motivation of feedback to support improvement should always be clear, with plans targeting specific behaviours and offering face-to-face support for confidence.

Trial registration number

ISRCTN11126923.

Background

Audit and feedback (A&F) interventions have been shown to improve compliance with desired practice in healthcare professionals; however, effect sizes have varied and some interventions are not successful.¹ There are calls for the more explicit use of theory to understand mechanisms of change in behaviour change interventions (BCIs) and to develop logic models to inform implementation.^{2 3} Theory has explained some variability^{4 5}; however, models often only focus on the intended benefits of the intervention and an additional explanation for variable effect size are unintended effects. In the public health sector BCIs involving human agency and complex social systems have been demonstrated to potentially have unintended or harmful consequences.⁶ These have included BCIs associated with higher rates of adolescent problem behaviour,⁷ teenage pregnancy⁸ and rates of sexually transmitted infections.⁹ It is hypothesised that modelling underlying mechanisms of paradoxical effects and harms provides an opportunity to avoid or minimise these problems.¹⁰

‘Dark logic’ is defined as the mechanisms by which an intervention hypothetically has adverse effects on the outcomes of interest (‘paradoxical effects’) and other outcomes (‘harmful externalities’). Dark logic models are developed by scrutinising models of intended change and their assumptions using a priori theorisation to actively hypothesise paradoxical effects and harms. These are recommended in public health and prevention science interventions,^{11 12} and have been used to critique and analyse public health intervention policy in the COVID-19 pandemic.^{13 14} To the knowledge of the authors, they have not been used in the development healthcare A&F BCIs.

Colonoscopy is a medical procedure that involves an endoscopist inserting a camera into the large bowel (intubation) then withdrawing the camera looking for pathology (withdrawal); beforehand, the bowel is cleansed (bowel preparation) to allow visualisation.¹⁵ Colorectal cancer (CRC) arises from polyps, and polyp detection and resection at colonoscopy is pivotal in preventing CRC. Performing colonoscopy can be challenging and poor-quality colonoscopy has serious consequences. Endoscopists with lower polyp detection rates have higher rates of CRC after colonoscopy.^16–18

The UK government has supported the implementation of a quality improvement programme in endoscopy, overseen by the Joint Advisory Group (JAG) on endoscopy and the British Society of Gastroenterology, and associated with a reduction in CRC mortality.¹⁹ The programme introduced a bowel cancer screening programme (BCSP) with advanced accreditation of screening endoscopists and national key performance indicators (KPI) of colonoscopy quality, including completeness of procedure (caecal intubation rate), polyp detection rates, withdrawal time and comfort.^{20 21} Previous trials of A&F to improve colonoscopy performance have had heterogenous results; this is hypothesised to be due to colonoscopy being a complex motor skill and poor implementation of BCIs.²²

In the development of the National Endoscopy Database Automated Performance Reports to Improve Quality Outcomes Trial (NED-APRIQOT), we undertook a wider qualitative interview study to explore the phenomenon of current A&F practices in colonoscopy to develop a BCI prior to its implementation later in 2020.²³ Feedback intervention theory (FIT)²⁴ has been demonstrated to be a suitable theoretical model for change in A&F interventions in healthcare settings,⁵ and has been recommended for the development of A&F processes in endoscopy.²² This FIT model was used to explore paradoxical effects, whereby a BCI increases a behaviour it seeks to prevent, and harmful effects to patient care. These are harms which differentially affect patients in the care of practitioners who, themselves, are the target of BCIs. Changes in intention partly predict behaviours²⁵ and the theory of planned behaviour (TPB) is moderately effective at predicting intention and behaviour.²⁶ The Cochrane review of A&F suggested that TPB is particularly useful to explore normative comparisons and was used to map participants’ beliefs within FIT.²⁷

The aim of this paper is to describe the phenomena of potential harms and adverse outcomes in A&F processes in endoscopy arising at interview using a theoretical model based on FIT, to inform the design of a future BCI.

Method

Independent endoscopists were recruited for face-to-face audio-recorded semistructured qualitative interviews at their workplace; these were followed by cognitive interviews assessing a draft BCI, reported in a separate study.²⁸ Interviews lasted up to 60 min. Clinical leads of English National Health Service endoscopy centres eligible for the NED-APRIQOT study in the Northern region or West Midlands were contacted by email. Sites which responded were selected with convenience sampling for participants’ availability. Eligible endoscopists²³ were purposively sampled with criteria comprising length of endoscopy experience and professional role (clinical lead, clinical nurse endoscopist, gastroenterologist, surgeon and trainee) or to aid data saturation. Up to five endoscopists were recruited at each site; recruitment continued until sampling strata were filled and data saturation was reached, defined as no new themes arising in the last three interviews after 10 interviews.²⁹

Interviewees were provided with a participant information sheet, explaining interviews would cover behaviours in endoscopy and A&F, and gave written consent. A topic guide was used, reviewed and revised (if needed) after each centre’s interviews to facilitate depth and data saturation (online supplemental appendix 1). Interviews were transcribed removing any identifiable information for analysis with demographic data pseudoanonymised using a unique participant identifier, and the interviewer (JC) kept a reflective log. Participants were provided with a copy of their transcripts to ensure anonymity and accuracy, and to drive meaningful conclusions from extracted quotes.

A framework method analysis was undertaken; FIT informed an initial logic model for the intended effects of A&F intervention in endoscopy, providing variables of interest, and a preliminary basis for a relationship between codes in the analytical framework FIT describes behaviours as tasks at three possible levels:

• Meta-task–beliefs about the self that are required to perform a task.

• Task motivation—where one applies an already learnt behaviour.

• Task learning—new tasks where one focuses on motor movements.

FIT suggests that if an individual identifies a gap between current performance and a target, they may adopt a strategy to reduce this gap and develop a coping mechanism, abandon the standard, reject the feedback message or change the standard (figure 1).

Figure 1.

A logic model of audit and feedback processes based on feedback intervention theory.

The TPB³⁰ was used to explore beliefs about behaviours within FIT. The TPB identifies beliefs as:

• Behavioural beliefs—attitudes towards and effects of behaviours.

• Control beliefs—perceived control of behaviours by participants.

• Normative beliefs—perceived social pressures around behaviours.

The framework method analysis used inductive ‘open coding’, based on Gale et al ³¹ involving the following steps:

• Preliminary reading of full transcripts, ensuring accuracy and adding context.

• Generation of initial descriptive codes, inductive ‘open coding’ paraphrasing the text ideally using participants’ own words.

• Developing an analytical framework after eight transcripts, grouping codes into subthemes tagged to FIT domains or TPB beliefs. Codes which did not sit within FIT or TPB were analysed in ‘bucket’ subthemes.³²

• Applying the analytical framework indexing subsequent transcript codes.

• Charting data: subthemes and their relationships were reviewed and mapped to FIT themes and a FIT logic model for behaviour change with corresponding quotes from data to ensure accuracy.

The dark logic model was developed through scrutinising the model for paradoxical effects and harms. Keeping in line with focus of this paper, themes of intended benefits and wider A&F phenomena are not reported. Interviews and analysis were undertaken by a single researcher (JC), codes were logged with an audit trail. Themes were reviewed with original quotation data to ensure accuracy and triangulated with observation data and personal reflection from the time of interview. As the coding and analysis progressed, the authors met to critically review, challenge and discuss findings. Data are reported in two major areas: paradoxical effects and harmful effects. Illustrative quotes are provided.

Results

Six endoscopy centres were recruited from April 2019 to January 2020, four in the North East of England and two in the West Midlands. Centres had a median of seven endoscopy rooms across sites; the range (two to eight rooms) demonstrated a good range of small to large endoscopy centres.

Saturation of themes was achieved by participant 19. Ten of the 19 participants identified as being female. Sampling criteria for professional background and experience were fulfilled and shown in table 1; there was a median of 7 years’ endoscopy experience, with a range of 2–29 years.

Table 1.

Endoscopy centres and their participants’ roles

Participant	Site ID	Independent endoscopy experience (years)	Professional background	Further BCSP accreditation
P1	1	5–15	Gastroenterology	BCSP
P2	1	<5	Nursing
P3	1	5–15	Gastroenterology
P4	1	<5	Colorectal surgery
P5	2	>15	Gastroenterology	BCSP
P6	2	>15	Gastroenterology	BCSP
P7	2	>15	Nursing	BCSP
P8	2	<5	Colorectal surgery
P9	3	5–15	Gastroenterology
P10	3	5–15	Nursing
P11	3	<5	Colorectal surgery
P12	3	5–15	Nursing
P13	4	5–15	Gastroenterology
P14	5	>15	Nursing	BCSP
P15	4	<5	Gastroenterology
P16	5	<5	Nursing
P17	1	<5	Gastroenterology (trainee)
P18	6	<5	Gastroenterology (trainee)
P19	6	>15	Colorectal surgery

BCSP, bowel cancer screening programme accredited endoscopist.

All sites provided endoscopists with A&F data by email at least every 6 months, as part of routine practice in line with JAG recommendations. This provided a comparison of performance to national standards and a social comparison to others in the endoscopy centre. Persistent underperformance was managed by centre leads as recommended in national guidance.^{20 33}

Paradoxical effects

Paradoxical effects are summarised in figure 2 in orange.

Figure 2.

Dark logic model for audit and feedback harms in endoscopy using feedback intervention theory.

Rejecting the gap: seeing peers’ performance

All centres provided national standards for minimum performance,²⁰ and a normative comparison to peers with information about other local endoscopists’ performance. Participants saw the aspirational performance of comparable peers as motivating and benchmarked against colleagues whom they recognised as experts. ‘I looked at who had the best polyp detection rate and I thought, “I would like my polyp detection rate to be nearer to that”’ (Participant (P) 13).

Participants often identified themselves in a social professional group of peers with comparable case mix, job plan or professional background, and this identity was viewed as important to them. Within these groups, social norms were identified for performance and endoscopists had a perception of ‘what your level is at’ (P9). Performance being perceived as similar to others within this referent group reduced motivation to improve, even if this performance was below a centre-wide average or an aspirational target achieved by other peers. ‘One of our other nurse endoscopist colleagues who was always the same as me [below target] and we did a lot of endoscopy, us two … the ones who actually had higher detection rates than me were actually, I thought personally, not as good endoscopists’ (P10, table 2: subthemes (S) 1–2).

Table 2.

Paradoxical effect subthemes and illustrative quotations

Subtheme	Quotation
1. FIT identify gap—TPB normative belief: norms for identified group	‘I think you kind of have a concept in your mind of what your level is at and who you should be, what group you should be in.’ (P9)
	‘When I look at other endoscopists they always seem to be a little bit higher than mine. There was also one of our other nurse endoscopist colleagues who was always the same as me and we did a lot of endoscopy, us two, and I wondered whether it was because we did so many that that had an effect and that the people who were actually higher detection rates did very few colons… the ones who actually had higher detection rates than me were actually, I thought personally, not as good endoscopists.’ (P10)
	‘You want to see where you stand within the group and everyone else. So, for me, it reinforced to do what I was doing. I don't think I made any particular changes, to be honest.’ (P17)
2. FIT identify gap—TPB normative belief: seeing others’ underperformance reassurance	‘I guess if your performance is not so good but everybody else’s is similar then you're a bit like, “Well that’s probably okay,” and that’s probably not quite the right way to go about it.’ (P3)
	‘I would want to see more people at my level, on the report, to know if I’m as crap as the report is making out or if this is reality of where I fall.’ (P16)
3. FIT cognitive interference: reduced confidence	‘Probably I would but I don’t want to bog myself down with [KPI]. Because I know there are some endoscopists who get bogged down with completion rates and things like that and you know at the end of the day if you can't manage to get where and you’ve tried every avenue to get that caecum you have to accept that.’ (P10)
	‘It was all quite negative feedback and all not positive, which I came back from the course seriously reconsidering whether I should actually continue in endoscopy. I lost a lot of confidence and I think by giving that sort of feedback without any positives … can actually destroy your confidence and I think to do this job you need to be confident in what you’re doing … [since the feedback] I struggled with a lot of different things. Things I had been doing naturally, I had been doing what I had been taught I was very hesitant to do, because it had not worked on the course, and all the crappy feedback I got.’ (P2)
	‘I feel quite confident and competent, although I suppose, like everybody, you get dips and peaks and troughs, which is the world, especially, of colonoscopy…Or you get a run, which you tend to, of incomplete colonoscopies, and you think, ”What am I doing here?”’ (P14)
4. FIT goal hierarchy: task motivation processes	‘So you’re busy thinking “get to the caecum”, what am I going to do, change position, withdraw some air, put some water in, what can I do to optimise my position and you’re thinking all these things in your head.’ (P12)
	‘I tend to break operations up into lots of little steps, that have to be achieved, before you proceed to the next one. So, I would achieve it by saying, the first step is intubation, … check the patient’s position, check you’ve given them the correct medications and then you can start the second part, which is withdrawal and detection and resection.’ (P4)
	‘It’s quite a nice technical challenge…Using the force and I use that in the “Star Wars” thing to work out how you’re going to coax the scope round. So, hope the force is with you and you can be like Luke with the helmet off, dropping the scope just down into the middle of the Death Star.’ (P19)
5. FIT cognitive interference: feedback’s ulterior motives	‘I think initially I'll often be like, “I'm not doing enough. Why am I not doing enough? Am I being got at?”’ (P3)
	‘I think what you need to be very careful of is that you don’t move beyond monitoring to ensure safe standard and to drive up and improve standards into persecution, … making more of an issue for an individual than is actually the case when perhaps that individual needs time just to settle in and let things calm down, rather than actually that individual is not competent at doing what they are doing.’ (P11)
	‘I know that there is concern amongst my colleagues that this kind of data is used against you and I think there’s some people that feel that it may be used to stop you doing certain procedures, which I think is worrying and I hope that’s not the intention of it.’ (P8)
	‘If [my performance] dropped then I guess there is always a worry of whether they will stop me doing this.’ (P15)
6. FIT cognitive interference: leading to gaming	‘I think the danger is not having people get too bogged down in [KPI] so they start gaming their numbers.’ (P3)
	‘I mean that a lot of my biggest concern about these KPI’s is they’re either encouraging people to lie, or they are encouraging people to attempt to do something which is maybe to the detriment of the patient, because they are concerned about their outcomes. … We didn’t want [endoscopists] to feel that they were under pressure to do these or to go the extra mile if the patient had comorbidity or was finding discomfort.’ (P5)
	‘We’re all aware of the goals of the unit but they’re not shoved down our throats, so we’re not made to do things that make us feel uncomfortable to hit these targets. We’re told what our data is and have the opportunity to discuss that with the endoscopy lead and if there are any concerns, that would be raised but it’s not rammed down us all the time, this is what you have to achieve, not at all.’ (P8)

FIT, feedback intervention theory; KPI, key performance indicator; TPB, theory of planned behaviour.

Cognitive interference and quitting

Participants described that thinking about A&F data could impede performance and risked endoscopists getting ‘bogged down’ (P10). One participant described ‘thinking about your figures, it’s probably not brilliant …I mean I find it can be really emotionally draining’ (P1). Participants described how negative feedback, without a plan to improve performance, reduced their confidence, worsened performance and increased the risk of quitting colonoscopy (table 2: S3).

Colonoscopy is a complex motor skill, with many underlying task motivation processes which participants were able to describe, including: position changes, managing air and resolving loops (table 2: S4). However, some of these task motivation processes were higher level behaviours which participants struggled to describe: one participant described ‘using “the force”’ (P19, table 2: S4), suggesting colonoscopy is related to confidence, a meta-task self-perception behaviour. A clinical lead described the reason endoscopists quit endoscopy ‘wasn’t because of any technical ability, it was just [their] confidence’ (P1). There were concerns that an individual’s feedback ‘might end up being a …bit destructive’ with the risk that ‘you think I’m not very good here… that might be quite demotivating’ (P1). However, participants recognised being receptive to negative feedback was an important part of a clinical role and a culture for quality improvement.

Participants generally accepted that KPIs showing repeated underperformance suggested something was wrong, and if an endoscopist was unable to address underperformance they should consider or may be asked to stop scoping: ‘if we’re not good at something and we’ve tried to address it and we can’t find what’s wrong and you can’t address it, then maybe you just need to think about something else or giving it up’ (P18). Stopping scoping was described as having financial and psychological consequences for endoscopists, ‘If [the A&F process] wasn’t successful … we couldn’t really maintain her as an endoscopist and therefore there would be a big salary hit… But I also think it was more she regarded herself as a failure’ (P1).

These high stakes lead to cognitive interference and anxieties focused on possible motives of the feedback other than to improve performance. These suspected ulterior motives included accusing endoscopists of not doing enough, persecuting endoscopists who were wrongly perceived to be incompetent and policing performance to stop people from scoping (table 2: S5). Cognitive interference put endoscopists ‘under pressure … to go the extra mile’ (P5) to reach targets, which could lead to gaming and harmful externalities. One participant described they were made aware of targets ‘but they’re not shoved down our throats’ (P8) which reduced the pressure to consider gaming (table 2: S6).

Harmful effects

When you are getting performance figures … at times you’ve got to think are you doing this [behaviour] for your figures or are you doing it for the patient … when you do it more for the patient, then you do notice your figures drop. So it is a hard one, to manage that. (P12)

Harmful effects are summarised in figure 2 in red and were mapped to a ‘gaming’ (P3) theme. Themes describing ‘harm’ were categorised as being indirect and direct. Indirect harms were generated from inaccurate documentation (‘fudging’ (P8)), and direct patient harms from removing polyps without clinical indication and persevering to complete procedures.

Inaccurate documentation: withdrawal time

A minimum withdrawal time of 6 min is set by the British Society of Gastroenterology.²⁰ Withdrawal time was perceived not to be taken seriously by some endoscopists who would document 6 min without accurately noting the time. ‘“So, for the purposes of a quiet day I’m going to say this is six minutes and I really don’t care if anyone around me knows it isn’t”… I’m sure it happens in every department’ (P5).

In three centres, nursing assistants were trained to time withdrawal on behalf of endoscopists with the goal of improving withdrawal time as demonstrated in previous trials.³⁴ This was initially perceived as intimidating external scrutiny by participants but they had come to consider that it reduced fudging withdrawal times. ‘I would like to think I wouldn’t [game withdrawal time] but it is hard for me to hide now because the nurses are documenting it … [Laughter]’ (P15). When assistants timed withdrawal, participants described other endoscopists engaging in time-wasting behaviours, such as starting the timer early and ‘hanging around’ in the rectum at the end of the test, these behaviours prolong the length of the test without improving colonic inspection or benefiting the patient (table 3: S1).

Table 3.

Harmful effects—documentation subthemes and illustrative quotations

Subtheme	Quotation (Participant (P))
1. FIT gaming indirect harm: fudge withdrawal time	‘That’s always one of the things that you worry about is are you just going to take off a load of rectal polyps in the same way that you could just withdraw and sit in the rectum for five minutes, couldn't you, and then say your withdrawal times.’ (P3)
	‘When it comes to withdrawal times … some people will say, “Start the clock,” and then [the nurses] say well, “Are you actually at the caecum?” … some endoscopists who had then kind of hung around the rectum for a couple of minutes saying, “I’m staying here around the rectum for a couple of minutes because I have to”.’ (P5)
	‘Of course, you have to be wary of withdrawal time. I hear anecdotally from the nurses that: “So and so has a long withdrawal time but eight minutes of that [they are] sitting in the rectum talking”.’ (P19)
2. FIT gaming indirect harm: conversion to sigmoidoscopy	‘People have changed what was an intended colonoscopy to a flexi sig because of poor prep or that’s as far as they’ve got and you can see the nursing documentation, the original referral.’ (P7)
	‘I know some people kind of falsify figures and I always put if it’s a colon it’s a colon. If it’s a failed colon it’s a failed colon. It’s not a sigmoidoscopy it’s a colonoscopy and I know that people do not always follow that… That skews the figures as well… It’s very common.’ (P10)
3. FIT gaming indirect harm: bowel preparation	‘I do understand that some endoscopists could potentially fudge their figures and are feeling tired, “oh poor bowel prep, let’s just come out”. So I do understand that, so I don’t know what the answer is but it is hard because you do get poor bowel preps.’ (P12)
	‘My caecal intubation rate is lower than it should be perhaps, mainly because of poor bowel prep.’ (P2)
	‘Well part of the problem with [detection rates] is some that will be down to poor prep. I can think of a number of cases recently where the prep in the right colon is fairly smeary. If you had better prep, you’d probably have [detection].’ (P19)
4. FIT gaming—TPB control belief: comfort	‘There’s no consistency with regard to whether the data is the endoscopist’s personal view or whether it’s they have taken feedback from the nurses in the room talking to the patient.’ (P5)
	‘I was signing off a colonoscopy portfolio for somebody for JAG certification and I just said, “You’ve done your 230 or whatever.” I said, “Why is everyone of your patients comfortable?” … I hadn’t done individual training with this person. And I just thought, “Oh my God. What on earth.” They said, “Well that’s what the consultant’s put on the thing.”… But it just wasn’t important to him. I just thought, oh my goodness, throughout his training, where has that perception come from. And I was really worried … I said, “I’m not signing you off until you go back and we do some more work and have a look at sort of these comfort scores with patients and whatever.” But it was through no fault of their own in some ways but I was absolutely horrified.’ (P7)
	‘Comfort has been such a difficult thing … every unit does it differently, newer nurses will call your patients as higher discomfort because they’ve never seen the procedures before …we’ve got everybody using different nursing scales.’ (P7)

FIT, feedback intervention theory; JAG, Joint Advisory Group; TPB, theory of planned behaviour.

Most participants expressed beliefs that polyp detection is important and longer withdrawal times improve detection. Participants assumed that endoscopists who undertake time-wasting behaviours did not appreciate the clinical importance of withdrawing slowly, ‘a lot of people just see it as getting the scope out. And maybe aren’t as aware that it’s a really key part of the examination, especially if they trained quite a long time ago’ (P12).

Inaccurate documentation: completion rates

The participants described examples where bowel preparation and procedure documentation could artificially inflate completion rates. Participants reported if endoscopists were unable to complete a colonoscopy that some converted the procedure documentation from a colonoscopy to a shorter flexible sigmoidoscopy (table 3: S2). One participant noted that if the insertion was difficult, endoscopists may inaccurately document inadequate bowel preparation, ‘oh poor bowel prep, let’s just come out’ (P12), to later justify a low completion rate (table 3: S3). Bowel preparation was not perceived to be under the endoscopist’s control, ‘you can’t change bowel prep’ (P17), and inadequate preparation limiting colonoscopy quality was not perceived as the endoscopist’s fault (table 3: S3).

Comfort score and patient experience inaccuracy

The participants perceived patient experience and comfort are important to colonoscopy quality and to patients. Comfort is a recognised colonoscopy quality KPI and used in colonoscopy A&F practice.²⁰ Comfort scores are an assessment of the patient’s experience by the endoscopist or nursing assistant, these were perceived as inconsistent and of variable quality (table 3: S4).

One participant described their experience as a trainer, reviewing a trainee endoscopist’s portfolio and their patient comfort scores. The participant noted that in all 230 procedures, all patients were documented as being comfortable and noted that this would not be possible. This ‘horrified’ the participant, ‘[the trainee] said, “Well that’s what the consultant’s put on the thing.”… it just wasn’t important to [them]’ (P7).

Harmful effects: patient care

Perseverance despite patient discomfort

Participants perceived that colonoscopy can be painful, and that persistent patient discomfort should limit colonoscopy (table 4: S1). Participants described being ‘frightened’ (P7) by their completion rate performance figures causing them to ‘drive on and cause [patients] discomfort and pain’ (P14) to achieve a complete test. One participant perceived pressure to have a high completion rate to achieve BCSP accreditation and described completing procedures with poor bowel preparation despite being aware the behaviour ‘was unsafe, I’m going to miss loads of pathology here’ (P12, table 4: S2).

Table 4.

Harmful effects—patient care subthemes and illustrative quotations

Subtheme	Quotation (Participant (P))
1. FIT gaming—TPB normative belief: patient experience should limit colonoscopy	‘I’m not one who’s going to push through a very painful, uncomfortable, difficult colonoscopy, just to get caecal intubation, if that’s not right for the patient, then I’m not going to do it.’ (P8)
	‘… [If] you’ve had a run of a few people who are uncomfortable, I don’t think that actually saying that you said, no I’m not going to torture you anymore and withdrew, I don’t think that’s an unreasonable situation to be in. Rather than when we have reasonable alternatives, in terms of CT colonography.’ (P11)
2. FIT gaming direct harm: perseverance despite patient comfort	‘I know sometimes people persevere with things that they shouldn’t be persevering with because they’re frightened of their figures, performance data. And I think that has become a bit of a danger.’ (P7)
	‘If you’re going to have bad bowel prep, do you then fight your way through the bad bowel prep to get to caecum. So then you get your caecal intubation high or do you actually say, look this is unsafe, I’m going to miss loads of pathology here, let’s call it a day. But then obviously recently, because I’m going for bowel cancer screening, caecal intubation rates is very important to me. So I have been fighting against poor bowel prep, where normally I would just call it a day, rebook, give enhanced bowel prep.’ (P12)
	‘If you’re looking at completion rates it’s different. Because people then drive on and cause people discomfort and pain.’ (P14)
3. FIT gaming direct harm: unnecessary polypectomy	‘I guess if you’re always slightly under you’d say, “Well do I need to start thinking about looking a bit harder or taking off something, rectal hyperplastic ones?”’ (P3)
	‘The goalposts have moved. So, I think initially, it was just, as I say, when we first started, people were leaving the small stuff in the 74-year old whereas now you’re just accepting that you’re taking everything off.’ (P7)
	‘Simply a polyp detection rate isn’t good enough because you can always find metaplastic polyps in an elderly population.’ (P19)
4. FIT gaming direct harm: unnecessary polypectomy risks harm	‘In a [frail elderly] patient where you find the diminutive polyp and are you really going to risk that patient having a perforation by taking off a small polyp which is not going to ever have any effect on their life span? … I think the [endoscopy] standards are potentially causing risk in those two areas. … the wrong incentive is there.’ (P5)
	‘More senior gastroenterologists will be more pragmatic … if I actually take that [insignificant polyp] off and make a hash of it and they’re elderly and frail, I’m doing this [colonoscopy] to prove that they haven’t got a big cancer … taking something little off, that could cause them more harm, they’re not going to do it.’ (P7)

FIT, feedback intervention theory; TPB, theory of planned behaviour.

Unnecessary polypectomy

Detection and removal of colonic polyps was described as important by all participants and the ‘main goal’ (P12) of colonoscopy. Participants described polyp detection and polypectomy KPIs as incentivising the removal of clinically insignificant lesions such as rectal diminutive hyperplastic polyps (table 4: S3). International guidance does not recommend the removal of such lesions.^{35 36} This behaviour to increase the recorded detection rate was recognised as having no clinical benefit to the patient, ‘snipping those off isn’t going to help a patient’ (P8), and potentially increasing the risks of complications particularly in the ‘elderly and frail’ causing ‘more harm’ (P7). Removing or leaving a polyp was not always a clear decision, and assessing risk and pragmatism were recognised as being important (table 4: S4).

Discussion

Statement of principal findings

Our study is the first to explore paradoxical effects and potential harms of current A&F interventions in colonoscopy using a dark logic model based on FIT. Paradoxical effects included social norms reassuring underperformance and performance anxiety causing cognitive interference which impacted the meta-task of confidence. Participants described inaccurately completed documentation so that completion rate and withdrawal time targets appeared to be achieved. Harmful behaviours included perseverance with the colonoscopy procedure despite patient discomfort and unnecessary polypectomy.

Strengths and weaknesses of the study

We are the first to present a dark logic model for A&F, such models focus on paradoxical effects and harms of BCIs, and to the reader and researcher can feel relentlessly negative. Dark logic models, like ours, should be situated in a wider model for behaviour change, incorporating intended effects and benefits, to inform A&F practice.

The study team perceived harms would be a difficult topic to discuss; however, participants frankly discussed gaming behaviours. On reflection, the interviewer was an endoscopist, who recognised participants’ experiences and used the same language and references. The interviewer was acquainted with four participants through academic or clinical work and had previously received training from five participants. The interviewer was junior to the participants in age, his position as a trainee and experience. It is possible that this shared clinical background and, in some instances, prior acquaintance helped establish rapport, encouraging an open dialogue, although occasionally communication had vestiges of the trainer–trainee relationship.

Our purposive sampling of participants with a range of professional backgrounds and clinical experiences adds to the transferability of these findings to wider clinical contexts.³⁷ Fewer participants were selected in sites 4–6 to fulfil sampling criteria. Although participants’ experiences of A&F across sites were similar, sites had different organisational contexts, including safety management approaches and clinical leadership training,^{38 39} which may impact perception of performance management. Reassuringly, data saturation of themes was maintained across sites.

In responding to correspondence and agreeing to be interviewed about performance, we may have a self-selected group of those with a personal interest in colonoscopy quality. Although the prevalence of gaming behaviours in endoscopists is unknown, examples of gaming were described by endoscopists across professional backgrounds and lengths of experience. Participants rarely described their own negative behaviours, but those of unnamed others. These were disclosed in a conversational tone, with an implied intention to prevent them. The findings of this work were presented locally to endoscopy colleagues, who confirmed they recognised these behaviours in their own practice, and the pressures to undertake them.

Strengths and weaknesses in relation to other studies

The Cochrane review found A&F interventions were modestly effective, but demonstrated high variation in effectiveness.¹ Ivers et al describe a lack of understanding regarding how A&F works; they recommend barriers to A&F effectiveness, including interpretation of interventions by clinicians, should be explored.⁴⁰ One barrier of organisational targets causing individuals to undertake paradoxical behaviours has been described in the English public health sector as gaming or ‘reactive subversion’.⁴¹ Another hypothesised barrier is poorly validated healthcare outcome data,⁴² causing a balance of harms and benefits from false-negative and false-positive ‘diagnoses’ of quality care.

Past research evaluating A&F barriers has focused on organisational effects and not individual behaviours. A&F work in blood transfusion has used empirical qualitative study and theory to address organisational barriers to intervention efficacy.⁴³ Paradoxical organisational findings have included variation in how hospitals received, shared and responded to feedback,⁴⁴ and worsening variation in performance by applying action in an on-off manner.⁴⁵

Our paper focuses on individual practitioner behaviours. Application of behavioural theories, such as clinical performance feedback intervention theory (CP-FIT), has been used to retrospectively explain why feedback may not have been effective at changing individuals’ behaviour, but without prospective theorisation of potential harms or paradoxical effects.⁴⁶ Our paper demonstrates a theoretical model to prospectively hypothesise and explore A&F’s pathways to harmful effects from individual behaviours. This analytical process did not balance pathways of intended and unintended effects, but explored mechanisms of potential harms, prior to the implementation of an A&F intervention.

FIT and the TPB complimented each other as working theories in the analysis; TPB aided exploration of control and normative beliefs impacting intention, and FIT allowed mapping of their potential impact on behaviours. Exploration of paradoxical effects was enriched with normative beliefs. Social norm feedback is effective in changing healthcare behaviours when practitioners see themselves as an outlier, such as reducing antibiotic overprescribing.⁴⁷ Psychology literature has described social norms having paradoxical ‘boomerang’ effects on high performers.⁴⁸ Our participants did not describe boomerang effects from social comparisons, perhaps as there is little ambiguity that high detection is positive.⁴⁹ However, low performers were reassured by low-performing peers. As suggested in social comparison theory, performance aligning with others reduces the motivation to change behaviour. This highlights the importance of using an aspirational social comparison of comparable peers (box 1).^{50 51}

Box 1. Implications for avoiding negative impacts of A&F.

To reduce negative impacts of A&F:

• Use aspirational social feedback.

• Measure what is important accurately, including patient-reported experience.

• Identify and address educational needs around behaviours and documentation.

• Avoid cognitive interference, anxiety and reducing confidence through:

  • Action plans targeting task motivation behaviours.

  • Providing personal support and buddying.

Participants reported inaccurate documentation caused by A&F pressures, where endoscopists identified targets they perceived as important and wished to appear to reach them. Changing documentation to game process outcomes is a recognised unintended consequence of A&F in endoscopy.^{52 53} Our participants described choosing inaccurate documentation over undertaking behaviours to improve quality. This may be related to low perceived control of behaviours from competing time pressures or workload.⁵⁴ We identified potential educational needs around behaviours to improve performance and documentation; educational interventions addressing these and supplementing A&F may be effective (box 1).²²

Mechanisms and implications for clinicians or policy makers

Cognitive interference

Colonoscopy performance has been described as a complex psychomotor skill requiring higher cognitive tasks,⁵⁵ and our participants described confidence (a meta-task belief) as a requirement. KPIs in healthcare often have high levels of complexity in their underlying tasks.⁵⁶ FIT suggests receiving negative feedback can confront perceptions of the self and cause anxiety. This draws attention away from undertaking tasks and increases pressure on performance, called cognitive interference.²⁴ Our participants’ anxiety was increased by underperforming against national guidelines.²⁰ Participants were aware that JAG recommends stopping endoscopists performing endoscopy if underperformance is assessed to be unsafe⁵⁷; with perceived personal, psychological and financial consequences. Participants described cognitive interference may pressurise endoscopists to perform gaming behaviours. To avoid harmful behaviours, A&F interventions need to address the underlying cognitive interference which drives these behaviours (box 1).⁵⁸

Measuring performance

Cognitive interference and gaming pressures are highest on behaviours with outputs perceived as being inaccurate or unmeasured, which may be sacrificed to achieve measured targets.⁴¹ A challenge for A&F is to identify what is important and measure it well.

Increasing the accuracy of targets to better measure important behaviours can prevent incentivising harmful behaviour. This study demonstrates that underperformance against detection targets incentivises removal of insignificant polyps, potentially risking patient safety. Improving targets to focus attention on clinically significant polyps and linking A&F systems to polyp histology data may reduce harmful behaviours.^{23 59 60}

Patient experience is a key aspect of healthcare quality. Comfort scores that are endoscopist reported are criticised, as patients and clinicians have different priorities around the healthcare experience.^{58 61} Our study demonstrates perceptions that patient comfort documentation is variable and sometimes inaccurate. In the poor recording of the patient experience, A&F processes potentially expose patients to the risk of discomfort as practitioners may prioritise achieving better measured performance targets. Assessment and recording of the patient experience with validated patient-reported experience measures, such as the ‘Newcastle ENDOPROM’ in endoscopy, may reduce this risk (box 1).⁵⁸

Unanswered questions and future research

Our dark logic model suggests addressing cognitive interference and anxiety of underperformance is critical for reducing potential A&F harms. Goal setting and action planning in A&F, focusing on task motivation behaviours which practitioners can implement to improve performance, may reduce cognitive interference (box 1). Such action planning is a clear tenet of FIT,²⁴ CP-FIT,⁴⁶ and is recommended for intervention design in the Cochrane review.²⁷ For example, in colonoscopy, task motivation behaviours to improve detection include increasing withdrawal time and turning the patient’s position on withdrawal.^{62 63} A colonoscopy BCI with action plans targeting these behaviours and supplemental educational material is being tested in the NED-APRIQOT study.²³

An endoscopy A&F intervention targeting leadership training demonstrated improved centre-wide colonoscopy performance.³⁹ Our study demonstrated that where there is underperformance in meta-task behaviours, such as low confidence, then addressing this is a complex social task. This is the challenging work of local clinical leaders. Our study suggests clinical leads should clearly identify their motivation to provide support and alleviate anxiety. Opportunities to be observed performing complex skills, or buddying, for those persistently underperforming may be used to explore understanding of targets, develop behaviours poorly assessed by KPI and bolster confidence (box 1).³³ Further study of leaders’ experiences implementing support for those persistently underperforming, and identification of leaders’ training needs should be explored in future research.

Conclusion

This example of using a dark logic model to map adverse effects has been insightful in our endoscopy setting and can be applied to different clinical settings where A&F is used to improve performance. Our dark logic model highlighted that A&F interventions, in accordance to FIT, may create a mix of desired and adverse effects. Without a priori theorisation, evaluations may disregard potential harms. In this setting, improved patient experience measures may reduce harm. To address cognitive interference the motivation of feedback to support improvement should always be clear, with plans targeting specific task motivation behaviours and offering face-to-face support for meta-task behaviours such as confidence.

Data availability statement

Data are available upon reasonable request. All participants provided written consent for non-identifiable publication of transcript extracts and direct quotations from data. Data were accessed in conjunction with Newcastle University data security policy. The data sets generated and/or analysed during the current study are not publicly available due to possible identification of participants through triangulation but are available from the corresponding author upon reasonable request.

Ethics statements

Patient consent for publication

Not required.

Ethics approval

This study involves human participants and ethics approval was granted as part of the National Endoscopy Database Automated Performance Reports to Improve Quality Outcomes Trial (NED-APRIQOT). Qualitative interview study ethics approval was granted by the Newcastle University Ethics Committee (Ref 9521/2018). Participants gave informed consent to participate in the study before taking part.