Skip to main content
NCBI home page
The BMJ logoLink to The BMJ
. 2002 Feb 16;324(7334):419–421. doi: 10.1136/bmj.324.7334.419

What to do about poor clinical performance in clinical trials

Su Mason a, Jon Nicholl b, Richard Lilford c
PMCID: PMC1122341  PMID: 11850378

The performance of individual clinicians is being monitored as never before. Su Mason and colleagues discuss the implications of this for clinical trials and recommend what should happen if during a trial the performance of one clinician or one centre is identified as being particularly poor. Tom Treasure, a surgeon, wants the monitoring to be done fairly and to take account of the complexities of clinical practice; and Heather Goodare, a patient, wants to be told when things go wrong.

The Department of Health in England has issued guidelines for research governance stating that healthcare organisations remain responsible for the quality of all aspects of patients' care whether or not some aspects of the care are part of a research study.1 We discuss how this obligation can be met in multicentre trials, given that data on the performance of clinicians are held by the trial management team, not by the host organisation.

Summary points

  • Guidelines on research governance from the Department of Health emphasise the importance of patient safety in trials

  • We suggest that healthcare organisations should make the trial management team responsible for monitoring safety through statistical analysis

  • Taking action on suboptimal results, however, remains the institution's responsibility

  • The rules for monitoring and responding to suboptimal performance should be made clear to everyone in advance

Should we monitor, and who should do it?

If neither the host organisation, nor the trial team, takes responsibility for monitoring performance then patients are left with no protection against substandard practice.2 We are aware of the dangers of applying a higher standard of scrutiny to clinical trials than to routine practice, but clinical trials often involve a relatively new treatment (such as an innovative surgical operation) where outcomes vary by skill.3,4 With any new treatment it is appropriate to scrutinise outcomes—whether or not the treatment is part of a comparative study. But in a clinical trial who should be responsible for statistical monitoring of outcomes?

Three possibilities arise:

  • Both the healthcare organisations and the trial team could collect and monitor performance data, but this is likely to lead to confusion of responsibility and a waste of effort.

  • Healthcare organisations could be required to collect and monitor outcomes independently of the trial, but they would have, at best, limited access to the comparative data. It would thus be hard to judge whether, say, a 5% rate of blood transfusion was too high. This applies especially to new technologies where good quality data for comparison do not (yet) exist.

  • The trial statistician responsible for analysing data for interim monitoring could scrutinise outcome by clinician or organisation at little extra effort. We suggest that this is the best option and fits with the concept of “tracker trials.”5 This is the idea that a trial is not simply a randomised comparison of two generic treatment methods but also an observational comparison of subcategories of treatment (such as different devices and different centres or clinicians). Thus, under our proposal, healthcare organisations would discharge part of their responsibility for patient welfare during a trial by explicitly giving responsibility for monitoring to the trial team.

We will not deal here with the tricky statistical issues of identifying outliers.6–9 The aim of the trial statistician, however, would be to identify clinicians or centres that lie outside the bounds of acceptable practice—those who are in a different division, not just bottom of their league. But when such a clinician is identified what should happen?

Action on poor performance

The responsibility to future patients, who might be harmed by a particular clinician, trumps all others. The Department of Health guidance states: “The dignity, rights, well being and safety of participants must be the primary consideration in any research study.”1 Thus, not only must the clinician's participation in the trial be suspended, but the person responsible for clinical governance in that clinician's healthcare organisation must be informed. The organisation's usual response might be to offer retraining to the clinician.

However, three further issues arise.

Firstly, we need to consider the welfare (particularly the anonymity) of collaborating clinicians who take part in trials. It would not be in the public interest if clinicians declined to take part in studies for fear that they might “incriminate” themselves.10 The anonymity of clinicians should therefore be respected even though their use of particular procedures is suspended. Since the purpose of the trial is to influence practice at large, trial procedures should follow routine practice as closely as possible. If we assume regular audit of results under clinical governance, then suboptimal performance in daily practice will be detected, and the consequences should be similar to those that occur in a trial—namely, suspension of activity pending retraining. (Detection of poor performance in routine post-trial practice would, of course, be facilitated by the availability of “benchmark” results from preceding trials.)

Secondly, a methodological issue arises if the data from one clinician is effectively censored. We advocate that the data accrued by the clinician should be included up to the point when he or she was suspended from the trial.

The third issue to consider is whether there is a duty to inform patients who have already suffered complications and who were under the care of a clinician subsequently found to have substandard results. If there is such a duty then it conflicts with the obligation to protect clinicians' anonymity. There are strong arguments against the routine feedback of individual performance to past patients, which include the creation of perverse incentives (for example, an incentive to treat only patients likely to have good outcomes) and the limited likelihood of net benefit from such retrospective disclosure. Nevertheless, however society decides to handle this issue, we think that trial practice should mirror routine practice.

Next steps

Following a reasonable public debate, we suggest that guidelines should be promulgated by organisations responsible for scientific governance. Such guidelines should state precisely who is responsible for doing what—for example, the trial statistician could be responsible for identifying “outliers,” the data monitoring and ethics committee11 for ratifying conclusions, and the trial steering committee for informing the relevant health service organisation, which in turn would be responsible for retraining. Secondly, clinicians and those responsible for clinical governance should know in advance that the outcomes of individual clinicians will be analysed as a trial goes on, and they should be aware of their rights and responsibilities if problems arise, so that they are not ambushed by the process. Patients should also be told what would happen if their centre or clinician turned out to have suboptimal results—for example, the circumstances under which this would or would not be divulged.

Figure.

Figure

ALEXANDER TSIARAS/SPL

Open in a new tab

Monitor this

Acknowledgments

We thank Professor Treasure and Dr Elizabeth Clough for helpful comments.

Footnotes

Funding: SM, JN, and RL are all supported by the Department of Health/NHS Executive, but the views expressed here are entirely their own.

Competing interests: None declared.

References

  • 1.www.doh.gov.uk/research/rd3/nhsrandd/researchgovernance.htm (accessed 18 Dec 2001).
  • 2.McArdle C. ABC of colorectal cancer. Primary treatment—does the surgeon matter? BMJ. 2000;321:1121–1123. doi: 10.1136/bmj.321.7269.1121. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Braunholtz DA, Edwards SJ, Lilford RJ. Are randomized clinical trials good for us (in the short term)? Evidence for a “trial effect”. J Clin Epidemiol. 2000;54:217–224. doi: 10.1016/s0895-4356(00)00305-x. [DOI] [PubMed] [Google Scholar]
  • 4.English TA, Bailey AR, Dark JF, Williams WG. The UK cardiac surgical register, 1997-82. BMJ. 1984;289:1205–1208. doi: 10.1136/bmj.289.6453.1205. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Lilford R, Braunholtz D, Edwards S, Greenhalgh R. Randomised trials and fast changing technologies: tracker trials. BMJ. 2000;320:43–46. doi: 10.1136/bmj.320.7226.43. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.De Leval MR, Francois K, Bull C, Brawn W, Spieghalter D. Analysis of a cluster of surgical failures. J Thorac Cardiovasc Surg. 1994;107:914–924. [PubMed] [Google Scholar]
  • 7.Treasure T. Rational decision making about paediatric cardiac surgery. Lancet. 2000;355:948. doi: 10.1016/S0140-6736(00)90010-2. [DOI] [PubMed] [Google Scholar]
  • 8.Stark J, Gallivan S, Lovegrove J, Hamilton JR, Monro JL, Pollock JC, Watterson KG. Mortality rates after surgery for congenital heart defects in children and surgeons' performance. Lancet. 2000;355:1004–1007. doi: 10.1016/s0140-6736(00)90001-1. [DOI] [PubMed] [Google Scholar]
  • 9.Rothwell PM, Warlow CP. Interpretation of operative risks of individual surgeons. European Carotid Surgery Trialists' Collaborative Group. Lancet. 1999;353:1325. doi: 10.1016/s0140-6736(99)01000-4. [DOI] [PubMed] [Google Scholar]
  • 10.Wu AW. Medical error: the second victim. BMJ. 2000;320:726–727. doi: 10.1136/bmj.320.7237.726. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.MRC guidelines for good clinical practice in clinical trials. London: MRC; 1998. [Google Scholar]
BMJ. 2002 Feb 16;324(7334):419–421.

Commentary: Of course patients should be told

Heather Goodare 1

Clinical trials have been proclaimed to be good for you on the grounds that patients in trials are “likely to be treated to the highest possible standards”1-1—with the implication that those not in trials are not. If this is true it is worrying. But is it true? Do such patients really do better?1-2 In some cases trials may actually damage your health.

My own experience of a clinical trial, the notorious Bristol study,1-3 was not a happy one,1-4 and I have since observed suboptimal practice in seeking patients' consent for trials, where the benefits are exaggerated and the risks downplayed. In one case the protocol for a multicentre chemotherapy trial had not even been submitted to the local research ethics committee, with unfortunate consequences. The local clinician had strayed from the protocol, omitting the assessment of quality of life, and there were other serious problems. Careful audit by the trial team, as envisaged by Mason and colleagues, would eventually have picked this up.

The authors seem to be more concerned with surgical practice, where skill is of the essence, than with drug treatments. Certainly, patients would want to know if their surgeon was found to have a consistently poor record. But we need to think of other common problems in medical research that statistical monitoring may well uncover. Outliers should not be the only concern. What about deliberate fraud? Statisticians should also be suspicious if one clinician's results are too good to be true, or too similar. There are many examples, such as the returned packs of topical cream in which the tubes had all been squeezed in exactly the same way (FO Wells, Joint Consensus Conference on Misconduct in Biomedical Research, London, 2000).

So if dubious practice is uncovered, should patients be told? The authors distinguish between present and future patients, but some patients survive for a long time after treatment. The Bristol women fought their own battle (one that statisticians should certainly have anticipated as the purported results were wildly improbable). Who is to speak up for truth, honesty, and integrity when trialists veil their errors under a cloak of confidentiality? This will only lead to further mistrust of medical research. Patients and their relatives are more robust than doctors might think. They are used to hearing bad news: but the telling of it has to be done with care and sensitivity.

References

  • 1-1.Tobias J. Cancer: what every patient needs to know. London: Bloomsbury; 1995. pp. 63–72. [Google Scholar]
  • 1-2.Bydder S. It needs to be established whether patients really fare better in trials. BMJ. 2000;321:1530–1531. [PubMed] [Google Scholar]
  • 1-3.Bagenal FS, Easton DF, Harris E, Chilvers CED, McElwain TJ. Survival of patients with breast cancer attending Bristol Cancer Help Centre. Lancet. 1990;336:606–610. doi: 10.1016/0140-6736(90)93402-b. [DOI] [PubMed] [Google Scholar]
  • 1-4.Goodare H, editor. Fighting spirit: the stories of women in the Bristol breast cancer survey. London: Scarlet Press; 1996. [Google Scholar]
BMJ. 2002 Feb 16;324(7334):419–421.

Commentary: The surgeon is only one factor

Tom Treasure 1

Patients deserve safety but it is disingenuous to link trials with risk rather than benefit. Patients within trials have better outcomes2-1 and it is arguable that trials are safer than routine care. Safety requires an amalgam of clinical judgment, attention to detail, commitment, care, technical skill, and the close monitoring of performance. These attributes are as likely to be found among doctors doing research as among those who are not. If we are to recruit patients to trials they must believe (with justification) that we will take good care of them. Monitoring of individual performance will help to ensure that.

There is precedent for trial monitors to be the first to identify poor technical results. In the first of the coronary surgery multicentre trials there was a high mortality in 1970-2, which was noted and documented.2-2 Simply archiving adverse data is not acceptable, but the statistical significance of outliers remains problematic when individual numbers are small. An extreme but real example is a trial of multimodality treatment of superior sulcus lung cancer in which 111 patients were operated on by 76 different thoracic surgeons, a median of one operation per surgeon.2-3 If we were to take the unreasonable step of attributing the outcome to the surgeon alone, most surgeons had 0% mortality but, of course, a few had 100%.

A specific problem taxing us now is recruiting patients into the MRC LU22 trial of induction chemotherapy for operable lung cancer. Persuasive evidence (short of proof) suggests that outcomes would be better if lung resection were preceded by chemotherapy, but there is the fear that the cancer will progress during the delay or that chemotherapy will increase the patient's vulnerability. Nothing but a randomised trial can resolve this issue, but surgeons, ever conscious of perioperative safety, are wary. Personally I would like every patient considered for multimodality cancer treatment to be within a randomised trial because the issues are far too complex and interwoven for any clinician to rely on experience, clinical judgment, or “what works best in my hands.” Without trials we will continue to trade anecdotes and uncontrolled retrospective clinical series that pass for research. We are woefully inadequate in the care of lung cancer2-4 and every operation outside a trial is data lost forever.

When trials involve skilled interventions such as surgery it is appropriate to check the track record of the clinicians taking part against explicit criteria. This has been done in trials of the technically exacting procedures on the carotid arteries,2-5 but clusters of poor outcomes in multimodality cancer trials are not necessarily attributable to the operating surgeon They are as likely to be due to preoperative assessment, anaesthesia, drug errors and side effects, and postoperative care. Surgical competence is part of the whole treatment package and should be documented along with everything else. But we also should remember that for the big three operations (in terms of volume)—cataract surgery, hip replacement, and coronary bypass operations—success relies on their reproducibility in the hands of many competent and committed surgeons—on whom every complication or death weighs heavily. I do not mind whether it is trusts or trialists who monitor standards as long as they do it fairly and competently.

References

  • 2-1.Braunholtz DA, Edwards SJ, Lilford RJ. Are randomized clinical trials good for us (in the short term)? Evidence for a “trial effect.”. J Clin Epidemiol. 2001;54:217–224. doi: 10.1016/s0895-4356(00)00305-x. [DOI] [PubMed] [Google Scholar]
  • 2-2.Hampton JR. Coronary artery bypass grafting for the reduction of mortality: an analysis of the trials. BMJ. 1984;289:1166–1170. doi: 10.1136/bmj.289.6453.1166. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2-3.Rusch VW, Giroux DJ, Kraut MJ, Crowley J, Hazuka M, Johnson D, et al. Induction chemoradiation and surgical resection for non-small cell lung carcinomas of the superior sulcus: Initial results of Southwest Oncology Group Trial 9416 (Intergroup Trial 0160) J Thorac Cardiovasc Surg. 2001;121:472–483. doi: 10.1067/mtc.2001.112465. [DOI] [PubMed] [Google Scholar]
  • 2-4.Does lung cancer need a lapel ribbon? Lancet. 2000;356:1205. [PubMed] [Google Scholar]
  • 2-5.Spence D, Eliasziw M. Endarterectomy or angioplasty for treatment of carotid stenosis. Lancet. 2001;357:1722–1723. doi: 10.1016/S0140-6736(00)04929-1. [DOI] [PubMed] [Google Scholar]

Articles from BMJ : British Medical Journal are provided here courtesy of BMJ Publishing Group

RESOURCES