Use of a modified Delphi approach to develop research priorities for the Association of Coloproctology of Great Britain and Ireland

Abstract

Aim

The modified Delphi approach is an established method for reaching a consensus opinion among a group of experts in a particular field. We have used this technique to survey the entire membership of the Association of Coloproctology of Great Britain and Ireland (ACPGBI) to reach a consensus on prioritizing clinical research questions in colorectal disease.

Method

Three rounds of surveys were conducted using a web-based tool. In the first, the ACPGBI membership was invited to submit research questions. In Rounds 2 and 3 they were asked to score questions on priority. A steering group analysed the results of each round to identify those questions ranked as being of highest priority.

Results

Five hundred and two questions were submitted in Round 1. Following two rounds of voting and analysis, a list of 25 priority questions was produced, including 15 cancer-related and 10 noncancer-related questions.

Conclusion

It is anticipated that these results will: (i) set the research agenda over the next few years for the study of colorectal disease in the United Kingdom, (ii) promote development and (iii) define funding of new research and prioritize areas of unmet clinical need where the potential clinical impact is greatest.

What does this paper add to the literature?

This paper sets forth the research agenda in colorectal surgery as determined by a modified Delphi process involving a group of experts to identify and prioritize the research questions of greatest clinical importance in current colorectal practice. Its strength lies in the involvement of the entire membership of the Association of Coloproctology of Great Britain and Ireland.

Introduction

Research in surgery may be more challenging than in other disciplines, leading to a relative lack of high-quality randomized controlled trials ^1,2. Some aspects of surgery, such as the learning curve for a new procedure or measurement of technical quality, present specific problems with trial design. There is risk of bias and true clinical equipoise is difficult to achieve. While an individual surgeon may lack certainty about a specific intervention or choice of interventions, the surgical community is likely to include a majority of individuals who may have only have a little uncertainty but who are unable together to agree a common position. This in turn has an impact on trial recruitment.

It is difficult for investigators and funding bodies to judge the potential impact of an innovation, technique or management strategy. A list of colorectal research questions with the greatest potential for improving patient care would aid investigators and funding bodies to address these problems and persuade clinicians to cooperate to resolve uncertainty in these areas.

The modified Delphi approach is an established method for reaching a consensus opinion among a group of experts in a particular field ^3,4. It is an iterative process, commonly used in health and social sciences, to determine the extent to which a group of experts agree on a particular issue. It was used by the American Society of Colon and Rectal Surgeons to establish a research agenda, which was published in 2009 ⁵. Elsewhere in colorectal surgery, the modified Delphi method has been applied to anastomotic leakage ⁶, classification of complications ⁷, development of core outcome sets after colorectal surgery ⁸, quality indicators for colorectal cancer care ^9,10, criteria for resectability in recurrent rectal cancer ¹¹, development of laparoscopic training tools ¹² and assessment of fitness for discharge from hospital after colorectal surgery ¹³.

We have used this technique to survey the entire membership of the Association of Coloproctology of Great Britain and Ireland (ACPGBI) with the aim of reaching a consensus on those research questions that our members, as the expert panel, considered to reflect the clinical issues of greatest importance in treating colorectal pathology. These will then be the focus of future grant applications, funding streams and recruitment drives. They will act as a guide for future investigators, funders and editors alike, allowing justification of project proposals and a guide to the potential clinical impact.

Method

The modified Delphi method involves collecting the individual anonymized opinions of a group of experts and refining these through controlled feedback and statistical analysis in a series of ‘rounds’. We conducted three rounds using a web-based tool (https://www.surveymonkey.net/). In each round, members were invited to participate by e-mail and the survey was open for a period of at least 5 weeks. Three reminder e-mails were sent following the initial invitation for each round. A nine-person steering group was formed to guide the process. It consisted of colorectal surgeons, a trainee surgeon, a clinical trials unit representative, a patient representative, an oncologist and a representative from the National Institute for Health Research (NIHR).

In Round 1, the entire ACPGBI membership was asked to submit research questions that they considered most important to practising ACPGBI members. There was no upper limit to the number of questions that each member could submit and no limit on choice of topic. The steering group then met and collated the questions, categorizing them by consensus into topics and amalgamating the various questions relating to the same fundamental issue into a single question. Care was taken to ensure that the underlying meaning of a question was not altered and no question was removed.

This process resulted in a list of questions for prioritization in Round 2. The ACPGBI membership was invited, as the expert panel, to score each and every research question on a numeric scale. Respondents assigned a score of between one (low priority) and five (high priority) to each question and were not permitted to leave a question unscored to avoid ‘ambush’ voting for just a few specific topics by interested parties.

The mean score, calculated from the percentage of high-priority (five) and low-priority (one) scores was calculated for each question and the questions then ranked accordingly. The steering group discussed the results and agreed by consensus to use the 45 highest scoring questions in Round 3. This cut-off point was chosen as there was a clear reduction in score for questions below 45 and the number was regarded to be manageable for use in Round 3.

ACPGBI members were asked to re-score the 45 questions in Round 3 with the same scoring techniques as used in Round 2. Analysis of the results was again undertaken using a composite scoring technique and questions ranked according to mean score and proportion of high/low scores. The steering group discussed the results and agreed on a final list of 25 questions that were considered to represent the consensus opinion on priority research questions.

Results

In Round 1, 1117 members in all categories (including allied professions and retired members) were invited to participate in the process (Fig.1). One hundred and twenty-eight (11.5%) responded, providing 502 questions with a mean of 3.9 questions per member. This compared favourably with the experience in a similar project conducted by the American Society of Colon and Rectal Surgeons (9.4% and 3.7 questions/member) ⁵. The categories of the submitted questions are shown in Table1. The 215 cancer questions included 32 on chemoradiotherapy, 28 on basic science, 22 on advanced disease, 21 relating to cancer management, 21 on surgical technique, 21 on early rectal cancer, 20 on cancer screening, 17 on quality measures and outcome, 16 on complete pathological response, 9 on follow-up, 5 on diagnostics and 3 on anal cancer. The steering group categorized each question, and amalgamation of entries asking the same fundamental question resulted in a list of 114 questions for Round 2.

Figure 1.

An overview of the modified Delphi process to prioritize research questions for the ACPGBI.

Table 1.

Topics of the questions included in each round

Topic	Submitted by expert panel in Round 1	Collated questions for Round 2 voting	Highest scorers for Round 3 voting	Final list of prioritized research questions
Total	502	114	45	25
Cancer	215	47	30	15
Emergency surgery including diverticular disease	46	12	4	1
Pelvic floor	43	8	1	1
Proctology	43	8	1	–
Surgical technique and perioperative care	33	6	1	1
Inflammatory bowel disease	33	7	2	2
Education/audit/governance	32	9	2	1
Complications of surgery	20	10	4	4
Innovation	18	4	–	–
Endoscopy	12	3	–	–
Miscellaneous	7	0	–	–

For Rounds 2 and 3 an updated membership list (following a database upgrade by ACPGBI) of 918 current, practising members was used. One hundred and thirty-eight members completed the scoring process in Round 2, giving a 15% response rate. The top 45 questions selected for scoring in Round 3 included 28 on cancer topics, 7 on benign disease, 7 related to surgical techniques and perioperative care and 3 related to outcome, research methodology and training.

One hundred and sixty-three members (17.8%) successfully completed the Round 3 survey. Overall 239/918 (26%) ACPGBI members took part in Rounds 2 and/or 3. The top 25 research questions included 15 relating to colorectal cancer (Table2) and 10 relating to topics other than cancer (Table3), including benign disease, surgical technique and clinical governance.

Table 2.

Highest priority list of cancer-related questions

What is the optimal treatment for early rectal cancer? What are the relative roles of endoscopic mucosal resection (EMR), transanal endoscopic microsurgery (TEMS), radiotherapy, chemotherapy and resectional surgery? In cases of early rectal cancer amenable to local excision techniques, are there benefits from additional treatment modalities? What is the best method for predicting complete pathological response to chemoradiotherapy in rectal cancer treated with neoadjuvant chemoradiotherapy prior to surgery? Do these patients require immediate resectional surgery? If not, what is the best strategy for surveillance? What is the optimal treatment for endoscopically removed polyp cancers? When is surgical resection necessary? What is the long-term outcome of polyp cancers treated with polypectomy alone? What are the short- and long-term outcomes after extralevator abdominoperineal excision of rectum (ELAPE)? Is there an oncological gain and is it justified? What biomarkers (including genetic profiling) affect the response to chemoradiotherapy for rectal cancer? Why do some patients develop colorectal cancer metastases? Can early markers of metastatic disease be developed? What is the optimal timing of resection of liver and/or lung metastases from colorectal cancer – before, during or after primary surgery? What is the optimal method of wound closure after abdominoperineal excision of rectum (APER)? In which situations are mesh or plastic reconstruction indicated, and is there a benefit from these techniques? Is there a benefit to preoperative (chemo)radiotherapy in T3 rectal cancer with nonthreatened margins? If so, does it justify any potential additional toxicity? Is chemotherapy better given before or after surgery for locally advanced colon cancer? Or both before and after? Is there a price to cancer survival after treatment for colon, rectal and anal cancer? What is the impact of treatment on quality of life? What level of poor function is justified to avoid a permanent stoma? What is the role of delayed resection of the primary tumour in chemoresponsive metastatic colorectal cancer? What are the optimal methods and intervals for population screening for colorectal cancer? How can uptake of screening be improved? Are there subgroups of the population who are at higher risk and should be screened earlier or at different intervals? Which colorectal adenomas indicate significantly increased risk of future colorectal cancer? What is the optimal screening strategy for these patients? What is the optimal surveillance strategy for patients who have undergone transanal local excision of rectal cancer?

Table 3.

Highest priority list of noncancer-related questions

How can early detection and outcome of anastomotic leakage be improved? Are there any new techniques or approaches that will reduce anastomotic leak rates in colorectal surgery? What is the best method of (1) preventing parastomal hernia formation and (2) repairing parastomal hernia? What are the indications for, and what is the optimal timing of, surgery for Crohn's disease in the era of biological therapy? What are the short- and long-term outcomes of minimally invasive approaches (e.g. percutaneous radiological drainage, laparoscopic washout and drainage) to managing complicated diverticulitis? How can postoperative ileus be reduced? What is the optimal multimodal strategy for managing fistulating perianal Crohn's disease? How does reporting and sharing of surgeon-specific outcomes affect clinical practice? What are the short- and long-term outcomes of laparoscopic ventral mesh rectopexy (VMR) and is the mesh material used important? What are the predictive factors for poor outcome in patients with severe intra-abdominal sepsis? How can the outcome be improved? When should a colorectal anastomosis be defunctioned? Are there predictive factors which would aid decision-making about the need for diversion?

Discussion

The modified Delphi method has produced a list of 25 research questions that can be considered to reflect the clinical matters of greatest importance in treating colorectal disease chosen by consensus of an expert panel of ACPGBI members. This is the first attempt to engage the entire ACPGBI membership to decide on research priorities that may have an impact on their practice. The modified Delphi process has been criticized as it often involves only a small ‘expert panel’ which introduces its own bias into studies. By widening the expert panel to include all members of the ACPGBI, this source of bias was largely avoided and this feature is one of the major strengths of the exercise. Another important advantage is the involvement of nonacademic ACPGBI members as well as academic surgeons. Typically, research proposals and funding body grant reviews tend to be undertaken by academic surgeons with specific research interests. It is hoped that this project has enabled a diverse range of research questions to be submitted for consideration and a realistic consensus on priority according to the potential clinical impact.

It is not surprising that research questions relating to colorectal cancer were predominant in all three rounds, but over a third of the final list constituted questions on noncancer matters. The top three cancer questions all reflected debate about the best management of rectal cancer and polyp cancers in an era when treatment is evolving towards organ preservation, with the important question whether the cancer-specific results will be maintained in the process. The second highest scoring cancer question was almost identical to the experience of the American Society of Colon and Rectal Surgeons 2009 exercise where ‘How can we identify which rectal cancer patients have had complete pathologic response to neoadjuvant therapy and what is the best treatment for them?’ was the top scorer ⁵.

Cancer receives a generous proportion of available research funding, but the authors welcome the relative importance given to benign disease by the ACPGBI membership. Four of the top-scoring noncancer questions related to prevention and treatment of colorectal complications, including anastomotic leakage, parastomal hernia, ileus and intra-abdominal sepsis and a fifth to weighing up the need for a defunctioning stoma. Further questions related to key aspects of multimodal treatment for inflammatory bowel disease and to increasingly conservative approaches to the management of diverticulitis. Early detection of anastomotic leakage is clearly a consistent concern among colorectal surgeons, as it was highlighted as an important area of research in 2013 ¹⁴ and the development of a signature diagnostic test to allow earlier detection of anastomotic leakage with emerging technologies based on detection of local biomarkers is now the subject of a NIHR Colorectal Therapies Healthcare Technology Co-operative project (http://colorectal.htc.nihr.ac.uk/ongoing-projects/earlier-detection-of-anastomotic-leak/).

In the UK, surgeons are now obliged by law to publish their results for anyone to see. The high prioritization of a research question relating to such national reporting reflects the topicality and importance with which this development is regarded by the members of ACBGBI. The inclusion of this question underpins the concern that the reporting of isolated negative outcomes out of context may result in patient harm if clinical practice changes through political pressure to publish individual surgeons’ results.

There are of course limitations to this type of study. The ideal modified Delphi process would have involved all ACPGBI members participating in all rounds. In reality between 11.5% and 17.8% participated and many did not contribute to every round. Despite this the number of respondents was still large, with over 500 questions submitted and over 400 votes registered. There was also a gradual increase in interest from the membership as the project progressed, with a quarter of ACPGBI members contributing to the final two rounds. The wide variety of academic and subspecialty interests of participating ACPGBI members means that the results give a unique and representative view of where the focus of future colorectal research should lie.

The question remains as to how this information should be disseminated and used. Most importantly there is a need to involve patient support groups in developing these research questions and establishing whether the questions resonate with patients as well as clinicians. Funding bodies will be informed of the results and urged to consider prioritization of research addressing these questions. Vignettes will be prepared for the NIHR. ACPGBI members will be encouraged to champion specific research questions with research ideas worked up in brainstorming days. The success or otherwise of this exercise to identify a research strategy for ACPGBI will ultimately be judged by what research is actually being carried out in 5 years’ time.

It is anticipated that these results will set the research agenda for the study of colorectal disease in the UK, promote development and funding of new research and prioritize areas of unmet clinical need where potential clinical impact is greatest.

Author contributions

NF and JT designed and ran the study, analysed the results and wrote the manuscript. AC, IG, BG, LM, JN, AV and JW were involved in the running of the project, formed the steering group and revised the manuscript.