A Systematic Review of Contouring Guidelines in Radiation Oncology: Analysis of Frequency, Methodology, and Delivery of Consensus Recommendations

Abstract

Purpose

Clinical trials have described variation in radiation therapy plan quality, of which contour delineation is a key component, and linked this to inferior patient outcomes. In response, consensus guidelines have been developed to standardize contour delineation. This investigation assesses trends in contouring guidelines and examines the methodologies used to generate and deliver recommendations.

Methods and Materials

We conducted a literature search for contouring guidelines published after 1995. Of 11,124 citations, 332 were identified for full-text review to determine inclusion. We abstracted articles for the intent of the consensus process, key elements of the methodology, and mode of information delivery. A Fisher exact test was used to identify elements that differed among the guidelines generated for clinical trials and routine care.

Results

Overall, 142 guidelines were included, of which 16 (11%) were developed for a clinical trial. There was an increase in guideline publication over time (0 from 1995-1999 vs 65 from 2015- 2019; P = .03), particularly among recommendations for stereotactic radiation and brachytherapy. The most common disease sites were head and neck (24%), gastrointestinal (12%), and gynecologic (12%). Methods used to develop recommendations included literature review (50%) and image-based methods (45%). Panels included a median of 10 physicians (interquartile range, 7-16); 70% of panels represented multidisciplinary expertise. Guidelines developed for a clinical trial were more likely to include an image-based approach, with quantitative analysis of contours submitted by the panel members and to publish a full set of image-based recommendations (P < .005).

Conclusions

This review highlights an increase in consensus contouring recommendations over time. Guidelines focus on disease sites, such as head and neck, with evidence supporting a correlation between treatment planning and patient outcomes, although variation exists in the approach to the consensus process. Elements that may improve guideline acceptance (ie, image-based consensus contour analysis) and usability (ie, inclusion of a full image set) are more common in guidelines developed for clinical trials.

Introduction

Contour delineation is a critical process in treatment planning because it involves outlining tumor (or areas at risk of microscopic disease) as well as nearby organs at risk (OARs) to guide radiation therapy plans that optimize tumor control and reduce radiation toxicity. However, variation in contour delineation among providers is common and can affect the resulting plan quality and patient outcomes.^1–3 Reviews of prospective clinical trials for radiation therapy quality assurance (QA) have shown that variations in target volume delineation can result in increased treatment toxicity and decreased survival.^4–6 The incidence of variation in contour delineation has been evaluated most rigorously in the setting of clinical trials, in which radiation therapy QA processes document protocol deviations; a recent review found that major deviations in target delineation occurred in up to 13% of radiation therapy plans across 5 different trials.⁷

Consensus guidelines with recommendations for contour delineation have emerged in an effort to reduce contour variation.⁸ Several studies have demonstrated that the use of guidelines and contouring atlases can reduce variation in delineation of both target volumes and OARs,^9–12 with additional evidence that these improvements in contour delineation can improve predicted tumor control and normal tissue complication probability.¹¹

Despite their ability to increase the consistency of contour delineation and improve predicted clinical outcomes, consensus guidelines are underused.¹³ Known barriers to their use include unfamiliarity with their existence and difficulty accessing the information when needed.¹³ Additionally, prior studies have shown that recommendations are inconsistent across guidelines developed for the same disease site, thus complicating guideline selection and subsequent use.^1,14–16 Although standards for the development of clinical practice guidelines exist to ensure guideline quality and usability,¹⁷ no such standards exist regarding guidelines for contour delineation, a uniquely image-based clinical skill. To our knowledge, characteristics of consensus contouring guidelines and methodologies have not been described. Thus, we conducted a comprehensive literature review to investigate trends in published recommendations, characterize and review methods used to develop consensus contouring guidelines, and explore how recommendations are distributed and displayed. This will allow us to identify potential barriers to consistent guideline development and dissemination to inform future implementation efforts.

Methods and Materials

Data sources

We conducted a comprehensive literature search of the PubMed, EMBASE, Cochrane Library, Web of Science, and Scopus databases to identify relevant consensus contouring guidelines published between January 1, 1995 and September 3, 2019 (Search Strategy; Text E1).

Data extraction and synthesis

A professional librarian (L.M.B.) performed the initial search, which produced 14,551 results; an additional 20 articles were identified by hand-searching the websites of professional organizations. After duplicates were removed, we identified 11,124 unique citations. One reviewer (D.L.) screened the abstracts of these citations for relevance and identified 332 potentially relevant publications. Three reviewers (D.L., K.L., M.V.S.) independently examined and identified articles for full-text inclusion. The inclusion criteria were as follows: development by a consensus group (defined as ≥2 authors), recommendations regarding contour delineation for radiation therapy included, and full-text publication available in English language after January 1, 1995. We selected this date because it represents the approximate time when intensity modulated radiation therapy (IMRT) became commercially available.¹⁸ Abstracts and guidelines that provided only recommendations for aspects of radiation therapy planning and did not include volume delineation (eg, patient selection, dose, and fractionation) were excluded. Publications were included if 2 reviewers independently agreed that they met the outlined inclusion criteria. Any discrepancies were discussed with the research team, and, if warranted, an additional reviewer determined final inclusion. Of the 332 full-text articles reviewed, 142 met the inclusion criteria. We selected datapoints of interest based on standards for clinical practice guidelines from the Guidelines International Network¹⁷; datapoints collected included year of publication, disease site, endorsing organization, purpose of guideline development (eg, clinical trial QA), inclusion of multidisciplinary panel members (eg, radiation oncologists, medical oncologists, radiologists, and surgeons), type of radiation therapy, methods used to develop recommendations (eg, literature review), and methods used to display recommendations.

Statistical analysis

We used descriptive statistics to analyze guideline characteristics, a 2-sided Mann-Kendall test to assess trends in guideline publication over time, and a Fisher exact test to evaluate associations between guideline components and purpose of guideline development. We performed all statistical calculations using R version 3.6.1 (R Foundation for Statistical Computing, Vienna, Austria). A significance level of α = 0.05 was set for statistical testing. This was adjusted for multiple comparisons using the Bonferroni correction,¹⁹ which resulted in P values <.005 (.05 / 9) being considered significant for the analysis on guideline components.

We performed this systematic review in compliance with the Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA) literature selection protocol (Fig. 1),²⁰ and it has been registered with the International Prospective Register of Systematic Reviews (PROSPERO).²¹ This study did not require institutional review board approval.

Fig. 1.

Study flow-diagram adapted from the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement.

Annotations:

A1. The arrow sizing is inconsistent, would it be possible to fix this?

Results

Trends over time

A total of 142 guidelines published between January 1, 1995 and September 3, 2019 met the inclusion criteria (Fig. 1). A comprehensive list of included articles is provided in Table E1. The complete list of guidelines is also available with interactive hyperlinks on eContour.org. An increase in the publication of consensus guidelines since the advent of 3-dimensional (3D) treatment planning was observed (P = .03); 0 articles were published from 1995 to 1999, 10 from 2000 to 2004, 22 from 2005 to 2009, 45 from 2010 to 2014, and 65 from 2015 to 2019 (Fig. 2). Most guidelines (82%) gave recommendations on contours for conventional external beam treatment (including both 3D conformal [3DCRT] and IMRT techniques); 11% gave recommendations on volume delineation for brachytherapy, 8% on stereotactic radiation, and 2% on proton therapy (Table 1). There was also an increase over time in the number of guidelines with recommendations for brachytherapy and stereotactic body radiation therapy/stereotactic radiosurgery (Fig. 2).

Fig. 2. Publication of guidelines with contour recommendations over time by type of radiation therapy.

External beam radiation therapy (EBRT) includes both 3-dimensional conformal (3DCRT) and intensity modulated radiation therapy (IMRT) techniques.

Table 1.

Summary of characteristics of identified guidelines^*

Characteristic	Overall (n = 142)	H&N (n = 34)	GI (n = 17)	GYN (n = 17)	GU (n = 16)	Breast (n = 12)	Lymphoma (n = 9)	Lung/Thorax (n = 8)	CNS (n = 9)	Sarcoma (n = 6)	Other (n = 14)
Endorsed	87 (61%)	9 (26%)	6 (35%)	14 (82%)	12 (75%)	9 (75%)	9 (100%)	7 (88%)	6 (67%)	4 (67%)	11 (79%)
Developed for trial	15 (11%)	0 (0%)	2 (12%)	5 (29%)	2 (13%)	2 (17%)	2 (22%)	0 (0%)	0 (0%)	0 (0%)	2 (14%)
Multidisciplinary panel	100 (70%)	27 (79%)	12 (71%)	11 (65%)	13 (81%)	8 (67%)	5 (56%)	5 (63%)	7 (78%)	3 (50%)	9 (64%)
Radiologist	56 (39%)	20 (59%)	10 (59%)	7 (41%)	6 (38%)	3 (25%)	0 (0%)	2 (25%)	3 (33%)	0 (0%)	5 (36%)
Surgeon	41 (29%)	13 (38%)	4 (24%)	4 (24%)	7 (44%)	3 (25%)	0 (0%)	1 (13%)	3 (33%)	2 (33%)	4 (29%)
Contoured volume
Target	88 (62%)	18 (53%)	12 (71%)	12 (71%)	9 (56%)	9 (75%)	7 (78%)	6 (75%)	6 (67%)	4 (67%)	5 (36%)
OARs	20 (14%)	10 (29%)	1 (6%)	0 (0%)	1 (6%)	1 (8%)	0 (0%)	0 (0%)	2 (22%)	0 (0%)	5 (36%)
Both	34 (24%)	6 (18%)	4 (24%)	5 (29%)	6 (38%)	2 (17%)	2 (22%)	2 (25%)	1 (11%)	2 (33%)	4 (29%)
Type of RT
EBRT	117 (82%)	33 (97%)	16 (94%)	10 (59%)	15 (96%)	9 (75%)	8 (89%)	8 (100%)	7 (78%)	5 (83%)	6 (43%)
Brachytherapy	16 (11%)	1 (3%)	0 (0%)	7 (41%)	1 (6%)	3 (25%)	0 (0%)	0 (0%)	0 (0%)	1 (17%)	3 (21%)
SBRT/SRS	11 (8%)	0 (0%)	1 (6%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)	1 (13%)	3 (33%)	0 (0%)	6 (43%)
Proton	3 (2%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)	1 (11%)	2 (22%)	1 (11%)	0 (0%)	0 (0%)
Type of scan
CT	90 (63%)	18 (53%)	12 (71%)	7 (41%)	12 (75%)	12 (100%)	9 (100%)	8 (100%)	2 (22%)	2 (33%)	8 (57%)
MR-assisted CT	50 (35%)	16 (47%)	4 (24%)	9 (53%)	4 (25%)	0 (0%)	0 (0%)	0 (0%)	7 (78%)	4 (67%)	6 (43%)
MR-based	15 (11%)	5 (15%)	2 (12%)	3 (18%)	1 (6%)	0 (0%)	0 (0%)	0 (0%)	4 (44%)	0 (0%)	0 (0%)
Publicly accessible	59 (42%)	9 (26%)	11 (65%)	9 (53%)	3 (19%)	4 (33%)	2 (22%)	5 (63%)	3 (33%)	3 (50%)	10 (71%)

Abbreviations: CNS = central nervous system; CT = computed tomography; EBRT = external beam radiation therapy; GI = gastrointestinal; GU = genitourinary; GYN = gynecologic; H&N = head and neck; MR = magnetic resonance; OARs = organs at risk; RT = radiation therapy; SBRT/SRS = stereotactic body radiation therapy/stereotactic radiosurgery.

^*Not all characteristics were mutually exclusive and percentages may not add up to 100%. If a single guideline fell under more than 1 characteristic category, it was counted within each category.

Guideline characteristics

The most common disease sites addressed by recommendations were head and neck (24%), gastrointestinal (12%), gynecologic (12%), and genitourinary (11%) (Table 1, Fig. 3). One guideline (1%) gave recommendations for more than one disease site: head and neck and genitourinary cancer.²² The majority of guidelines addressed delineation for only radiation therapy target volumes (62%), whereas guidelines for both target volumes and OARs (24%) and OARs alone (14%) were less frequent. Five (4%) of the guidelines identified were developed for delineation of normal tissue or OARs without specifying a disease site.^23–27 Most guidelines (90%) gave recommendations for the delineation of volumes on a computed tomography (CT) scan; only 15 guidelines (11%) gave recommendations for magnetic resonance imaging (MRI)-based contouring exclusively (Table 1).

Fig. 3.

Distribution of identified guidelines by disease site of focus.

Consensus group and purpose

Overall, 62% (n = 87) of the guidelines were endorsed by a cooperative group or professional organization, the largest proportion of which were endorsed by national organizations based in the United States (30%). Following those endorsed by US-based organizations, guidelines were most commonly endorsed by organizations from Europe (22%), international organizations (20%), and national organizations from Spain (6%) (Fig. E1). The number of participants involved in the consensus was available in 49% of the guidelines and ranged from 2 to 129, with a median of 10 (interquartile range, 7–16). The majority of participants were radiation oncologists. Most (70%) of the consensus groups were composed of multidisciplinary panelists; they involved a radiologist in 56 (39%) and a surgeon in 41 (29%) of the guidelines. Only 15 (11%) of the guidelines were developed specifically to ensure the quality of contours for a clinical trial (Table 1). Guidelines developed as part of a clinical trial were more likely to use image-based methods (P < .005), use tools to estimate a consensus contour and assess contour variation (P < .005), and provide a complete contoured image set (P < .005) (Table 2).

Table 2.

Comparison of characteristics between clinical trial and nontrial guidelines

Characteristic	Trial (n = 15)	Nontrial (n = 127)	Total	P value
	n (%)	n (%)	n
Endorsed by organization	13 (87%)	74 (58%)	87	.047
Multidisciplinary panel	11 (73%)	89 (70%)	100	1.000
Literature review	3 (20%)	68 (54%)	124	.021
Image-based methods	12 (80%)	52 (41%)	64	.001
STAPLE analysis	6 (40%)	13 (10%)	19	.004
Assessment of contour variation	8 (53%)	17 (13%)	25	<.005
Full image set available	8 (53%)	15 (12%)	23	<.005
Displayed definition table	10 (67%)	68 (54%)	78	.416
Full-text publicly available	9 (60%)	50 (39%)	59	.167

Abbreviation: STAPLE = simultaneous truth and performance level estimation.

Consensus process

Most (92%) of the guidelines described the methods used to reach a consensus on contour recommendations. Without being mutually exclusive, the most common elements of the consensus process were a literature review (50% of guidelines) and the use of image-based methods (45%), in which members of the group contoured on patient imaging to reach a consensus (Fig. 4). Additional consensus methods included group discussions, surveys, and the use of cadavers and other gross specimens. The use of quantitative contour assessment was variable. Reported quantitative metrics are defined and summarized in Figure 4. Simultaneous truth and performance level estimation (STAPLE), an approach to calculating the degree of volume overlap on an image, was used in 13% of guidelines, whereas tools used to assess variation among multiple contours were used in 18% of guidelines.

Fig. 4.

Summary and explanations of methods used to develop guidelines.^40–46

Display of consensus recommendations

The presentation and display of contour recommendations within the publications were variable. Less than half of the guidelines (42%) were publicly accessible (ie, the full text could be accessed without a paid journal subscription). Guidelines often included representative axial imaging with a median of 11 images (range, 0-94), whereas 53% of the guidelines displayed a table summarizing definitions and contouring instructions. A complete atlas or image set for a contoured case outlining the provided recommendations was available for only 16% of the guidelines.

Discussion

This investigation represents the first systematic review of consensus contouring guidelines in radiation oncology. Although guideline use has been shown to enhance the accuracy of radiation treatment, improve clinical outcomes, and reduce toxicity,^10–12 inconsistent recommendations and poor dissemination challenge the utilization of contouring guidelines.¹³ Although standards exist for the development of clinical practice guidelines in other medical fields,¹⁷ contour delineation for radiation therapy lacks such structure. By analyzing the methods used to develop and disseminate available consensus guidelines, with an emphasis on processes undertaken in the setting of clinical trials, this study could potentially serve as a basis for the development of formalized standards for contouring guidelines.

With advances in highly conformal radiation therapy techniques, such as IMRT, stereotactic radiation (stereotactic body radiation therapy/stereotactic radiosurgery), and image guided brachytherapy, increasing the precision of radiation treatment,^8,18 this investigation confirms the concomitant increase in publication of contouring guidelines, as hypothesized in prior studies.^8,13 Only 2% of guidelines addressed proton therapy, and all were published after 2018. This may be explained by proton therapy accounting for less than 1% of radiation treatments delivered in the United States,²⁸ as well as contouring guidelines defining the clinical target volume, which is thought to change minimally in the setting of proton treatment planning. Similarly, historically limited availability of magnetic resonance (MR) simulators and hybrid MR- linear accelerator (linac) systems²⁹ likely resulted in few published MR-based guidelines to date. However, with an emergence of MR-linac use in routine care (and with adaptive planning that can generate an even higher contouring burden), the need for consensus guidelines for MRI-based planning is clear and reportedly ongoing by various MR-linac consortia.

Despite the rapid growth in the publication of contouring guidelines, there are no widely accepted standards for contouring guideline development. Methods used to reach consensus were diverse in form and rigor. Among essential components of high-quality and reproducible guideline development standards established by the Guidelines International Network,¹⁷ only half of the identified guidelines performed a literature review, 70% involved a multidisciplinary panel, and, although the median number of panelists was 10, only 22% of guidelines included the Guidelines International Network recommended number of 10 to 20 panel members.

Contouring guidelines developed as part of a clinical trial protocol often follow a more standardized and democratic process, including routine use of image-based approaches that incorporate quantitative contour analysis. STAPLE has been adopted by the Radiation Therapy Oncology Group, now part of NRG, and is used to create a single contour from multiple expert contours; contours created via STAPLE often are provided in atlases and are used as reference volumes for trials.^30,31 This standardization process may reflect the use of platforms and frameworks, such as the Global Clinical Trials Radiation Therapy Quality Assurance Harmonization Group, to discuss and endorse guidelines developed for clinical trials.^8,32

Guideline delivery correlates with the likelihood of their use.³³ Overall, a minority (42%) of guidelines identified could be accessed without paid journal subscriptions. Moreover, although radiation oncology is increasingly a 3D image-based treatment, only a small subset (16%) of the guidelines included a full case image set, and most of these were in the context of clinical trial QA. These findings highlight real-world barriers to guideline dissemination and may play a role in their poor utilization.^2,13 There is a preference and a need identified among practicing clinicians for accessible, easy-to-use, image-based contouring resources, which are hypothesized to enhance accessibility at the point of care.^13,34

Although the most common disease sites treated with radiation are breast, lung, and prostate cancer (based on recent data publicly released by the Center for Medicare and Medicaid Services, analysis under submission), contouring guidelines were identified most typically for head and neck, gynecologic, and gastrointestinal cancers. For these disease sites, however, more data exist that support a correlation between poor quality radiation and clinical outcomes,^4,5,35 and a recent analysis of an online contouring decision-support reference showed their guidelines to be the more frequently reviewed, demonstrating the real-world demand for contouring guidance in these areas.³⁴ We did find multiple guidelines were sometimes available for the same disease type. This has been investigated recently in breast and anal cancer.^14–16 Methodological inconsistencies, as well as institutional differences, likely contribute to this phenomenon,³⁶ emphasizing the need to (1) optimize multi-institutional, and often multinational, expert involvement, and (2) reduce bias in the consensus process to ensure buy-in from diverse stakeholders.^7,37

Although limited data exist to define a preferred approach to contouring guideline development, those conducted in the context of clinical trial QA are often subjected to formal review by multiple institutions and organizations and may be considered best practice. As such, the following elements should be considered to optimize the quality of consensus guidelines for volume delineation:

• involvement of a multidisciplinary panel (including a radiologist) with formal voting from members, either via survey or actively contouring an example case

• inclusion of a literature review with consideration of publications on patterns of recurrence as well as clinical trials with acceptable outcomes (and subsequent review of protocol specifications)

• quantitative contour analysis, such as STAPLE, the kappa statistic, and the Dice similarity coefficient, to assess contour variability and estimate consensus contours

• dissemination of a complete reference image set to improve point-of-care usability

However, we recognize the unique nature of contouring and recognize that additional adjustments for disease site and institutional feasibility may be necessary and appropriate. For example, radiologist expertise may be more useful in head and neck cancer—which involves complex anatomy and fused MRI^38–39—than in breast cancer.

Several limitations of the present study exist. First, it is possible that some consensus contouring guidelines were not indexed within the searched databases; some may have been available only on professional-organization websites or within unpublished clinical trial protocols. This practice may be more pragmatic, but it omits journal peer review, which is an important component of this process, particularly regarding consensus methodology. Second, we excluded guidelines that were not published in English, even if they would have otherwise met inclusion. As a result, we may have identified a larger proportion of guidelines from English-speaking countries, potentially skewing the results of the geographic analysis for endorsed guidelines (Fig. E1). Third, although we observed some inconsistencies among recommendations in guidelines developed for the same disease site while we indexed data points, we did not conduct a formal or complete comparison of guideline recommendations. Our ability to comment on differences in recommendations within disease site groups is thus limited. Finally, the availability and reporting of some data points (eg, expertise of panel members) varied among the identified guidelines, thus limiting the analysis. Although precautions were taken to limit discrepancies by selecting straightforward endpoints, reviewing data abstraction techniques as a team, and discussing any uncertainties with additional reviewers, it is possible that certain results of our review may not be completely reproducible.

Conclusions

This systematic review highlights trends in consensus contouring guideline publications and summarizes the methodology and components included in guidelines on contour delineation. With the increase in radiation complexity, guideline publication has increased over time and most commonly focuses on disease sites regarded as difficult to contour. Although guidelines developed for clinical trial protocols were more likely to include image-based methods, quantitative assessments, consensus contour estimations, and distribution of a complete reference-imaging data set, these components were often lacking in the overall population of published guidelines. Although recommendations based on processes used in the setting of clinical trials are provided here, there is an opportunity to develop more formal consensus contouring guideline standards to enhance their validity, dissemination, and usability.