Prospective Assessment of an Atlas-Based Intervention Combined With Real-Time Software Feedback in Contouring Lymph Node Levels and Organs-at-Risk in the Head and Neck: Quantitative Assessment of Conformance to Expert Delineation

Abstract

Purpose

A number of studies have previously assessed the role of teaching interventions to improve organ-at-risk (OAR) delineation. We present a preliminary study demonstrating the benefit of a combined atlas and real time software based-feedback intervention to aid in contouring of OARs in the head and neck.

Methods and Materials

The study consisted of a baseline evaluation, a real-time feedback intervention, atlas presentation, and a follow-up evaluation. At baseline evaluation, 8 resident observers contoured 26 organs-at-risk on a computed tomography scan without intervention or aid. They then received feedback comparing their contours both statistically and graphically to a set of atlas-based expert contours. Additionally, they received access to an atlas to contour these structures. The resident observers were then asked to contour the same 26 organs-at-risk on a separate computed tomography scan with atlas access. In addition, 6 experts (5 radiation oncologists specializing in the head and neck, and 1 neuroradiologist) contoured the 26 organs-at-risk on both scans. A STAPLE composite of the expert contours was used as a gold-standard set for analysis of organs-at-risk contouring.

Results

Of the 8 resident observers who initially participated in the study, 7 completed both phases of the study. Dice Similarity Coefficients (DSCs) were calculated for each user-drawn structure relative to the expert STAPLE composite for each structure. Mean DSC across all structures increased between Phase 1 and Phase 2 for each resident observer demonstrating a statistically significant improvement in overall OAR-contouring ability (p < 0.01). Additionally, intervention improved contouring in 16/26 delineated organs-at-risk across resident observers at a statistically significant level (p ≤ 0.05), including all otic structures and suprahyoid lymph node levels of the head and neck.

Conclusions

Our data suggest that a combined atlas and real-time feedback-based educational intervention detectably improves contouring of OARs in the head and neck.

Introduction

In order to plan for IMRT, manual segmentation (contouring) of regions of interest (ROIs), either tumors or organs-at-risk (OARs), is performed by physician observers. As these ROIs serve as the input functions for all subsequent planning steps, accurate segmentation, leading to the proper voxel assignment of both tumors and organs-at-risk, is crucial to optimize therapeutic ratio. However, data shows there is a great degree of inter-observer variability in manual ROI segmentation.¹ Both under- and over-contouring of tumors and OARs can have deleterious consequences, leading to local failure and normal tissue sequelae respectively. The importance of accurate manual segmentation and the high demonstrated inter-observer operator-dependence of this process indicate a specific and substantial impediment to execution of multi-institutional clinical trials involving conformal radiotherapy.² Despite the requirement for accurate ROI delineation for radiation therapy treatment planning, instruction in target definition is often based on ad hoc instruction, with limited educational resources provided to many residents.³ Previous cooperative group studies involving practicing physicians suggest that reference to a simple anatomic atlas can substantially standardize and improve conformality of target volumes to an expert reference.⁴ Likewise, Bekelman⁵ and Tai⁶ have demonstrated educational interventions may improve trainee target definition.

Consequently, we sought to investigate the potential gain of a standardized atlas-based, software feedback-assisted intervention to improve head and neck OAR/ROI segmentation, having developed an open source on-line segmentation analysis software.^7-9 The specific aims of the current study were:

1. Estimate potential improvement in OAR/ROI manual segmentation conformance with a multi-expert composite ROI attributable to a combined atlas/visual software-feedback educational intervention.

2. Validate utility of an open-source software solution for execution of said educational study.

3. Hypothesis-generation and sample size estimation for future prospective series.

Materials and Methods

Approval and Compliance

Institutional Review Board approval as an exempt, 45 CFR 46.101(b)(4)-compliant study was obtained, allowing collection of anonymized DICOM files. Clinical datasets were anonymized and stripped of identifiers, and fictionalized case histories were constructed for all cases.

Study Design

This single-arm pilot, prospective feasibility analysis was designed to determine the requisite sample and effect size required for a planned larger atlas-based software-feedback assisted effort. The study was designed as a test-retest sequence, with comparison to a “gold-standard” multi-expert composite ROI (Figure 1).

Figure 1.

Study Design/workflow.

Software Utilization

For this study we utilized TaCTICS (Target Contour Testing/Instructional Computer Software, https://github.com/kalpathy/tacticsRT), which has been presented in detail previously.^7-9 TaCTICS provides a data collection and analysis platform for manual or automated segmentation ROIs. TaCTICS is capable of collecting, displaying, and analyzing ROIs with multiple distinct metrics, and can generate multi-observer probabilistic composite ROIs using Warfield's STAPLE (Simultaneous Truth And Performance Level Estimation)¹⁰ methodology. This feature was used for the current study to create multi-expert estimation of a ground truth “gold-standard” ROIs. TaCTICS was also used to calculate Dice Similarity Coefficients (vide infra) for analysis of individual resident ROIs.

Observer Manual Segmentation/Educational Intervention

Eight resident observers were asked to contour all structures listed in Table 1 on axial CT images obtained from a patient with a head and neck malignancy, using their normal clinical practice (but without referring to any atlas). TaCTICS software^7-9 was used for ROI submission.

Table 1.

List of 26 ROIs/OARs to contour grouped by category.

Lymph Node Levels	Ear Structures
Left Level 1	Left Cochlea
Left Level 2	Right Cochlea
Left Level 3	Left Middle Ear
Left Level 4	Right Middle Ear
Left Level 5	Left Vestibular Apparatus
Right Level 1	Right Vestibular Apparatus
Right Level 2	Salivary Glands
Right Level 3	Left Parotid Gland
Right Level 4	Right Parotid Gland
Right Level 5	Left Sublingual Gland
Retropharyngeal Level	Right Sublingual Gland
Velar/Palatal Structures	Left Submandibular Gland
Lower Lip	Right Submandibular Gland
Upper Lip
Soft Palate

After contouring this case, the resident observers were asked to contour the same structures on a new case after an educational intervention. This intervention consisted of real-time feedback (within 5 minutes), as both numerical and visual DSC scoring of submitted ROIs (Supplemental Figure A). Feedback included axial slice-by-slice ROI comparisons with other submitted user ROIs (Supplemental Figure B), as well as two expert sets of contours defined as a “reference caution” (Supplemental Figure C), and a “reference flag” (Supplemental Figure D). Segmentations outside the “reference caution” indicated an ROI > 0.25 cm outside an atlas-based ROI contour for said OAR or > 0.5 cm outside an atlas-based ROI contour for atlas-based lymph node levels (conceptually equivalent to a clinical trial “minor deviation”). Segmentations outside the “reference flag” indicated ROIs > 0.5 cm outside an atlas-based ROI contour for an OAR, or > 1.0 cm outside atlas-based lymph node levels ROIs (conceptually a “major deviation”).

Simultaneously, observers were provided immediate access to relevant peer-reviewed reference atlases^11-13 as well an in-house reference atlas (courtesy XXXXXXXXX, MD). Upon online submission of the second case ROIs, TaCTICS again provided feedback and metrics as described above.

Five expert head and neck radiation oncology attendings (XXX, XXX, XXX, XX, XXX) and one neuroradiologist (XX) were asked to manually segment the same OAR ROIs for both cases. Using Warfield's STAPLE¹⁰ methodology, a probabilistic estimate of ground truth segmentation of these contours was generated to create idealized “gold-standard” ROIs using TaCTICS. This multi-expert probabilistic composite ROI set was then used for all subsequent comparisons.

TaCTICS was used to calculate the Dice similarity coefficient (DSC) for all resident observer ROIs for all residents prior to the atlas/feedback intervention and after the intervention. The DSC is defined as:

$$D=\frac{2\mid A\cap G\mid }{\mid A\mid +\mid G\mid }$$

where A represents each resident observer OAR ROI and G is the “gold-standard” multi-expert STAPLE contour. The DSC characterizes the intersection of the user with the reference STAPLE while penalizing observers for excessively large contours.¹⁴

Statistics

Statistical analysis was performed using the JMP software package. The Wilcoxon Signed-Rank test was used as a non-parametric measure to determine two outcomes. The first outcome that was assessed was whether the intervention improved a user's ability to contour a particular OAR ROI as measured by DSC as compared to a expert composite STAPLE of the same OAR ROI. The second outcome that was assessed was whether the intervention improved a user's ability to contour the set of all OAR ROIs. A non-Bonferroni-corrected confidence level of a = 0.05 was considered statistically significant for this hypothesis generating pilot study. Pre-study power and sample size calculations using G*Power¹⁵ were performed using a minimum asymptotic relative efficiency¹⁶ of the Wilcoxon Signed Rank test of 0.864 relative to its parametric equivalent, the paired t-test, to ensure an equivalent 1 - b = 0.80 using a large effect size (≥ 0.7) for seven users. Post-hoc evaluation of detected effect sizes for all OAR ROIs, again performed using G*Power¹⁵, demonstrated an evident mean±SD effect size (calculated as Cohen's D) of 0.77±0.32, broadly consistent with pre-study estimates.

Results

Feasibility

Seven resident observers completed both phases (assigned henceforth as Users A-G). All six experts contoured all 26 structures on both cases with two exceptions. One expert omitted contouring the right cochlea while another omitted right Level 2 on the second case. STAPLE composites were generated from all 6 expert contours excepting the two aforementioned OARs for which composites were generated from 5 expert contours. TaCTICS was used to analyze overlap of the structures and obtain DSCs for each resident observer for each structure relative to the STAPLE multi-expert composite ROIs. A summary of the DSCs by structure is seen in Table 2. Notable is the improvement in mean DSC for all resident observers for all OAR ROIs. Of note, in the initial phase, at least one resident observer was unable to contour the majority of the structures with any overlapping voxels relative to the expert composite.

Table 2.

Summary of DSC Across Structures for All Resident Observers

	Mean Resident DSC		Median Resident DSC		Range of Resident DSC		Wilcoxon Signed Rank
Organ-at-Risk	Phase 1	Phase 2	Phase 1	Phase 2	Phase 1	Phase 2	(*p < 0.05)
Lymph Node Levels
Left Level 1	0.38±0.36	0.66±0.22	0.16	0.79	0.00-0.78	0.31-0.82	0.008*
Left Level 2	0.42±0.30	0.64±0.19	0.54	0.69	0.00-0.75	0.28-0.82	0.008*
Left Level 3	0.46±0.31	0.68±0.13	0.36	0.75	0.00-0.78	0.43-0.78	0.039*
Left Level 4	0.39±0.36	0.55±0.17	0.52	0.63	0.00-0.75	0.31-0.75	0.148
Left Level 5	0.42±0.26	0.57±0.14	0.51	0.57	0.00-0.70	0.30-0.72	0.109
Right Level 1	0.40±0.37	0.67±0.21	0.18	0.79	0.00-0.82	0.33-0.83	0.016*
Right Level 2	0.42±0.28	0.63±0.19	0.53	0.68	0.00-0.70	0.26-0.81	0.008*
Right Level 3	0.47±0.31	0.68±0.10	0.38	0.74	0.00-0.80	0.50-0.77	0.109
Right Level 4	0.42±0.37	0.58±0.13	0.63	0.62	0.00-0.77	0.33-0.69	0.289
Right Level 5	0.40±0.27	0.54±0.15	0.53	0.54	0.00-0.66	0.30-0.76	0.055
RP Level	0.27±0.20	0.40±0.17	0.33	0.48	0.00-0.45	0.04-0.52	0.016*
Ear Structures
Left Cochlea	0.18±0.14	0.48±0.36	0.18	0.63	0.00-0.35	0.00-0.82	0.031*
Right Cochlea	0.15±0.12	0.50±0.32	0.16	0.64	0.00-0.30	0.00-0.75	0.031*
Left Middle Ear	0.28±0.28	0.52±0.22	0.15	0.54	0.00-0.67	0.07-0.72	0.039*
Right Middle Ear	0.26±0.26	0.52±0.22	0.11	0.58	0.00-0.64	0.08-0.75	0.039*
Left Vestibular Apparatus	0.02±0.03	0.42±0.32	0.00	0.45	0.00-0.07	0.00-0.81	0.031*
Right Vestibular Apparatus	0.01±0.02	0.50±0.25	0.00	0.58	0.00-0.05	0.00-0.69	0.031*
Velar/Palatal Structures
Lower Lip	0.30±0.10	0.31±0.13	0.26	0.31	0.20-0.50	0.12-0.52	0.469
Upper Lip	0.25±0.28	0.27±0.15	0.10	0.26	0.00-0.70	0.00-0.51	0.422
Soft Palate	0.47±0.29	0.49±0.25	0.63	0.53	0.00-0.76	0.00-0.70	0.344
Salivary Glands
Left Parotid Gland	0.76±0.17	0.85±0.05	0.81	0.84	0.39-0.87	0.77-0.91	0.023*
Right Parotid Gland	0.77±0.09	0.85±0.06	0.81	0.87	0.58-0.84	0.74-0.91	0.016*
Left Sublingual Gland	0.06±0.15	0.28±0.21	0.00	0.38	0.00-0.40	0.00-0.49	0.047*
Right Sublingual Gland	0.07±0.18	0.29±0.17	0.00	0.33	0.00-0.49	0.10-0.57	0.039*
Left Submandibular Gland	0.59±0.40	0.82±0.10	0.82	0.84	0.00-0.86	0.61-0.89	0.055
Right Submandibular Gland	0.64±0.40	0.84±0.08	0.86	0.86	0.00-0.89	0.66-0.91	0.234

Atlas Intervention with Real-Time Feedback Improves Contouring of Multiple Head and Neck OARs

Atlas-introduction with real-time software-based visual feedback demonstrated p < 0.05 for multiple candidate ROIs by Wilcoxon Signed-Rank Analysis (Table 2). Improved resident DSC conformance with expert ROIs was seen for otic structures (bilateral cochleae, bilateral middle ears, bilateral vestibular apparatuses), lymph node levels above the hyoid (bilateral levels 1 and 2 and the retropharyngeal space) as well as bilateral parotid and sublingual glands. Figure 2 illustrates the alteration in an individual resident observer's conformance with expert STAPLE ROIs for lymph node Level 1 after the atlas/software intervention. There was no significant improvement in DSC of the ROIs for the velar and palatal structures.

Figure 2.

Segmentation of Lymph Node Level I by User A relative to Expert STAPLE Composites. Phase 1 represents the initial segmentation, while Phase 2 represents the segmentation after intervention. Cranio-caudal contouring of Lymph Node Level I in Phase 2 is improved after atlas/feedback intervention.

Additionally, the average user conformance with expert STAPLE ROIs across all ROIs improved after the intervention (p = 0.0078). Figure 3 shows the improvement in the mean DSC for each individual user across all 26 contoured structures.

Figure 3.

DSC distribution for Users A through G before and after the intervention, with improvement in mean DSC for each resident observer. (p = 0.0078)

The Benefit of Atlas Intervention May Relate to Level of Training and OAR Size

Post-hoc secondary analysis was performed to assess potential trends in resident observer OAR contouring experience and ROI DSC interval improvement post-intervention. Users A, B and G were “Upper level” third or fourth year radiation oncology residents. Users C and F were incoming PGY-1 residents matched to radiation oncology programs, and Users D and E were in their first two years of residency in radiation oncology, grouped as “Junior Level”. For both junior (p < 0.0001) and upper (p < 0.0001) level resident observers, there was a statistically significant mean DSC improvement in subgroup analysis. (Supplemental Figure E) Figure 4 demonstrates a plot of average change in DSC versus STAPLE composite OAR size from Phase 2 for Junior (Figure 4a, R² = 0.06) and Upper (Figure 4b, R² = 0.33) Level resident observers. This shows that for experienced trainee observers, there is demonstrable gain primarily in the contouring of small-volume OARs while for novice users the gain is seen across all OARs.

Figure 4.

Volume-dependent correlation was seen among upper level resident observers (4b), with significant improvement in low-volume OAR segmentation, while no correlation was noted for junior level observers (4a).

Discussion

The utility of atlas-based and other teaching interventions in improving contouring of ROIs has been addressed by a number of authors showing mixed effects. Our previous work demonstrated improvement in OAR segmentation when using an atlas-based intervention⁴. Here, we perform a prospective feasibility study to evaluate a combination of an atlas-based educational component with a real-time software feedback and visualization assessment, and also to estimate effect size and perform sample size calculation for a future larger scale directed atlas-based, software-assisted longitudinal effort planned with the goal of improving segmentation of difficult to contour organs-at-risk in the head and neck.

Our study demonstrated detectable improvement in overall contouring of OAR ROIs through the use of the aforementioned intervention. Cumulatively, resident observers improved their average DSC across all OAR ROIs demonstrating that through the use of our intervention, an individual's overall contouring of normal OAR ROIs more closely approximated expert observers. In a previous technical paper, we reported such interventions lead to more homogenous contours among trainees.⁸In toto, our intervention appears to improve both contouring uniformity, and accuracy as compared to an expert-derived “gold-standard”.

Our study also demonstrated OAR-specific differentials in contouring improvement with detectable improvement in suprahyoid lymph node level, otic, parotid and sublingual gland ROIs after the intervention. We suspect improvement in contouring of the suprahyoid and retropharyngeal lymph node levels is related to the complex anatomy of the upper neck. Thus, a more precise visual/atlas-based anatomical definition, as obtained from our intervention, is more readily characterized. A similar rationale applies to sub-centimeter otic structures and sublingual glands. Regarding the parotids, improved contouring was likely due to a novice error in neglecting to contour the deep lobe of the parotid gland. There was no improvement in contouring of the ROIs for the velar and palatal structures selected for this study.

Segmentation of head and neck ROIs is notoriously difficult as OARs are particularly small and confined to an anatomically complex region. Bekelman et al.⁵ examined the utility of a teaching intervention in contouring tumor ROIs in the head and neck. 14 residents segmented three CTVs on 6 CT slices of a single base-of-tongue case. The residents then underwent a series of oncology and anatomy seminars, including didactics and a hands-on sessions before recontouring. There was improvement in the node-negative neck ROIs, but difficulty remained in coverage of subclinical disease. These data, in concert with our findings, show training interventions have potential to improve head and neck segmentation.

Our web-based intervention provides an ideal mechanism for low-cost educational and clinical trial implementation. As an open-source online training system, radiation oncology departments need not invest additional educational funding, and clinical trialists may easily implement web-based training/credentialing programs for ROI quality-assurance. During this study, observers downloaded DICOM images into their treatment planning system, exported contours as an RTSTRUCT file from their treatment planning system, and uploaded these files for real-time feedback and analysis. We have since developed an entirely web-based ROI segmentation system to streamline this process. As this entire intervention was web-based, it provides a blueprint for a low-cost mechanism to train and credential future radiation oncologists in the segmentation of head and neck OAR ROIs.

Conclusions

The results of our study demonstrate that a combined atlas-based and real-time feedback intervention was associated with improved contouring of OAR ROIs in the head and neck, as defined by ROI conformance with a multi-expert gold-standard.