An in-silico quality assurance study of contouring target volumes in thoracic tumors within a cooperative group setting

Graphical abstract

Highlights

• •We aimed at quantifying inter-observer Pancoast tumors delineation variability.
• •Experts’ delineations were used to define ground truth.
• •Other observers’ delineations were compared against ground truth.
• •High degree of variability was noted for most target volumes except GTV_P.
• •This unveils potentials for protocol modification for future IMRT studies.

Abstract

Introduction

Target delineation variability is a significant technical impediment in multi-institutional trials which employ intensity modulated radiotherapy (IMRT), as there is a real potential for clinically meaningful variances that can impact the outcomes in clinical trials. The goal of this study is to determine the variability of target delineation among participants from different institutions as part of Southwest Oncology Group (SWOG) Radiotherapy Committee’s multi-institutional in-silico quality assurance study in patients with Pancoast tumors as a “dry run” for trial implementation.

Methods

CT simulation scans were acquired from four patients with Pancoast tumor. Two patients had simulation 4D-CT and FDG-FDG PET-CT while two patients had 3D-CT and FDG-FDG PET-CT. Seventeen SWOG-affiliated physicians independently delineated target volumes defined as gross primary and nodal tumor volumes (GTV_P & GTV_N), clinical target volume (CTV), and planning target volume (PTV).

Six board-certified thoracic radiation oncologists were designated as the ‘Experts’ for this study. Their delineations were used to create a simultaneous truth and performance level estimation (STAPLE) contours using ADMIRE software (Elekta AB, Sweden 2017). Individual participants’ contours were then compared with Experts’ STAPLE contours.

Results

When compared to the Experts’ STAPLE, GTV_P had the best agreement among all participants, while GTV_N showed the lowest agreement among all participants. There were no statistically significant differences in all studied parameters for all TVs for cases with 4D-CT versus cases with 3D-CT simulation scans.

Conclusions

High degree of inter-observer variation was noted for all target volume except for GTV_P, unveiling potentials for protocol modification for subsequent clinically meaningful improvement in target definition. Various similarity indices exist that can be used to guide multi-institutional radiotherapy delineation QA credentialing.

1. Introduction

The advent of conformal radiotherapy, as well as the developing paradigm of image-guided radiotherapy, affords delivery of tumoricidal radiation doses to user-defined target volumes while minimizing dose to proximal organs at risk (OARs). Nonetheless, the steep dosimetric gradients involved mean that even minor geometric uncertainties may result in substantial dose variation which may in turn reduce delivered dose to the tumor and/or increase exposure to radiosensitive tissues [1], [2].

In an effort to ameliorate toxicity, improve clinical outcomes and increase the overall therapeutic ratio, incorporation of multimodality imaging inputs in the target volume delineation process has become normative for specific organ sites. The addition of multimodality imaging parameters, such as fluorodeoxyglucose positron emission tomography fused with computed tomography (FDG PET-CT) [3], and magnetic resonance imaging (MRI), has been demonstrated in several series to improve reproducibility of target volume definition among users in selected anatomic applications [4], [5]. Additionally, temporal imaging techniques, like 4D-CT or cine-MR have increasingly been explored as a mechanism to accurately localize the target and OARs, and/or reduction of geometric margins required to ensure adequate dose prescription (i.e. planning target volume ‘PTV’ reduction) [6].

Cooperative group studies that plan to implement these novel forms of image-guided radiotherapy (IGRT) in protocols necessitate consideration of target volume variability as a function of quality control [7]. Even minor geometric variation in target volumes may impact local control outcomes. Additionally, it is known that clinical trial radiation deviation may result in measurable decrement in clinical outcomes [8]. Furthermore, while medical physics quality assurance parameters face strict scrutiny, multi-center clinical trials may enroll patients from diverse radiation treatment centers. Radiation oncologists may vary regarding degree of expertise in target delineation generally, unknown acumen in target delineation within the anatomic region of interest, and/or unknown standard practices of incorporating either functional imaging (FDG PET, FDG PET-CT) or temporally-indexed imaging (4D-CT) into target definition⁹. Even when the aforementioned clinical practices are known, the practical consistency by a given center may still vary.

Efforts are increasingly made to formalize target volume strategies (e.g. the recent advent of standardized nodal atlases) [10], [11], and educate radiation oncologists regarding specific expert-derived skillsets (e.g. ASTRO IMRT guidelines and symposia) [12]. Nonetheless, there remains scant data regarding how to optimize clinical trial protocol implementation of multimodality functional imaging combined with temporal information (e.g. 4D-CT) [13]. This represents an unmet need given the fact that 4D-CT is currently the standard-of-care for RT planning of potentially curative lung cancers [14]. Furthermore, a recent systematic review by Frood et al. [15] shed some light on the discrepancies in target volume definition using 4D FDG PET-CT versus 3D FDG PET-CT.

Moreover, quality assurance studies using computer-estimated consensus contours have been tested in multiple cancer sites [16], [17], [18], however -to our knowledge- not specifically in Pancoast tumors. Thus, in addition to testing the feasibility of computer-generated consensus contouring with multimodality functional and/or temporal imaging, we sought to assess inter-observer variability in the setting of Pancoast tumor contouring. Guideline adherence may theoretically reduce the risk of radiation-induced toxicities such as brachial plexopathy [19] or pneumonitis [20].

Consequently, the Southwest Oncology Group (SWOG) Radiation Treatment Committee designed this prospective in-silico quality assurance pilot study to determine how diverse institutions participating in clinical trials are incorporating FDG PET-CT, 3D-, and 4D-CT data in RT planning. Additionally, we sought to provide insight into how such integration might be optimized, standardized, and scrutinized by central review boards, like the IROC Rhode Island (formerly QARC).

The specific aims of this study are to:

• 1.Determine the feasibility of multi-site electronic data collection methods for evaluation of target delineation and user survey.
• 2.Summarize the effect of 4D-CT incorporation upon target volume delineation as assessed by custom target delineation evaluation software.
• 3.Determine specific criteria for target volume “credentialing” in trials where IMRT is to be implemented.
• 4.Generate pilot data and testable hypotheses for future research efforts.

2. Material and methods

This study was approved by the SWOG Radiation Therapy Committee for execution [21], and started accruing radiation oncologists in spring 2012. The study was designed in a multi-user fashion combining objective evaluation of specific target delineation and dose parameters with user reported data. Pancoast tumors were selected as test cases for this concept study; given the relatively strong agreement on target and OARs delineation among radiation oncologists [22], [23]. Radiotherapy guidelines were drafted and multimodality simulation images were acquired as a part of a then-existing SWOG Pancoast tumor clinical trial protocol.

2.1. Data

2.1.1. Scans and simulation guidelines

Non-contrast-enhanced CT simulation scans–both bone and soft tissue windows- were acquired from four patients with Pancoast tumors from a single center. These DICOM (Digital Imaging and Communications in Medicine) files belonged to actual patients treated with radiotherapy (RT), albeit without any Health Insurance Portability and Accountability Act of 1996 (HIPPA)-defined patient-specific information [24]. This was performed by anonymizing patient DICOM datasets using ‘DCMAnonymize’ DICOM Validation Toolkit (https://www.dvtk. org). This software tool removes all patient specific data (e.g. name, ID codes, date of scan, gender) from DICOM file headers.

Two patients had simulation 4D-CT and 18-FDG-FDG PET-CT scans while two patients had simulation 3D-CT and FDG-FDG PET-CT scans. The 4D-CT scans were contoured using an average CT that was generated by averaging 4D CT acquisitions over ten phases of the full breathing cycle. These scans were made available on the IROC Rhode Island (formerly QARC), website at www.QARC.org for participants to download scans, instructions and participation survey. IROC Rhode Island provides radiation therapy, diagnostic imaging and data management services for SWOG among other National Clinical Trials Network (NCTN) institutes [25] (Fig. 1).

Fig. 1.

SWOG in-silico study tasks completed via the IROC Rhode Island platform.

2.1.2. Summary of thoracic radiotherapy

All contouring efforts for this study were undertaken on participants’ treatment planning system (TPS) of choice, provided the capacity for DICOM-RT export was present.

Participants were supplied with relevant imaging and clinical evaluation information to help them define target volumes precisely. Additionally, a priori definitions for target volumes, organs at risk, dose prescription and constraints were specifically provided as a part of broader consensus guidelines. For example, clinical target volume (CTV) for radiation fields was defined as the primary tumor, plus involved mediastinal, ipsilateral paratracheal lymph nodes (levels 2 and 4), and ipsilateral supraclavicular nodes. The total planned dose was set to be 54 Gy prescribed to an isodose line that encompassed the planning target volume (PTV) and that satisfies the dose uniformity guidelines. Participants were tasked to create an intensity-modulated RT (IMRT) plan for a treatment course that was to be delivered in 30, once daily fractions of 1.8 Gy each, to a total dose of 54 Gy to the PTV. The percent of normal lung volume receiving 20 Gy or more (V20) must be kept at less than 37%. 54 Gy was chosen in anticipation of testing this dose in a future prospective, cooperative group combined-modality clinical trial for Pancoast tumors.

2.1.3. Target volume definition

To that end, the definitions of tumor and target volumes as well as guidelines for delineation were clearly provided for participants. These were in accordance with International Commission on Radiation Units and Measurements (ICRU) Reports 50 and 62 in order to mitigate sources of delineation congruency among participants [26], [27]. The instructions included standardized nomenclature of target volumes, hence facilitating subsequent analysis in a time-efficient manner. Tumor volumes encompassed gross tumor volume (GTV) either of the primary tumor (GTV_P) or the clinically positive lymph nodes (GTV_N). Other target volumes included: clinical target volume (CTV), planning target volume (PTV), and internal target volume (ITV). Detailed descriptions are depicted in Table 1. Of note, the fourth set of scans belonged to a patient with no nodal disease, hence no GTV_N delineation was required.

Table 1.

Definitions and specified designations for target volumes of interest: instructions for participants.

Target volume	Definition	Specified designation
Gross tumor volume (GTV)	The primary tumor (GTV_P) and clinically positive lymph nodes (GTV_N) seen on the pretreatment PET scan (standardized uptake value; SUV > 3), diagnostic CT scan, and/or treatment planning CT (>1 cm short axis diameter) will comprise the GTV. The GTV will always be located in the apex of the ipsilateral lung in this particular study. This volume(s) may be disjointed.	GTV_P GTV_N
Internal target volume (if used)	The ITV includes the envelope that encompasses the tumor motion for a complete respiratory cycle.	ITV
Clinical target volume	The CTV is defined to be the GTV plus a 0.5 cm to 1 cm margin as appropriate to account for microscopic tumor extension. The ipsilateral paratracheal lymph nodes (levels 2 and 4) and supraclavicular fossa lymph nodes will be defined as comprising part of the CTV for this protocol. If an ITV approach is used then the ITV plus 0.5 cm to 1 cm is added to the ITV to form the CTV. Elective treatment of the entire mediastinum will not be done. If a uniform margin is used for CTV expansion, please specify CTV_xxMM, where xx = the margin expansion in millimeters, as structure name (e.g. CTV_05MM denotes a 5 mm expansion margin); if a non-uniform margin is used, please designate the volume CTV.	CTV
Planning target volume	The PTV margin should account for setup uncertainties and may be individualized. The PTV will comprise the CTV with a minimum 0.5 cm (if daily imaging correction will be used), or a minimum 1.0 cm if daily imaging will not be performed. If a uniform margin is used for PTV expansion, please specify PTV_xxMM, where xx = the margin expansion in millimeters, as structure name (e.g. PTV_05MM denotes a 5 mm expansion margin); if a non-uniform margin is used, please designate the volume PTV.	PTV_xxMM

2.1.4. Prescribed doses and fractionation guidelines

The total planned dose was set to be 54 Gy prescribed to an isodose line that encompassed the PTV and that satisfied the dose uniformity guidelines. The 54 Gy dose was based upon SWOG recommendations to incorporate higher doses than the classic 45 Gy (per SWOG-9416 and SWOG-0220) for Pancoast trials [28], [29], [30]. This was to be planned as thirty, once daily fractions of 1.8 Gy. The entire PTV was to receive at least 93% of the protocol dose and a contiguous volume of no>2 cc within the PTV was not to exceed 120% of the protocol dose [31].

2.2. Participants

Seventeen SWOG-affiliated radiation oncology participants (physicians), with median career experience of 11 years (IQR: 2.5–18.75), independently delineated treatment target volumes as well as OARs. Each participant used his/her preferred TPS for target delineation. Moreover, six board-certified thoracic radiation oncologists were designated as the ‘Experts’ for this study. Selection criteria for ‘Experts’ included all the following: (1) board certification in radiation oncology; (2) a minimum of 10 years of practice; Thoracic radiation oncology subspecialty. After plan completion, each participant, identified by a specific ID and login, submitted the plan via the internet using FTP transfer (www.qarc.org). Recorded information extracted from the contouring session were then available for central analysis.

2.3. Data submission and curation

Target, plan, and dose data were collected centrally via the QARC platform. Analysis was performed using quantitative software-assisted analysis of contour, dose and imaging data. Each participant was asked to fill out an on-line survey assessing subjective responses to contouring tasks. This survey was made available via QARC website. It included categorical Likert scale evaluation of institutional practices regarding FDG PET/CT and 4DCT incorporation into target delineation, as well as subjective responses regarding the implemented target delineation instruction materials, TPS used, applied software tool for FDG PET-CT registration and 4D-CT analysis, among others (Supplementary Material 1).

After data collection was completed, plan suitability analysis was performed. Designated Experts evaluated each contouring session in order to determine if the plan was clinically acceptable without modification, or whether it represented a major or minor deviation from expert-determined acceptability, using criteria as specified in the initial protocol.

2.4. Imaging analytics and evaluation metrics

The six designated Experts’ contours were incorporated into the final definition of target volumes to serve as comparators for subsequent analysis. This expert composite, i.e. ground truth, was constructed using Warfield’s Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm, a feature embedded in Elekta ABAS (Atlas-based auto-segmentation software) (http://www.elekta.com/ABAS). Segmentation simply implies “classifying” whether each image voxel belongs to the volume of interest or the background. The STAPLE algorithm allows for multiple segmentations fusion by automatically estimating the segmentation quality (as characterized by the sensitivity and specificity parameters) of each classifier while simultaneously deriving a weighted combination of multiple classifiers [32]. This approach provides a probabilistic estimate of the true delineation and a measure of the performance level represented by each expert user (Fig. 2A).

Fig. 2.

(A) Defining ‘ground truth’ and (B) comparing individual contours against ‘ground truth’ (The green volume in the center of each figure represents the Experts’ composite). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Contours by each of 17 participants were then compared with the experts’ STAPLE composite contour, i.e. ground truth (Fig. 2B). The following evaluation metrics were used to assess delineations similarity:

• A.Dice Similarity Coefficient [33]: a spatial overlap index and a reproducibility validation metric. The value of a DSC ranges from 0, indicating no spatial overlap to 1, indicating complete overlap between two sets of binary segmentation results [34] (Fig. 3A),
• B.Hausdorff metric: gives the mean (Mean Surface Distance) or the largest length (Maximum Hausdorff Distance) out of the set of all distances between each point of a set (individual contour) to the closest point of a second set (expert composite) [35] (Fig. 3B),
• C.Volume Overlap Ratio (VOR, or the Jaccard similarity coefficient) was computed to represent the ratio of the volume of intersection to the volume of union per target volume (TV) per case [36] (Fig. 3C). All three aforementioned metrics were calculated using a commercially available image registration software (VelocityAI™ 3.0.1)
• D.Intraclass correlation coefficient (ICC) was calculated as a measure of concordance of clustered segmentations between participants using IBM SPSS Statistics 22.0. The ICC is basically a signal-to-noise ratio, where higher values close to 1 indicate higher concordance between volumes of the same group [37].

Fig. 3.

Evaluation metrics for assessment of delineations similarity; (A) Dice similarity coefficient, (B) Hausdorff distance (HD), and (C) Volume overlap ratio. (FN: false negative; TP: true positive; FP: false positive; TN: true negative).

2.5. Statistical analysis

Based on estimates from our previous pilot series [7], power and sample size analysis (G*Power 3 statistical software) [38] was performed, assuming a minimum possible asymptotic relative efficiency of ≥0.86423, using an a priori power goal 1-β of 0.8, non-Bonferroni corrected one-tailed α = 0.5, for detection of an effect size of 0.8 (large effect). This resulted in a minimum requisite sample size of at least 16 participants, hence the inclusion of 17 radiation oncologists.

Statistical assessment was performed using JMP v 11Pro (SAS institute, Cary, NC). One-way analysis of variance (ANOVA) was applied to assess whether there were any statistically significant differences among the mean values of evaluation metrics per target volume. Additionally, the differential impact of incorporating 3D-CT versus 4D-CT on delineation accuracy was assessed using the ANOVA test.

3. Results

Seventeen SWOG-affiliated physicians, with median career experience of 11 years (IQR: 2.5-18.75) participated in this study. Of note, four out of 17 participants didn’t submit the questionnaires.

In this paper, we’re reporting target volume variation, namely: GTV_P, GTV_N, CTV, and PTV in comparison to reference expert composites. We excluded ‘ITV’ from our analysis as the vast majority of participants didn’t assign them. More importantly, there were not enough expert ‘ITV’ contours per case to feed into the STAPLE algorithm to define the ‘ground truth’. Almost all other target volumes in all 4 cases were not unanimously retrievable from QARC database or contoured by 17 participants (range: 9–17 contours/target volume/case).

Moreover, GTV_N contours showed some inter-observer variability of included thoracic nodal stations. One example was GTV_N in case #1. Out of 12 available delineations of GTV_N: 2 participants delineated (1L) nodal station, 5 participants included stations (3A) and (6), while the rest included all 3 stations. All contours were included in the analysis as long as there were sufficient expert contours to serve as an input to the STAPLE algorithm to define the ground truth.

For each case, all 4 evaluation metrics mentioned in the ‘Methods’ section were derived by comparing individual participants’ target volumes to the corresponding Experts’ STAPLE composite. Median values and corresponding interquartile ranges are also provided in Table 2. When compared to Experts’ STAPLE, GTV_P had the best agreement among all participants with median DSC of 0.87. On the contrary, GTV_N demonstrated the lowest agreement with median DSC of 0.35 as shown in Fig. 4. Likewise, GTV_P demonstrated the lowest median HD & MSD values; 12.6 mm and 0.37 mm, respectively. Whereas, GTV_N showed the largest median HD and MSD values; 68.27 and 17.5, respectively (Fig. 5, Fig. 6).

Table 2.

Median values of individual metrics of all-participants’ delineations across all cases; including interquartile ranges (DSC = Dice coefficient; HD = maximum Hausdorff distance; MSD = Mean Surface Distance; GTV_P = gross primary tumor volume; GTV_N = gross nodal tumor volume; CTV = clinical target volume; PTV = planning target volume; mm = millimeters; IQR = Inter-Quartile Range).

	GTV_P	GTV_N	CTV	PTV
Number of participants
Case 1	13	12	16	17
Case 2	14	9	16	16
Case 3	15	13	15	15
Case 4	16	0	16	16
DSC median (IQR)	0.87 (0.78–0.94)	0.35 (0.10–0.46)	0.80 (0.72–0.90)	0.81 (0.70–0.90)
HD median in mm (IQR)	12.6 (10.09–17.75)	68.27 (41.20–80.31)	43.73 (23.48–64.90)	34.22 (24.80–59.18)
MSD median in mm (IQR)	0.37 (0.15–0.81)	17.5 (8.5–25.45)	2.03 (0.40–4.31)	1.56 (0.64–3.98)
VOR (%)
Case 1	21.9	7.1*	14	13
Case 2	67.1	0	46.4	48.1
Case 3	41	0	23.8	28
Case 4	7.2	N/A**	4.4	9.9

^*Given the inter-observer variability in selection of included lymph node stations, this VOR was calculated based on the volumes segmented by group of participants with most harmonious approach (6 out of 12).

^**N/A: Not Applicable because this scan belongs to a patient with no nodal disease.

Fig. 4.

One-way analysis of Variance of Dice similarity coefficient across target volumes. (Green diamonds represent mean and standard deviation; and red boxes encompass interquartile ranges with the transverse line representing median value). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 5.

One-way analysis of Variance of maximum Hausdorff distance across target volumes. (Green diamonds represent mean and standard deviation; and red boxes encompass interquartile ranges with the transverse line representing median value). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 6.

One-way analysis of Variance of mean surface distance (MSD) across target volumes. (Green diamonds represent mean and standard deviation; and red boxes encompass interquartile ranges with the transverse line representing median value). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

In between lies PTV and CTV which were associated with the second and third best median DSC, respectively (Fig. 4). Similarly, the second and third highest HD and MSD were reported with CTV and PTV as tabulated in Table 2 and shown in Fig. 5, Fig. 6. The median VOR for all TVs in the included 4 cases was 0.14 (range 0–0.67). Along the same lines, the VOR was consistently highest for GTV_P and lowest for GTV_N across all 4 cases. Given the inter-observer variability in selection of included lymph node stations, the GTV_N VOR was calculated based on the volumes segmented by group of participants with most harmonious approach (6 out of 12). Intraclass correlation coefficients (ICC) for DSC, HD, MSD metrics are also reported in Table 2.

Interestingly, there were no statistically significant differences in all studied metrics for all TVs for cases with 4D-CT (i.e. contoured on average CT) versus cases with 3D-CT simulation scans (Fig. 7). Table 2 depicts the number of participants’ delineations for each of the studied TVs as well as medians and interquartile ranges (IQRs) of all individual metrics derived from all cases.

Fig. 7.

One-way analysis of Variance of maximum Hausdorff distance by simulation imaging modality. (Green diamonds represent mean and standard deviation). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

4. Discussion

Target delineation is a key element in modern radiotherapy treatment planning, particularly following the introduction of intensity modulated radiotherapy (IMRT)[39]. Accurate target identification has become as important as beam delivery to ensure tumor coverage, and reduce unnecessary normal tissue exposure [40]. Despite efforts to provide guidelines for more reproducible target delineation across different institutes [14], [16], [41], inter-observer variations are still a barrier that faces multi-institutional trials [2]. Another barrier is the absence of an optimized protocol to account for different imaging modalities when it comes to RT planning for Pancoast tumors [22]. Consequently, our group sought to quantitatively determine the inter-observer variability of expert radiotherapy target-volume delineation for Pancoast tumors, towards developing an expert-consensus contouring atlas.

Per our study, we have shown that the best agreement was in GTV_P across all participants when compared to experts’ STAPLE with a median Dice coefficient of 0.873 (range 0.781–0.935). Also, we showed that the lowest agreement was in GTV_N with a median DSC of 0.347 (range 0.095–0.458). No statistically significant difference was shown between different imaging simulation scans (i.e. 4D-CT and 3D-CT).

Some limitations faced our study including long time taken by experts and other participants to complete the needed contours (an average of 18 months). The 4D-CT scans were contoured using only a single static time point (i.e. different phases of respiration were not accounted for), which may explain why a statistically significant difference was not detected between 3D-CT and 4D-CT simulation studies. Of note, participants used FDG PET images to define tumor volumes for all four cases. Consequently, investigating the value of FDG PET fusion in improving or standardizing target volumes delineation was not feasible. The selection of participants was limited to SWOG institutions which might represent a selection bias. Some target volumes contours were not included in the analysis of all 4 cases. This was attributed to either available contours being not retrievable from IROC Rhode Island database or contours not originally made by all 17 participants (range: 9–17 contours available for analysis per target volume for each case).

The instruction set assigned GTV_N to the clinically positive lymph nodes seen on pre-treatment PET scan (SUV > 3), diagnostic CT scan, and/or treatment planning CT (>1 cm short axis diameter). However, wide inter-observer variability in GTV_N contours was observed. This is chiefly attributed to neither enumerating affected thoracic nodal stations in the instruction set nor suggesting standardized reference lymph node mapping atlas; another two areas for improvement for future studies [11], [42]. A similar study by Mercieca et al also named interpretational differences among observers as the leading cause for large variations in GTV_N delineation [43]. For example, in case #2, among the 9 participants who delineated GTV_N, only 3 participants included the same nodal stations. On the other hand, the remaining 6 participants included partly or completely different nodal stations in 5 different permutations. This significantly affected the interpretability of the overlapping/similarity metrics we selected for evaluating inter-observer delineation agreement. Similarly, GTV_N VOR was calculated as zero for cases #2 and #3.

To avoid this pitfall in case #1, GTV_N VOR was calculated based on volumes segmented by group of participants with most harmonious choice of nodal stations (6 out of 12). Also, it was difficult to interpret DSC for GTV_N given the small size of lymph nodes and the multiple non-contiguous objects within one VOI, i.e. one GTV_N volume may encompass multiple affected lymph nodes. Hausdorff distance may represent an appropriate alternative in such a situation where the probability of overlap is small [44].

Nonetheless, this study represents one of the earliest multi-institutional efforts to automate target volume delineation quality assurance for Pancoast tumors. To that end, fully anonymized inter-institutional data sharing was optimized via the IROC Rhode Island platform. Additionally, we provided all the participating radiation oncologists with contouring guidelines, target volumes definitions and standardized nomenclatures.

Our work follows previous SWOG efforts made to address inter-observer variability across different organs sites [9]. In previous work, we showed that educational intervention with a SWOG-approved consensus atlas reduced inter-observer variability in rectal cancer target volume delineation [7]. However, that study was performed using a CT-only dataset. The goal of the present study was quantitative assessment of variation in target volumes delineation for Pancoast tumors in a clinical trial group setting.

Previous work by other groups suggested that atlas and real time feedback hugely improve OARs delineation in head and neck cancers [45]. Similar results were reported by the Australasian Gastrointestinal Trials Group (AGITG), showing that planning guidelines could be significantly improved and help radiation oncologists optimize IMRT delivery in anal cancer [41]. Another study was also conducted to assess atlas implementation effect on anorectal target delineation, and also showed that use of atlas and contouring guidelines do reduce inter-observer variability [46].

The authors of this study are planning to expand these pilot data into a protocol designed to serve as a template for future SWOG studies involving radiotherapy for Pancoast tumors. By providing clear instructions, well-defined standardized criteria and subsequent calibration tool for target delineation, future multi-institutional studies could execute credentialing/QA efforts. That would invariably facilitate the task of central review boards for multi-institutional clinical trials in a timely- and cost-effective manner. These results can also be integrated into development of educational tools for residency training programs or continued medical education programs to help clinicians improve their target delineation skillset [47].

5. Conclusions

In conclusion, a relatively high degree of inter-observer variation was noted for all target volumes except for GTV_P, revealing potentials for future protocol modifications to improve the accuracy, and reduce the variability of target volume definitions. This study helps explain the variance that can be seen between radiation oncologists treating the same patient and thus provides a tool to better understand and assess the acceptable range of treatment volumes satisfactory to meet QA standards.

6. Link to research Data

https://www.qarc.org/

7. Co-authors specific contributions

All listed co-authors performed the following:

• 1.Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work;
• 2.Drafting the work or revising it critically for important intellectual content
• 3.Final approval of the version to be published
• 4.Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Specific additional individual cooperative effort contributions to study/manuscript design/execution/interpretation, in addition to all criteria above are listed as follows:

• •HE – Manuscript writing, statistical analysis, imaging data curation, and segmentation analysis
• •BE, TL – Manuscript writing
• •ASRM – Statistical analysis, and segmentation analysis
• •RD,TJF, FL, KU, JKC, TG – data management and curation, platform support through IROC, and oversight of segmentation analysis
• •ASRM, EBH, GR, AP, WJ, GW, MA, MC, RD, OM, AS, SK, DG, JZ, FS, LG, CRT– Image segmentation, contributory analytic support, clinical review and review of manuscript.
• •LK, PO: contributory analytic support, clinical review and review of manuscript, conceptual feedback and support
• •CRT, CDF: Co-corresponding authors; conceived, coordinated, and directed all study activities, responsible for data collection, project integrity, manuscript content, and editorial oversight and correspondence; direct oversight of trainee personnel. Statistical analysis and guarantors of statistical quality.

Appendix A. Supplementary data

The following are the Supplementary data to this article: