Abstract
Purpose/Objective
Radiation oncology trainees frequently learn to contour through clinical experience and lectures. A hands-on contouring module was developed to teach delineation of the postoperative prostate clinical target volume (CTV) and improve contouring accuracy.
Methods
Medical students independently contoured a prostate fossa CTV before and after receiving educational materials and live instruction detailing the RTOG approach to contouring this CTV. Metrics for volume overlap and surface distance (Dice similarity coefficient, Hausdorff distance (HD), and mean distance) determined discordance between student and consensus contours. An evaluation assessed perception of session efficacy (1 = “not at all” to 5 = “extremely”; reported as median[interquartile range]). Non-parametric statistical tests were used.
Results
Twenty-four students at two institutions completed the module, and 21 completed the evaluation (88% response). The content was rated as “quite” important (4[3.5–5]).
The module improved comfort contouring a prostate fossa (pre 1[1–2] vs. post 4[3–4], p<.01), ability to find references (pre 2[1–3] vs. post 4[3.5–4], p<0.01), knowledge of CT prostate/pelvis anatomy (pre 2[1.5–3] vs. post 3[3–4], p<.01), and ability to use contouring software tools (pre 2[2–3.5] vs. post 3[3–4], p=.01). After intervention, mean DSC increased (0.29 to 0.68, p<0.01) and HD and mean distance both decreased, respectively (42.8 to 30.0, p<.01; 11.5 to 1.9, p<.01).
Conclusions
A hands-on module to teach CTV delineation to medical students was developed and implemented. Student and expert contours exhibited near “excellent agreement” (as defined in the literature) after intervention. Additional modules to teach target delineation to all educational levels can be developed using this model.
Keywords: Medical education, radiation oncology, contouring, prostate fossa
INTRODUCTION
Radiation therapy (RT) requires highly conformal dose distributions that treat the target to a high dose while sparing the surrounding normal tissues. Although technological advances in RT planning and delivery have improved, the target volume still must be delineated by the radiation oncologist, and contouring variation among treating physicians is a possible source of error [1]. Inaccurate contouring may lead to morbidity, owing to unnecessary normal tissue irradiation or decreased tumor control if subclinical disease is omitted from the target volume [2].
The Radiation Therapy Oncology Group (RTOG) has made efforts to standardize contour delineation by developing contouring guidelines and atlases for numerous disease sites based on cooperative group consensus and published articles. However, very few studies are available on the teaching of contouring. Traditionally, radiation oncology (RO) education has utilized an apprenticeship model, but recently, medical education has shifted toward more structured curricular programs that can be validated by specific objective achievement (eg, the ACGME Milestones initiative) [3].
Given the importance of contouring accuracy, educational programs are necessary to standardize target delineation for radiation oncologists at all stages. We therefore developed a simulation-based interactive session focusing on target delineation, using Kolb’s [4] experiential learning model as a conceptual framework. In the module, we incorporated concrete experience, observation, conceptualization, and active experimentation.
The postoperative prostate fossa (PF) clinical target volume (CTV) was chosen for our module, as this target has contouring atlases and clearly defined guidelines [2,5–7]. Our primary objective was improvement of medical student contouring accuracy after live educational intervention. Secondary objectives included increased knowledge of prostate/pelvic anatomy, ability to use contouring resources, and confidence with treatment-planning software contouring tools. We hypothesized that implementation of a simulation-based didactic session would achieve these objectives.
METHODS
Study Participants and Design
Medical students at two institutions participated in the module as part of an RO clerkship. The entire educational module is available online [8]. Before the module, a “gold standard” contour for the prostate fossa CTV was created by two academic genitourinary radiation oncologists for evaluation of contour accuracy.
Teachers were given instructions for leading the module. Students were given a module description and a clinical scenario (prostate cancer patient with rising prostate-specific antigen after prostatectomy; referred for RT) to guide the task. The students independently contoured the CTV (ie, the prostate and seminal vesicle fossae) using a planning CT scan on a Pinnacle workstation (Philips Medical Systems, Fitchburg, Wisconsin). Students identified (yes/no) any prior experience they had in contouring the PF.
The students were given a PowerPoint-based educational document [8] and a live 30-minute teaching intervention (given by a resident or attending physician). The presentation (37 slides) reviewed intact and postoperative prostate/pelvis anatomy and RTOG contouring guidelines [2,5,6] supplemented with clarifying comments for students. Questions were encouraged during the presentation.
The students recontoured the CTV using the educational materials and any resources (web-based contouring atlases, publications, etc.). Student and “gold standard” CTVs were compared, and areas of difference were discussed with the instructor. Full data sets were unavailable for six of the students, as a result of files being either improperly saved or incomplete.
Data Analysis
Presession and postsession contours were compared with the gold standard contour using three metrics: the dice similarity coefficient (DSC), the Hausdorff distance (HD), and the mean distance [9–12]. DSC = 1 indicates a perfect overlap, and DSC = 0 denotes no overlap. In the literature, a DSC > 0.7 is commonly reported to indicate excellent agreement [13]. In addition, the contoured volume, volumetric difference (volumetric difference = (Vstudent pre/post − Vgold standard)/Vgold standard) [12], and slice number were compared. Volumetric data were collected using VelocityAI, v.3.01 commercial software (Varian Medical Systems, Atlanta, Georgia).
Postmodule Evaluation
A postmodule evaluation was distributed via REDCap (Research Electronic Data Capture) [14], a secure, web-based survey application that allows anonymous responses. Evaluation data were collected and managed using REDCap electronic data capture tools. The application is designed to support data capture for research studies, providing the following: (1) an intuitive interface for validated data entry; (2) audit trails for tracking data manipulation and export procedures; (3) automated export procedures for seamless data downloads to common statistical packages; and (4) procedures for importing data from external sources.
The evaluation included 13 questions (Appendix 1). The overall importance of the content covered in the module was evaluated. Students were asked about prior experience contouring a PF CTV. Confidence in four separate domains (ability to contour a PF, ability to use references to aid contouring, knowledge of CT prostate/ pelvis anatomy, and ability to use contouring tools in the treatment planning software) both before and after module completion was evaluated. These questions were scored on a Likert-type scale of 1 to 5 (1 = not at all; 2 = slightly; 3 = moderately; 4 = quite; and 5 = extremely). These data were analyzed using nonparametric statistics, with JMP Pro version 11 (SAS Institute, Cary, North Carolina). A value of P < .05 was considered statistically significant. This study was approved as exempt by the institutional review board.
RESULTS
Study Participants
Twenty-four students participated in the module (MD Anderson Cancer Center, n = 17; University of Chicago, n = 7). Table 1 reports relevant demographics. None had participated in the educational curriculum before this elective. All students were either at the end of third year (n = 2) or in the fourth year of medical school (n = 22). A total of 21 of 24 (88%) students responded to the postsession evaluation; 8 of 21 (38%) students had previously participated in the care of a post-prostatectomy patient; and 3 of 21 (14%) were familiar with postprostatectomy contouring guidelines.
Table 1.
Participant characteristics
| Medical School Year | 24 Students |
| 3 | 2/24 (8%) |
| 4 | 22/24 (92%) |
| Before Contouring Module | 21 Survey responders |
| Experience caring for post- prostatectomy patient | 8/21 (38%) |
| Familiarity with contouring guidelines | 3/21 (14%) |
| Prior participation in medical student educational curriculum | 0/21 (0%) |
CTV Analysis
Figure 1 displays the pre- and post-training CTVs of one representative student (row 1) and all MD Anderson Cancer Center students (row 2: pre-training; row 3; post-training) in the axial/coronal/sagittal planes compared to the expert contour. On qualitative assessment, most students erred by omitting the inferior portion of the bladder from the target volume before the teaching intervention. The volumes contoured were smaller than the gold standard, as many outlined only the space between the bladder and the rectum. After the module, the inferior bladder was included in the target volume, but most students incorrectly used the rectum as the posterior boundary (superior to the pubic symphysis), rather than the mesorectal fascia [5,6].
Figure 1.
(A), Axial; (B) sagittal; and (C) coronal planning CT images with CTV contours. The images in row 1 compare one student’s pre- and post-training contours with the expert contour. Rows 2 and 3 compare the CTVs of all the students from MD Anderson Cancer Center with the expert CTV, both before and after training, respectively. CTV = clinical target volume.
Differences in CT slice number, CTV, and volumetric difference are shown in Figure 2. The gold standard CTV encompassed 126.7 cc with 28 CT slices contoured. Mean CTV volume before instruction was 75.2 cc, and this increased to 169.1 cc. The volumes were, on average, 40.7 cc smaller than the gold standard before intervention and 33.5 cc larger after instruction (P < .01). Mean number of slices contoured increased from 22.2 (standard deviation [SD]: 9.9) to 28.7 (SD: 4.9) after instruction (P = .01). Figure 2 reports the DSC, HD, and mean distance for each student’s pre- and post-training CTV compared with the gold standard. After intervention, the mean DSC increased (0.29 to 0.68, P < .01), and HD and mean distance both decreased (42.8 to 30.0 mm, P < .01; 11.5 to 1.9 mm, P < .01).
Figure 2.
Metrics for analysis of CTV accuracy before and after teaching intervention including (A) number of CT slices contoured; (B) volume of contour; (C) volumetric difference of student and gold standard CTV; (D) dice similarity coefficient; (E) Hausdorff distance; and (F) mean distance. CTV = clinical target volume.
During the module, three students (two with saved postsession contours) identified themselves as “experienced” in contouring this target. For these two students, a mean DSC of 0.14 improved to 0.62. The maximum HD and the mean distance improved, from 53.4 and 18.1 mm, to 15.9 and 1.5 mm, respectively.
Postsession Evaluation
Twenty-one students completed the postsession evaluation (88% response) and rated the module as “quite” important (median: 4 [interquartile range: 3.5–5]). Students reported improved comfort and/or confidence in the following skill areas: contouring a PF (pre-1 [1–2] versus post-4 [3–4], P < .01); ability to find references to aid in contouring (pre-2 [1–3] versus post-4 [3.5–4], P < .01); knowledge of CT prostate/pelvis anatomy (pre-2 [1.5–3] versus post-3 [3–4], P < .01); and ability to use contouring tools in treatment planning software (pre-2 [2–3.5] versus post-3 [3–4], P = .01).
DISCUSSION
As part of a trend toward evidence-based medical education, medical specialties are developing standardized curricula with goals and objectives that can be assessed and evaluated. RO as a specialty has traditionally depended on the apprenticeship model to train new physicians and educate medical students [15]. In 2012, a standardized RO clerkship curriculum was established and has been evaluated at multiple institutions [16]. We augmented the curriculum with a simulation-based educational module to teach the accurate delineation of a CTV. Utilizing the experiential learning model of Kolb [4], students were led through an independent contouring experience, a pre-training contour review, an educational intervention, and active experimentation with post-training contouring. The data reported above demonstrate successful implementation of the module and achievement of the primary objective: improvement in contouring accuracy. Subjective, postmodule evaluations demonstrated achievement of secondary objectives as well.
Similar educational interventions have been reported in the literature. Teaching modules that are intended to improve resident head and neck target delineation skills have been developed, implemented, and evaluated [17, 18]. Khoo and colleagues [1] developed an educational program for practicing radiation oncologists that reduced inter- and intra-observer variation of prostate contours and was suggested for use in continuing medical education [1]. Similarly, Szumacher and colleagues [19] evaluated an educational intervention that improved the performance of RO trainees on prostate and rectal contouring [19].
Our educational module differs from these in several key aspects. As the target audience was medical students (instead of residents or attending physicians), the objectives and instructional tools were appropriate for learners who had little prior experience or baseline knowledge. Although other studies have evaluated teaching interventions using the intact prostate CTV, we used the PF CTV, as this presents unique challenges to the learner (eg, absence of target organ, changes in postoperative anatomy). In addition, we utilized more advanced analysis metrics for contour congruence than those in prior studies.
Multiple metrics were utilized to assess contouring accuracy. Initially, we simply compared the contoured volumes and the number of slices contoured. Before the teaching intervention, student contoured volumes were smaller than the gold standard contour, likely secondary to the avoidance and omission of the bladder (easily visualized in Figure 1). This error was corrected after the teaching intervention. After the intervention, many students used the rectum instead of the mesorectum as the posterior boundary of the CTV. This change inflated the overall contour volume and increased the treated volume. Initially, students contoured too few slices, often because they omitted the inferior portion of the target; they began contouring inferiorly only once the bladder separated from the rectum. After discussion of superior/inferior landmarks (eg, vesico-urethral anastomosis, penile bulb) and the necessity of including the bladder in the target, the number of slices contoured and the inferior target volume delineation were more accurate.
Although they are descriptive, these measures provide no information on the spatial overlap of the contoured volumes [9], and more sophisticated metrics were necessary. The teaching intervention significantly improved the DSC, and the postsession DSC of 0.68 is just shy of the widely quoted cutoff value of 0.7 for “excellent agreement” between test and gold standard volumes. As expected, the HD and mean distance decreased significantly after the teaching intervention. Even in those who self-identified as “experienced” in contouring this target, the concordance metrics showed improvement, confirming the utility of undergoing formalized teaching.
A few mistakes, common to a number of students, persisted after the educational intervention, and additional time will be spent in future teaching sessions to ensure mastery of these concepts. For example, students struggled with identification of the mesorectum as the posterior boundary of the CTV superior to the pubic symphysis. In addition, students commonly contoured a few slices more superiorly than the gold standard contour. In the module’s sample patient, contouring 3–4 cm superior to the pubic symphysis (as outlined in the guidelines) includes a loop of bowel in the target volume. Thus, the gold standard CTV stopped once the necessary seminal vesicle fossa was covered, and the bowel could be excluded. This example highlights the need for clinical experience that would allow finessing of the guidelines for individual patients. This level of experience has not yet been attained at the medical student level. An assessment of whether these errors persist at the resident and attending level would be interesting.
Postsession evaluations demonstrated that students valued the content covered during the educational session and perceived subjective improvement in all objectives. The students reported the smallest improvement on the objective addressing skill in prostate and pelvic anatomy, which is likely a result of prior anatomic knowledge from the standard medical school curriculum. The students reported the most improvement in their ability to contour a PF, a reflection of the success of the session in meeting the primary objective. For the three students who reported familiarity with PF contouring guidelines before the module, only one reported an improvement in the ability to contour this target; however, all of them reported improvements in utilizing contouring resources and in knowledge of anatomy. Two of three students reported improvement in use of contouring tools. For the five students who had participated in the care of a postprostatectomy patient but were not familiar with contouring guidelines, all reported improvement in ability to contour a PF by at least two levels on the Likert-type scale.
Although standardization of education is desirable, blindly following generalized guidelines is not appropriate, and this module is limited in its ability to teach learners the subtle nuances of CTV creation for an individual patient with a specific set of risk factors. The participants are informed during the session that the entire clinical picture must be considered and the guidelines adjusted for each individual patient, but this more advanced understanding, dependent somewhat on experience, is difficult to convey in a module such as this.
Unfortunately, even though we assessed contour concordance with multiple metrics, based on both surface distance and volume overlap measures, no single objective measure can fully characterize the differences in these volumes. Errors of similar volume can have drastically different clinical consequences depending on the location and the normal structures that overlap; evaluation of this is difficult using only mathematical computations. Additionally, having data from other participants, as a means to highlight common mistakes during teaching sessions, would be helpful. We are encouraged by the demonstrated benefit of the module, but we hope to further delineate the specific aspects (eg, live teaching, example contour review) that are most beneficial to the learner, perhaps by experience level.
This report details the successful development and implementation of a simulation-based contouring module designed to teach delineation of the PF CTV for use in RT planning. This module augmented an established medical student curriculum for the RO clerkship [16], but it can be implemented independently. The data demonstrate module success in both subjective, survey-based responses of the participants, and in objective data, utilizing volume and surface distance–based metrics that demonstrate improvement in student contours after intervention. This session specifically focused on medical students, but a module in this format would likely benefit RO residents and even practicing physicians (potentially incorporated into the maintenance of certification process). In addition, this tool could be used to provide objective data on contouring accuracy for assessment of resident ACGME milestone achievement. Additional educational modules to teach delineation of other common CTVs should be developed for all training levels using this model.
Acknowledgments
This work was supported in part by the National Institutes of Health grant UL1 TR000430.
APPENDIX 1
Prostate Fossa Contouring Module Evaluation
How important was the content covered in the prostate fossa contouring module?
Extremely important
Quite important
Moderately important
Slightly important
Not at all important
Did not attend
Prior to participating in the prostate fossa contouring module were you familiar with prostate fossa contouring guidelines?
Yes/No
Prior to participating in the prostate fossa contouring module, had you participated in the care of a post-prostatectomy patient in a radiation oncology clinic?
Yes/No
PRIOR TO COMPLETING this module, how comfortable were you in your ability to contour a prostate fossa?
Extremely comfortable
Quite comfortable
Moderately comfortable
Slightly comfortable
Not at all comfortable
AFTER COMPLETING this module, how comfortable are you with your ability to contour a prostate fossa?
Extremely comfortable
Quite comfortable
Moderately comfortable
Slightly comfortable
Not at all comfortable
PRIOR TO COMPLETING this module, how confident were you in your ability to find and use outside references including atlases and articles to aid in contouring?
Extremely confident
Quite confident
Moderately confident
Slightly confident
Not at all confident
AFTER COMPLETING this module, how confident are you in your ability to find and use outside references including atlases and articles to aid in contouring?
Extremely confident
Quite confident
Moderately confident
Slightly confident
Not at all confident
PRIOR TO COMPLETING this module, how confident were you in your knowledge of CT anatomy of the prostate and pelvis?
Extremely confident
Quite confident
Moderately confident
Slightly confident
Not at all confident
AFTER COMPLETING this module, how confident are you in your knowledge of CT anatomy of the prostate and pelvis?
Extremely confident
Quite confident
Moderately confident
Slightly confident
Not at all confident
PRIOR TO COMPLETING this module, how comfortable were you in your ability to use the contouring tools in the treatment planning software?
Extremely comfortable
Quite comfortable
Moderately comfortable
Slightly comfortable
Not at all comfortable
AFTER COMPLETING this module, how comfortable are you in your ability to use the contouring tools in the treatment planning software?
Extremely comfortable
Quite comfortable
Moderately comfortable
Slightly comfortable
Not at all comfortable
The most useful components of this module included:
Please describe any changes that could be made to improve the prostate fossa contouring module:
Footnotes
This work was presented at the American Society for Radiation Oncology 57th Annual Meeting in San Antonio, Texas, October 18–21, 2015.
ADDITIONAL RESOURCES
Additional resources can be found online at: http://dx.doi.org/10.1016/j.jacr.2016.02.030.
References
- 1.Khoo EL, Schick K, Plank AW, et al. Prostate contouring variation: Can it be fixed? Int J Radiat Oncol Biol Phys. 2012;82:1923–9. doi: 10.1016/j.ijrobp.2011.02.050. [DOI] [PubMed] [Google Scholar]
- 2.Michalski JM, Lawton C, El Naqa I, et al. Development of RTOG consensus guidelines for the definition of the clinical target volume for postoperative conformal radiation therapy for prostate cancer. Int J Radiat Oncol Biol Phys. 2010;76:361–8. doi: 10.1016/j.ijrobp.2009.02.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.ACGME. [Accessed September 25, 2015];Radiation oncology milestones. Available at: www.acgme.org/acgmeweb/portals/0/pdfs/milestones/radiationoncologymilestones.pdf.
- 4.Kolb DA. Experiential learning: experience as the source of learning and development. 2. Pearson; 2015. [Google Scholar]
- 5.RTOG. [Accessed September 25, 2015];Prostate post-op: post-op positive apex margins. Available at: www.rtog.org/CoreLab/ContouringAtlases/ProstatePostOp.aspx.
- 6.RTOG. [Accessed September 25, 2015];Prostate post-op: post-op positive seminal vesicles. Available at: www.rtog.org/CoreLab/ContouringAtlases/ProstatePostOp.aspx.
- 7.Wiltshire KL, Brock KK, Haider MA, et al. Anatomic boundaries of the clinical target volume (prostate bed) after radical prostatectomy. Int J Radiat Oncol Biol Phys. 2007;69:1090–9. doi: 10.1016/j.ijrobp.2007.04.068. [DOI] [PubMed] [Google Scholar]
- 8.Gunther JR, Liauw S, Choi S, Stepaniak C, Das P, Golden D. Postoperative prostate and seminal vesicle fossae contouring module: evaluation of medical student target delineation before and after a teaching intervention. [Accessed September 25, 2015];MedEdPORTAL Publications. 2015 Sep; Available from: https://www.mededportal.org/publication/10199.
- 9.Kalpathy-Cramer J, Fuller CD. Target contour testing/instructional computer software (tactics): a novel training and evaluation platform for radiotherapy target delineation. AMIA Annu Symp Proc. 2010;2010:361–5. [PMC free article] [PubMed] [Google Scholar]
- 10.Huttenlocher DP, Klanderman GA, Rucklidge WJ. Comparing images using the Hausdorff distance. IEEE Trans. 1993;15:850–63. [Google Scholar]
- 11.Chalana V, Kim Y. A methodology for evaluation of boundary detection algorithms on medical images. IEEE Trans Med Imag. 1997;16:642–52. doi: 10.1109/42.640755. [DOI] [PubMed] [Google Scholar]
- 12.Babalola KO, Patenaude B, Aljabar P, et al. An evaluation of four automatic methods of segmenting the subcortical structures in the brain. Neuroimage. 2009;47:1435–47. doi: 10.1016/j.neuroimage.2009.05.029. [DOI] [PubMed] [Google Scholar]
- 13.Zijdenbos AP, et al. Morphometric analysis of white matter lesions in MR images: method and validation. IEEE Trans Med Imag. 1994;13:716–24. doi: 10.1109/42.363096. [DOI] [PubMed] [Google Scholar]
- 14.Harris PA, et al. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377–81. doi: 10.1016/j.jbi.2008.08.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Jagadeesan VS, et al. A national radiation oncology medical student clerkship survey: didactic curricular components increase confidence in clinical competency. Int J Radiat Oncol Biol Phys. 2014;88:51–6. doi: 10.1016/j.ijrobp.2013.11.206. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Golden DW, et al. Multi-institutional implementation and evaluation of a curriculum for the medical student clerkship in radiation oncology. J Am Coll Radiol. 2016;13:203–9. doi: 10.1016/j.jacr.2015.06.036. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Bekelman JE, Wolden S, Lee N. Head-and-neck target delineation among radiation oncology residents after a teaching intervention: a prospective, blinded pilot study. Int J Radiat Oncol Biol Phys. 2009;73:416–23. doi: 10.1016/j.ijrobp.2008.04.028. [DOI] [PubMed] [Google Scholar]
- 18.Deraniyagala R, Amdur RJ, Boyer AL, et al. Usability study of the EduMod eLearning Program for contouring nodal stations of the head and neck. Pract Radiat Oncol. 2015;5:169–75. doi: 10.1016/j.prro.2014.10.008. [DOI] [PubMed] [Google Scholar]
- 19.Szumacher E, Harnett N, Warner S, et al. Effectiveness of educational intervention on the congruence of prostate and rectal contouring as compared with a gold standard in three-dimensional radiotherapy for prostate. Int J Radiat Oncol Biol Phys. 2010;76:379–85. doi: 10.1016/j.ijrobp.2009.02.008. [DOI] [PubMed] [Google Scholar]