PROBLEM
National surveys conducted in 2012 and 2013 of students applying to radiation oncology residency revealed a high degree of variability in radiation oncology clerkship educational experiences [1,2]. The majority of clerkships had no structured didactic curricula, but students who completed clerkships that included structured didactics reported greater self-perceived preparedness for radiation oncology residency. In response, a structured didactic curriculum was developed and piloted at two institutions [3]. With the formation of the Radiation Oncology Education Collaborative Study Group, the curriculum was expanded to 22 institutions by 2016. Subjective feedback from participating students was positive, and students who completed at least one clerkship at a Radiation Oncology Education Collaborative Study Group institution reported greater postclerkship radiation oncology knowledge and preparedness for residency [4]. However, this evidence was subjective. Additionally, with the majority of medical students completing their radiation oncology clerkships at the start of their fourth year, several months pass before residency, which may degrade any impact provided by a structured curriculum. Do radiation oncology clerkships with structured didactics provide an objective improvement to student knowledge that is retained beyond the clerkship experience?
WHAT WE DID
An anonymous, Internet-based survey and knowledge assessment was developed with input from radiation oncology faculty members and senior radiation oncology residents. Study data were collected and managed using Research Electronic Data Capture. These electronic data capture tools are hosted at the University of Chicago [5]. The survey was divided into two sections (see Supplementary Material, available online). The first section collected baseline demographic information and contained questions to characterize respondents’ radiation oncology experiences before beginning radiation oncology clerkships. The second section collected details regarding the curricula of each clerkship completed and asked whether respondents completed a clerkship with a formalized lecture curriculum designed for medical students. To maintain anonymity, respondents were not asked to identify institutions by name. Branching logic was used in the survey to elaborate on specific responses. Therefore, the total number of survey questions varied depending on individual responses.
The knowledge assessment consisted of multiple-choice questions (MCQs) covering general radiation oncology knowledge, radiation biology and physics, simulations and emergencies, and treatment planning. An initial pool of 66 MCQs was developed with input from multiple stakeholders: a radiation oncology program director, a radiation oncology medical student clerkship director, a radiation oncology resident, and a medical student. The 66 MCQs were piloted with nine postgraduate year 2 to 5 radiation oncology residents. Questions with a resident proportion correct of 0.9 to 1.0 were deemed to be basic enough for a knowledge assessment at the medical student level. Feedback and performance data were used to identify 26 MCQs for inclusion in the assessment. Calculation of individual item statistics, option statistics, point biserial correlation, and test reliability (determined by Kuder-Richardson coefficient) was used to determine the final set of MCQs to include in performance analysis (see Supplementary Material, available online).
The survey and knowledge assessment were estimated to take 30 minutes to complete. Respondents were permitted to save their responses and to return at a later date. Only fully completed surveys and knowledge assessments were used in data analysis. Survey invitations were e-mailed after the 2016 US National Resident Matching Program (NRMP) rank-list deadline (February 26, 2016) to all applicants to a single radiation oncology residency program. Participants had 16 days to complete the survey. Three automated reminder e-mails were sent to invited participants who had not completed the survey. The survey was closed March 13, 2016, before the NRMP match day, to avoid any impact of match results on survey responses.
Statistical analysis was performed using Stata version 14 (StataCorp, College Station, Texas). Descriptive statistics were used to report general clerkship experiences. Using unpaired single-tailed t tests, the knowledge assessment performance of students who completed radiation oncology clerkships that incorporated formalized medical student lecture curricula (“curriculum students”) was compared with that of noncurriculum students. Effect size was calculated using standardized mean difference: [(mean performance of curriculum students) – (mean performance of noncurriculum students)/(standard deviation of all students)] [6]. Knowledge assessment performance is reported as average percentage correct ± SD. The University of Chicago institutional review board and the NRMP approved this project.
OUTCOMES
The survey was distributed to 200 applicants. Seventy-eight surveys (39%) were returned complete, from 35 curriculum and 43 noncurriculum students. Fifty-eight respondents (74.4%) were male. Sixty-one (78.2%) were MD students, while 16 (20.5%) were MD/PhD students. Demographics and prior experience with radiation oncology were similar between curriculum and noncurriculum students (Table 1).
Table 1.
Demographics and respondent characteristics
| Variable | All Students | Curriculum Students | Noncurriculum Students | P |
|---|---|---|---|---|
| Total | 78 (100) | 35 (100) | 43 (100) | |
| Medical school education track | ||||
| MD | 61 (78.2) | 28 (80.0) | 33 (76.7) | .485 |
| DO | 1 (1.3) | 1 (2.9) | 0 (0.0) | |
| MD/PhD | 16 (20.5) | 6 (17.1) | 10 (23.3) | |
| Medical school location | ||||
| United States | 78 (100) | 35 (100) | 43 (100) | |
| Gender | ||||
| Female | 19 (24.4) | 7 (20.0) | 12 (27.9) | .459 |
| Male | 58 (74.4) | 27 (77.1) | 31 (72.1) | |
| Does your medical school hospital or a hospital directly affiliated with your medical school have a radiation oncology residency program? | ||||
| Yes | 55 (70.5) | 24 (68.6) | 31 (72.1) | .734 |
| Prior radiation oncology experience | ||||
| Worked in a radiation oncology department conducting research | 48 (61.5) | 23 (65.7) | 25 (58.1) | .494 |
| Spent time in a radiation oncology department shadowing physicians | 55 (70.5) | 23 (65.7) | 32 (74.4) | .402 |
| Had a lecture on radiation oncology during preclinical years | 29 (37.2) | 14 (40.0) | 15 (34.9) | .642 |
| Other radiation oncology experience | 14 (17.9) | 4 (11.4) | 10 (23.3) | .176 |
| Prior to your first rotation, you had no radiation oncology clinical or research experience | 12 (15.4) | 6 (17.1) | 6 (14.0) | .698 |
| Radiation oncology clerkships (including home institution) | ||||
| 1 | 9 (11.5) | 3 (8.6) | 6 (14.0) | .898 |
| 2 | 17 (21.8) | 7 (20.0) | 10 (23.3) | |
| 3 | 39 (50.0) | 19 (54.3) | 20 (46.5) | |
| 4 | 10 (12.8) | 5 (14.3) | 5 (11.6) | |
| ≥5 | 3 (3.8) | 1 (2.9) | 2 (4.7) | |
| Prior to beginning your first radiation oncology rotation, were you definitely going to apply for a radiation oncology residency position? | ||||
| Yes | 41 (52.6) | 18 (51.4) | 23 (53.5) | .856 |
Note: Data are expressed as number (percentage).
The median number of radiation oncology clerkships completed by respondents was 3. Of the 215 total clerkships reported, 80 (37.2%) had curricula that included at least one lecture designed for medical students. Clerkship details are summarized in Table 2. Clerkships that included student lectures more often included other formal educational elements, such as prepared case discussions (53.8% versus 25.0%, P < .001) and hands-on didactic sessions (eg, contouring at a planning station on a preset plan) (56.3% versus 19.3%, P < .001).
Table 2.
Clerkship characteristics for the clerkships on which students reported
| Variable | All Clerkships | Curriculum Clerkships | Noncurriculum Clerkships | P |
|---|---|---|---|---|
| Total | 215 (100) | 80 (100) | 135 (100) | |
| This rotation was at a(n) | ||||
| University medical center (ie, affiliated with a medical school) | 193 (89.8) | 73 (91.3) | 120 (88.9) | .253 |
| Academic medical center not affiliated with a medical school | 17 (7.9) | 7 (8.8) | 10 (7.4) | |
| Community practice | 5 (2.3) | 0 (0.0) | 5 (3.7) | |
| How many weeks was this rotation? | ||||
| 1 | 2 (0.9) | 1 (1.3) | 1 (0.7) | .818 |
| 2 | 8 (3.7) | 2 (2.5) | 6 (4.4) | |
| 3 | 7 (3.3) | 2 (2.5) | 5 (3.7) | |
| 4 | 197 (91.6) | 75 (93.8) | 122 (90.4) | |
| Other | 1 (0.5) | 0(0) | 1 (0.7) | |
| Formal educational curricular elements | ||||
| Lectures specifically designed for medical students | 80 (37.2) | 80 (100) | 0(0) | * |
| Prepared case discussion (this excludes informal discussions in clinic) | 79 (36.7) | 43 (53.8) | 36 (26.7) | <.001 |
| Hands-on didactic session (eg, contouring at a planning station on a pre-set plan; this excludes contouring for a patient during the actual planning process) | 71 (33.0) | 45 (56.3) | 26 (19.3) | <.001 |
| Other (please specify) | 7 (3.3) | 5 (6.3) | 2 (1.5) | .105 |
| None of the above | 87 (40.5) | 0(0) | 87 (64.4) | <.001 |
Note: Data are expressed as number (percentage). Data were not collected on the students’ fifth clerkship.
”Lectures specifically designed for medical students” determined which clerkships were “curriculum.”
Analysis of MCQ difficulty and discrimination on the basis of respondent performance resulted in 15 level I, 3 level II, 2 level III, and 6 level IV MCQs [7]. The 6 MCQs of level IV difficulty and discrimination were excluded from performance analysis, narrowing the final assessment to 20 questions. The Kuder-Richardson coefficient was measured as 0.68. Respondents averaged 64.2 ± 17.0% correct responses on the knowledge assessment. Overall, curriculum students performed better than noncurriculum students (68.5 ± 18.0% versus 61.0 ± 16.0%, P = .025), averaging 1.5 more correct responses. Effect size was 0.44. By subject area, the largest difference between the two groups was on the six radiation biology and physics MCQs (curriculum students 63.3 ± 23.3% versus noncurriculum students 51.7 ± 23.3%, P = .013). Differences in performance on general radiation oncology knowledge, simulations and emergencies, and treatment planning MCQs were not statistically significant, but all trended toward higher performance in curriculum students (Table 3).
Table 3.
Respondent knowledge assessment performance
| Variable | Curriculum Students (n = 35) | Noncurriculum Students (n = 43) | P |
|---|---|---|---|
| Overall | |||
| Mean score | 13.7/20 | 12.2/20 | .025 |
| Percentage | 68.4% | 60.8% | |
| SEM | 0.61 (3.0%) | 0.48 (2.4%) | |
| General RO knowledge | |||
| Mean score | 3.9/5 | 3.6/5 | .147 |
| Percentage | 77.7% | 72.1% | |
| SEM | 0.20 (3.9%) | 0.18 (3.6%) | |
| Radiation biology/physics | |||
| Mean score | 3.8/6 | 3.1/6 | .013 |
| Percentage | 63.8% | 51.9% | |
| SEM | 0.23 (3.8%) | 0.21 (3.5%) | |
| Simulations and emergencies | |||
| Mean score | 4.9/7 | 4.4/7 | .103 |
| Percentage | 69.4% | 63.1% | |
| SEM | 0.26 (3.7%) | 0.23 (3.3%) | |
| Treatment planning | |||
| Mean score | 1.1/2 | 1.0/2 | .297 |
| Percentage | 55.7% | 51.2% | |
| SEM | 0.13 (6.4%) | 0.11 (5.6%) |
Note: RO = radiation oncology.
This study had several limitations. First, the knowledge assessment was not tested for comprehensiveness or externally validated. Further work is needed to develop a reliable and comprehensive knowledge assessment for medical students completing radiation oncology clerkships. Second, curriculum students were defined as anyone having completed a clerkship that included lectures at the medical student level. Recall bias may have affected which respondents reported receiving medical student lectures, and the content and quality of these lectures may have varied. Additionally, about half of the clerkships that included medical student lectures also incorporated hands-on didactic sessions or case discussions. The MCQ format may not accurately test objective knowledge obtained from these exercises, thereby limiting the applicability of these results. Finally, there were 223 applicants to radiation oncology programs in 2016 [8]. Thus there were 23 applicants who did not apply to our program who were not included in this study. Those applicants and the 61% of applicants who did not respond to our survey may differ from our sample in terms of their experience with structured curricula and knowledge retention, which limits the generalizability of our results.
Although a well-designed standardized test may identify high and low performers in terms of objective knowledge, whether a standardized examination is truly predictive of performance in residency is debated. To date, there are no objective attributes of a medical student that can reliably predict performance during residency. Prior examination performance can predict future examination performance, but this may not correlate with clinical aptitude [9,10]. Other studies have found certain correlations between residency performance and an applicant’s rank order list position, personality characteristics, surgical aptitude test performance, or Alpha Omega Alpha designation, but these results have been inconsistent across specialties or difficult to replicate [11–14]. A study of factors predictive of performance specifically in radiation oncology residency could not be identified. In a competitive specialty such as radiation oncology, an additional objective data point from a validated knowledge assessment may help applicants differentiate themselves and program directors identify the most qualified candidates.
Prior studies showed an improvement in students’ subjective perceptions of preparedness for residency training and radiation oncology knowledge [4]. This study objectively demonstrates that clerkships that incorporate didactics for medical students are associated with higher long-term (approximately 6 months) objective knowledge of radiation oncology principles among students. Curriculum students demonstrated significantly higher objective knowledge of basic radiation oncology, particularly with regard to radiation biology and physics. These results support the use of structured didactics as a standard component of a radiation oncology medical student clerkship curriculum.
Supplementary Material
Acknowledgments
This project was funded in part by the 2013 Philips Healthcare/RSNA Education Scholar Grant and National Institutes of Health Clinical Translational Science Award UL1 RR024999. Dr Golden has received grants from the RSNA and has financial interest in RadOncQuestions LLC. All other authors have no conflicts of interest related to the material discussed in this article.
Footnotes
ADDITIONAL RESOURCES
Additional resources can be found online at: http://doi.org/10.1016/j.jacr.2018.05.019.
Contributor Information
Ryan P. McKillip, University of Chicago Pritzker School of Medicine, Chicago, Illinois.
Gregory Kauffmann, Department of Radiation and Cellular Oncology, University of Chicago, Chicago, Illinois.
Steven J. Chmura, Department of Radiation and Cellular Oncology, University of Chicago, Chicago, Illinois.
Daniel W. Golden, Department of Radiation and Cellular Oncology, University of Chicago, Chicago, Illinois.
REFERENCES
- 1.Golden DW, Raleigh DR, Chmura SJ, Koshy M, Howard AR Radiation oncology fourth-year medical student clerkships: a targeted needs assessment. Int J Radiat Oncol Biol Phys 2013;85:296–7. [DOI] [PubMed] [Google Scholar]
- 2.Jagadeesan VS, Raleigh DR, Koshy M, Howard AR, Chmura SJ, Golden DW. A national radiation oncology medical student clerkship survey: didactic curricular components increase confidence in clinical competency. Int J Radiat Oncol Biol Phys 2014;88:51–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Golden DW, Spektor A, Rudra S, et al. Radiation oncology medical student clerkship: implementation and evaluation of a bi-institutional pilot curriculum. Int J Radiat Oncol Biol Phys 2014;88:45–50. [DOI] [PubMed] [Google Scholar]
- 4.Oskvarek JJ, Brower JV, Mohindra P, Raleigh DR, Chmura SJ, Golden DW. Educational impact of a structured radiation oncology clerkship curriculum: an interinstitutional comparison. J Am Coll Radiol 2017;14:96–102. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009;42:377–81. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Cohen J Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum; 1988. [Google Scholar]
- 7.Haladyna TM. Developing and validating multiple-choice test items. New York: Routledge; 2015. [Google Scholar]
- 8.National Resident Matching Program. Results and data: 2016 main residency match. Washington, District of Columbia: National Resident Matching Program; 2016. [Google Scholar]
- 9.Zuckerman SL, Kelly PD, Dewan MC, et al. Predicting resident performance from pre-residency factors: a systematic review and applicability to neurosurgical training. World Neurosurg 2018;110:475–84. [DOI] [PubMed] [Google Scholar]
- 10.Burish MJ, Fredericks CA, Engstrom JW, Tateo VL, Josephson SA. Predicting success: what medical student measures predict resident performance in neurology? Clin Neurol Neurosurg 2015;135:69–72. [DOI] [PubMed] [Google Scholar]
- 11.Wagner JG, Schneberk T, Zobrist M, et al. What predicts performance? A multicenter study examining the association between resident performance, rank list position, and United States Medical Licensing Examination Step 1 scores. J Emerg Med 2017;52:332–40. [DOI] [PubMed] [Google Scholar]
- 12.Bhat R, Takenaka K, Levine B, et al. Predictors of a top performer during emergency medicine residency. J Emerg Med 2015;49:505–12. [DOI] [PubMed] [Google Scholar]
- 13.Moore EJ, Price DL, Van Abel KM, Carlson ML. Still under the microscope: can a surgical aptitude test predict otolaryngology resident performance? Laryngoscope 2015;125:E57–61. [DOI] [PubMed] [Google Scholar]
- 14.Yost MJ, Gardner J, Bell RM, et al. Predicting academic performance in surgical training. J Surg Educ 2015;72: 491–9. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.