Interactive dedicated training curriculum improves accuracy in the interpretation of MR imaging of prostate cancer

Abstract

Objective

To assess the effect of interactive dedicated training on radiology fellows’ accuracy in assessing prostate cancer on MRI.

Methods

Eleven radiology fellows, blinded to clinical and pathological data, independently interpreted pre-operative prostate MRI studies, scoring the likelihood of tumour in the peripheral and transition zones and extracapsular extension. Each fellow interpreted 15 studies before dedicated training (to supply baseline interpretation accuracy) and 200 studies (10/week) after attending didactic lectures. Expert radiologists led weekly interactive tutorials comparing fellows’ interpretations to pathological tumour maps. To assess interpretation accuracy, receiver operating characteristic (ROC) analysis was conducted, using pathological findings as the reference standard.

Results

In identifying peripheral zone tumour, fellows’ average area under the ROC curve (AUC) increased from 0.52 to 0.66 (after didactic lectures; p<0.0001) and remained at 0.66 (end of training; p<0.0001); in the transition zone, their average AUC increased from 0.49 to 0.64 (after didactic lectures; p=0.01) and to 0.68 (end of training; p=0.001). In detecting extracapsular extension, their average AUC increased from 0.50 to 0.67 (after didactic lectures; p=0.003) and to 0.81 (end of training; p<0.0001).

Conclusion

Interactive dedicated training significantly improved accuracy in tumour localization and especially in detecting extracapsular extension on prostate MRI.

Introduction

Advances in imaging technology and techniques have led to increased use of medical imaging in oncology, in both routine patient care and clinical trials. However, while such technical advances offer great potential, their successful implementation requires special human expertise, a fact that is often overlooked.

The importance of reader training and experience in oncologic imaging is evident from the literature on magnetic resonance (MR) imaging of prostate cancer. Risk-adjusted, patient-specific treatment planning for prostate cancer requires accurate determination of the tumour location and extent, and MR imaging is gaining acceptance as the most accurate imaging method available for this purpose [1-3]. Numerous studies on MR imaging of the prostate have shown substantial interobserver variability and provided evidence that interpretive accuracy is related to reader experience [4-7]. Mullerad et al. examined reader accuracy levels in detecting extracapsular extension of prostate cancer—an important prognostic factor that directly affects treatment selection and planning [8]. They found that genitourinaryradiologists experienced in prostate MR imaging had higher accuracy than general body MR radiologists [8]. In their study, MR imaging contributed significant incremental value to clinical variables only when it was read by genitourinary radiologists [8]. Despite such findings, no studies have yet been published examining whether a dedicated training curriculum could be used to improve the interpretation of prostate MR imaging.

The purpose of our study was to assess the effectiveness of an interactive dedicated curriculum in training our radiology fellows to interpret MR imaging of prostate cancer, using whole-mount pathology tumour maps as the reference standard. We hypothesized that such a training curriculum would significantly improve the fellows’ accuracy in determining tumour location and detecting extracapsular extension of prostate cancer on MR imaging.

Materials and methods

Patients

The institutional review board approved and issued a waiver of informed consent for our retrospective study, which was compliant with the Health Insurance Portability and Accountability Act. Data from consecutive patients who underwent MR imaging followed by radical prostatectomy between January 2003 and March 2006 were reviewed using institutional urology and radiology databases. Patients who received any kind of treatment for their prostate cancer, whether radiation, chemotherapy or hormone therapy, before undergoing MR imaging and/or surgery and patients who did not have whole-mount pathology tumour maps were excluded. A total of 215 patients were eligible for our study.

MR imaging technique

MR imaging studies were performed using a 1.5-T system (Signa; GE Medical Systems, Milwaukee, Wis.). A body coil for excitation and a pelvic phased-array coil (GE Medical Systems, Milwaukee, Wis.) in combination with an endorectal coil (Medrad, Pittsburgh, Pa.) for signal reception were used. Transverse spin-echo T1-weighted images were obtained through the pelvis with the following parameters: repetition time/echo time, 400–600 ms/8–10 ms; section thickness, 5 mm; intersection gap, 1 mm; field of view, 24 cm; matrix, 256×192; and two signals acquired. Thin-section, high-spatial-resolution transverse, coronal and sagittal T2-weighted fast spin-echo images were obtained through the prostate and the seminal vesicles with the following parameters: repetition time/echo time, 4,000–6,000 ms/96–120 ms; echo train length, 12–16; section thickness, 3 mm; intersection gap, 0 mm; field of view, 12–14 cm; matrix, 256×192; and four signals acquired.

Readers and MR image interpretation

Eleven radiology fellows (six body imaging, three breast and body imaging, two research), who had finished radiology residency, served as readers in our study. They were aware that patients had been diagnosed with prostate cancer and had decided to go to surgery, but they were blinded to all other clinical and laboratory data, biopsy results, original MR imaging reports, and histopathological and surgical findings.

The radiology fellows interpreted the MR imaging studies independently on our picture archiving and communication system (PACS) (Centricity, GE Medical Systems). Using a 1–5 scale (1, definitely absent; 2, probably absent; 3, possibly present; 4, probably present; 5, definitely present), they recorded the likelihood of tumour location on each side separately for the peripheral and transition zones of the prostate. Using the same 1–5 scale, they also indicated the likelihood of extracapsular extension separately on each side of the prostate. If more than one lesion was found in the same patient, each lesion was recorded separately.

Baseline MR imaging readings before training

Fifteen MR imaging studies were selected randomly by the research study assistant (HFE) from the list of eligible patients. The fellows each read the same 15 MR imaging studies independently before receiving any dedicated training so that their baseline interpretation accuracy levels could be determined. In performing these readings, the fellows relied on their previous general radiology training.

The 15 initial MR imaging studies were assessed by one of the expert genitourinary radiologists (JZ) who confirmed that the extent of the findings and the degree of difficulty they presented were similar to those presented by the subsequent training cases.

Interactive dedicated training curriculum

Our dedicated training curriculum (Fig. 1) consisted of didactic lectures with interactive case presentations, the radiology fellows’ independent MR imaging interpretations, and interactive tutorials in which expert radiologists reviewed the fellows’ interpretations, using whole-mount pathology tumour maps as the reference standard.

Fig. 1.

Summary of interactive dedicated training curriculum in the interpretation of MR imaging of prostate cancer

Interactive didactic lectures

The fellows received five interactive didactic lectures in the week following their baseline MR imaging interpretations. The didactic lectures were given by three genitourinary radiologists experienced in MR imaging of prostate cancer (HH, OA, JZ), one of them with 20 years of experience and two of them each with 4 years of experience. During the didactic lectures, the normal anatomy of the prostate, the appearance of prostate cancer and the features of tumour spread on MR imaging were discussed, as previously described in the literature [2, 3]. All of these topics were introduced in the first lecture and reiterated in each subsequent lecture. The discussions were carried out interactively. The fellows were encouraged to ask questions to improve their understanding of the subject. Each interactive didactic lecture contained quiz cases to supplement their learning.

Weekly MR imaging readings and interactive review tutorials

Each radiology fellow independently interpreted 200 prostate MR imaging studies, which were assigned in groups of 10 per week over 20 weeks. Throughout the study, all fellows reviewed the studies in the same, randomized order in which they were assembled by the research study assistant (HFE). The fellows were allowed to work at different speeds and spend as much time as they needed on their interpretations. They filled out data sheets to indicate their interpretations and delivered them to the study coordinator at the end of each week.

After delivering their MR imaging interpretations, the radiology fellows as a group received an interactive review tutorial of that particular week’s MR imaging studies. The review tutorials were given by the same three expert genitourinary radiologists who gave the didactic lectures. The experts reviewed the MR imaging studies on PACS and discussed the findings of tumour location and extent on each case. During this review, both MR imaging studies and the corresponding whole-mount pathology tumour maps were projected on the screen. The fellows were free to ask questions and received feedback on their correct and incorrect interpretations during the tutorials.

Reference standard

In each case, the whole-mount histopathology tumour map obtained from the institutional pathology database served as the reference standard. After prostate resection, the specimen was step-sectioned into 3- to 5-mm slices. A pathologist experienced with prostate cancer, who was blinded to MR imaging results, determined the tumour location, surgical Gleason score, and pathologic stage for each patient. Cancer foci were outlined in ink on whole-mount slices so as to be grossly visible. These slices were then photographed to provide whole-mount pathology tumour maps. In each tumour map, the margins of cancer foci indicating their location within the prostate and the presence or absence of extracapsular extension or seminal vesicle invasion were marked.

Only prostate cancer foci that were greater than 1 cm on pathology were included in the analysis of tumour localization on MR imaging. Smaller foci were not included in the analysis because they can often be found in multiple regions of the prostate, making analysis of tumour localization unfeasible [9].

Statistical methods

We performed receiver operating characteristic (ROC) analysis to determine the radiology fellows’ individual and average accuracy levels in determining tumour location and in detecting extracapsular extension of prostate cancer at MR imaging. The areas under the ROC curves (AUCs) were estimated non-parametrically using methods previously described [10], to account for the correlation due to the multiple observations per patient.

We examined the radiology fellows’ interpretations for the initial group of 15 prostate MR imaging studies separately to determine their baseline accuracy levels. We analysed the interpretations they performed in the first 2 weeks after the interactive didactic lectures, to examine the effects of these didactic lectures. We also analysed their interpretations for the last 2 weeks to observe additional changes in their accuracy levels at the end of the dedicated training curriculum.

We plotted the fellows’ individual and average AUC scores on a chart to graphically represent the changes in the accuracy levels they achieved before training, after didactic lectures and at the end of training. To test whether their AUCs changed during these intervals of the study, linear mixed models were used. Linear mixed models are a generalization of the linear regression model. They can accommodate analysis of a more complicated data structure and can incorporate the standard fixed effects used in linear regression as well as random effects. We modelled the AUCs as a function of time and fellow. Time was used as a fixed effect, because we were interested in the specific time points under study. The variable representing the fellow was included as a random effect, because we thought of each fellow as being a sample of the entire population of fellows who participated in the study.

Results

Pathology findings

Among the 215 patients included in our study, there were a total of 296 prostate cancer foci greater than 1 cm on pathology; 260 (87%) (120 on the right and 140 on the left) of these prostate cancer foci were in the peripheral zone and 39 (13%) (18 on the right and 21 on the left) were in the transition zone. The mean patient age was 58.2 years (range, 40–76 years), and the mean baseline serum prostate-specific antigen level was 6.2 ng/mL (range, 0.5–25 ng/mL). The median Gleason score at surgical pathological examination was 7 (range, 6–9). The mean interval between MR imaging and surgery was 39.6 days (range, 1–126 days).

Sixty-three (29%) patients had extracapsular extension of prostate cancer on pathology. In 20 patients extracapsular extension was on the right side, in 37 it was on the left side and in 6 it was on both sides. Four (1.8%) patients had seminal vesicle invasion on pathology.

Radiology fellows’ accuracy in interpreting MR imaging

Determining tumour location

In identifying cancer foci in the peripheral zone of the prostate, the radiology fellows’ baseline average AUC before training was 0.52 (range, 0.41–0.66). Their average AUC significantly increased to 0.66 (range, 0.57–0.77) after didactic lectures (p<0.0001 vs. baseline) and remained at 0.66 (range, 0.57–0.78) at the end of the training curriculum (p<0.0001 vs. baseline) (Table 1, Fig. 2).

Table 1.

Determining tumour location in the prostate

	Peripheral zone			Transition zone
Radiology fellow no.	Before training	After didactic lectures	End of training	Before training	After didactic lectures	End of training
1	0.42	0.68	0.63	0.33	0.19	0.61
2	0.42	0.66	0.66	0.46	0.42	0.71
3	0.57	0.63	0.78	0.64	0.91	0.90
4	0.66	0.71	0.71	0.54	0.62	0.79
5	0.41	0.57	0.62	0.48	0.58	0.44
6	0.45	0.63	0.62	0.45	0.70	0.50
7	0.59	0.71	0.66	0.46	0.83	0.85
8	0.54	0.60	0.57	0.35	0.74	0.56
9	0.52	0.77	0.67	0.59	0.64	0.70
10	0.59	0.72	0.59	0.59	0.63	0.58
11	0.54	0.63	0.70	0.50	0.73	0.88
Average	0.52	0.66	0.66	0.49	0.64	0.68

Data are AUC scores

Fig. 2.

Scatterplots (a, b) and ROC curves (c, d) summarize the radiology fellows’ individual and average accuracy levels in determining tumour location in the peripheral and transition zones of the prostate before training, after interactive didactic lectures and at the end of dedicated training. For the peripheral zone, the average AUC significantly increased from 0.52 (baseline before training) to 0.66 (after didactic lectures) and remained at 0.66 (end of training) (p<0.0001 for both) (c). For the transition zone, the average AUC significantly increased from 0.49 (baseline before training) to 0.64 after didactic lectures (p=0.01 vs. baseline) and to 0.68 (end of training) (p=0.001 vs. baseline) (d)

In identifying cancer foci in the transition zone, the radiology fellows’ baseline average AUC before training was 0.49 (range, 0.33–0.64). Their average AUC significantly increased to 0.64 (range, 0.19–0.91) after didactic lectures (p=0.01 vs. baseline) and to 0.68 (range, 0.44–0.9) at the end of training (p=0.001 vs. baseline) (Table 1, Fig. 2).

Determining tumour extent

In identifying extracapsular extension of prostate cancer, the radiology fellows’ baseline average AUC before training was 0.50 (range, 0.35–0.70). Their average AUC significantly increased to 0.67 (range, 0.48–0.80) after didactic lectures (p=0.003 vs. baseline) and further increased to 0.81 (range, 0.63–0.96) at the end of training (p<0.0001 vs. baseline); the increase in average AUC that occurred between the period just after the didactic lectures and the end of the training was also significant (p=0.008) (Table 2, Fig. 3).

Table 2.

Detection of extracapsular extension of prostate cancer

	Extracapsular extension
Radiology fellow no.	Before training	After didactic lectures	End of training
1	0.55	0.63	0.80
2	0.48	0.68	0.88
3	0.64	0.80	0.96
4	0.42	0.76	0.63
5	0.42	0.59	0.75
6	0.41	0.65	0.76
7	0.68	0.75	0.93
8	0.35	0.74	0.69
9	0.49	0.75	0.77
10	0.70	0.48	0.92
11	0.41	0.53	0.87
Average	0.50	0.67	0.81

Data are AUC scores

Fig. 3.

Scatterplot (a) and ROC curves summarize the radiology fellows’ individual and average accuracy levels in detecting extracapsular extension of prostate cancer before training, after interactive didactic lectures and at the end of dedicated training. The average AUC significantly increased from 0.5 (baseline before training) to 0.67 (after didactic lectures) (p=0.003 vs. baseline) and to 0.81 (end of training) (p<0.0001 vs. baseline). The increase in average AUC that occurred between the period just after the interactive didactic lectures and the end of dedicated training was also significant (p=0.008) (b)

The small number of patients (4/215) with seminal vesicle invasion on pathology in this study precluded analysis of the effect of the training curriculum on the fellows’ accuracy in detecting seminal vesicle invasion. We noted that among the four cases of seminal vesicle invasion in the study population, one case was detected by nine fellows, one was detected by three fellows, one was detected by one fellow, and one was not detected by any of the fellows.

Discussion

Our study shows that radiologists’ accuracy in interpreting prostate MR imaging can be improved by the use of an interactive dedicated training curriculum. Our radiology fellows’ average accuracy in determining tumour location in both the peripheral and the transition zones of the prostate significantly increased after interactive didactic lectures and remained essentially the same at the end of the training curriculum. In identifying extracapsular extension, the fellows displayed not only significant performance improvement after didactic lectures, but further significant performance improvement at the end of the training curriculum, ultimately achieving substantially greater average accuracy than they achieved in tumour localization. Their lesser performance in tumour localization could be explained by the inherent limitations of MR imaging and patient-related factors, most importantly post-biopsy changes in the prostate, that hamper tumour localization at MR imaging [2]. In general tumour detection has been considered more difficult in the transition zone than in the peripheral zone of the prostate. However, two recent studies showed that MR imaging can be used to assess transition zone prostate cancers [11, 12]. In our study, accuracy levels for tumour localization were similar in the two zones.

The extent of prostate cancer is one of the most important factors considered in assessing prognosis and making treatment decisions. Our results suggest that although interactive didactic lectures are helpful, continued hands-on experience with expert feedback is critical for further improving the detection of extracapsular extension and can lead to a fairly high level of accuracy.

It remains to be determined whether the performance of our radiology fellows could be further improved. Previous studies found that experienced readers performed better than less experienced readers in interpreting prostate MR imaging studies [5-8, 13]. It is a complex task to measure and compare reader experience, which is often expressed by the years in practice and/or the number of cases interpreted. In general, the radiology fellows in our study achieved satisfactory accuracy levels, and some achieved accuracy levels comparable to those of experts, especially in detecting extracapsular extension.

Experience and subspecialty training can affect diagnostic performance with any imaging study [14]. Breast imaging is one of the subspecialty areas of radiology for which the relationship between reader experience and diagnostic performance has been studied most. One study found that in interpreting screening and diagnostic mammograms, breast imaging specialists detected more cancers and more early-stage cancers, recommended more biopsies and had lower recall rates than did general radiologists [15]. The superior performance of the breast imaging specialists was explained by several factors, including more initial training and more continuing education in mammography, as well as greater continuing experience in mammographic interpretation [15]. Another study showed that a minimum of 2,500 screening mammography interpretations per year was associated with lower abnormal interpretation rates and average or better cancer detection rates [16]. Dedicated mammography teaching courses have also been shown to improve radiologists’ diagnostic performance [17].

Similar studies have been done in other radiology subspecialty areas. For example, a multicentre study on the interpretation of CT colonography investigated the effect of directed training on the performance of inexperienced readers relative to that of experienced readers [18]. The study found that the overall accuracy of the experienced readers (74.2%) was significantly higher than that of the trained radiologists (66.6%) and the technologists (63.2%), but some trainees reached the mean performance level achieved by the experienced readers [18]. Individual performances overlapped considerably among all groups and the investigators suggested that competence might be achieved by certain talented individuals through a training program [18].

MR is gaining acceptance as the most accurate imaging investigation for the local assessment of prostate cancer. The improved performance of MR imaging within the last several years has probably been due mainly to advances in MR technology [2, 3]. Understanding of imaging criteria and experience in image interpretation are also growing [4, 8, 19, 20]. Yet MR imaging of prostate cancer is not universally performed and may be under-utilized even in some centres with the necessary equipment. This under-utilization is likely partly due to a lack of experienced radiologists whose interpretations could add significantly to the clinical assessment of prostate cancer [8, 20]. We believe that greater availability of radiologists trained in prostate MR imaging could expand the utilization of this imaging method and improve patient care. Furthermore, dedicated training of radiologists before they participate in technology assessment studies or clinical trials of prostate MR imaging could lead to results that more accurately reflect the potential of the technology to contribute to patient care.

Our study had a number of limitations. The radiology fellows only assessed conventional MR images to determine tumour location and extent. It is well known that post-biopsy changes in the prostate may cause under- or overestimation of tumour presence and extent at conventional MR imaging. Advanced MR imaging techniques, such as MR spectroscopic imaging, diffusion-weighted and dynamic contrast-enhanced MR imaging, which may improve tumour assessment by providing metabolic and functional information, were not assessed [21-24]. Counter to routine practice, the radiology fellows were blinded to the locations of prostate cancer foci on biopsy, and this may have caused lower levels of accuracy than would be found among clinical readings. However, this blinding was necessary to better analyse the effects of the training curriculum. Our results could also have been affected by interobserver variability in the interpretation of MR imaging, which has been reported previously [25]. Another possible limitation of our study is that the readers were radiology fellows with other educational and clinical responsibilities and varied career plans, which could have caused differences in their levels of dedication to the study. Their individual performances varied despite their similar educational backgrounds and levels of experience, and a small number of fellows did not show improved performance. It is predictable that some fellows would outperform others despite similar backgrounds and experience. However, in a study such as this it is difficult to measure individual talent.

To Err is Human: Building a Safer Health System, a report from the Institute of Medicine of the National Academies of Science, indicated the need for improvement in the training and education of health professionals [26]. Traditional radiology education is based on reading textbooks, attending lectures, and observing and assisting experienced radiologists during the residency and fellowship years. While exclusively didactic education increases physicians’ knowledge, it is not sufficient to change physicians’ performance levels and patient outcomes [27, 28]. The explosion in medical imaging technology demands a switch from this traditional approach to subspecialty training and a largely self-directed learning system, in which didactic instruction, small-group sessions and new educational tools are used to give radiology trainees the skills to apply their knowledge [29].

In summary, our study demonstrates the power of interactive dedicated training, supplementing didactic lectures with case-based discussions, hands-on experience and feedback sessions, to improve reader performance in assessing prostate cancer on MR imaging. The use of such an interactive dedicated training curriculum could increase the availability of radiologists qualified to interpret prostate MR imaging and expand the use of this imaging method; the large and continuous improvement found in the accuracy of detection of extracapsular extension of prostate cancer, an important factor in treatment planning, indicates that such a dedicated training curriculum could have immediate positive effects on patient care.

Our study could serve as a model for future studies to further establish evidence-based training guidelines for imaging methods. Similar interactive dedicated curricula could be used to train and perhaps certify radiologists in the use of imaging techniques before they apply them in patient care or clinical trials.