A GRANT-BASED EXPERIMENT TO TRAIN CLINICAL INVESTIGATORS: THE AACR/ASCO METHODS IN CLINICAL CANCER RESEARCH WORKSHOP

Structured Abstract:

To address the need for clinical investigators in oncology, AACR and ASCO established the Methods in Clinical Cancer Research Workshop (MCCRW). The workshop’s objectives were to: (1) provide training in the methods, design, and conduct of clinical trials; (2) ensure that clinical trials met federal and international ethical guidelines; (3) evaluate the effectiveness of the workshop; and (4) create networking opportunities for young investigators with mentoring senior faculty. Educational methods included: (1) didactic lectures; (2) Small Group Discussion Sessions; (3) Protocol Development Groups; (4) one-on-one mentoring. Learning focused on the development of an IRB-ready protocol, which was submitted on the last day of the workshop. Evaluation methods included: (1) pre- and post-workshop tests; (2) students’ workshop evaluations; (3) faculty’s ratings of protocol development; (4) students’ productivity in clinical research after the workshop; (5) an independent assessment of the workshop. From 1996-2014, 1932 students from diverse backgrounds attended the workshop. There was a significant improvement in the students’ level of knowledge from the pre- to the post-workshop exams (p < 0.001). Across the classes, student evaluations were very favorable. At the end of the workshop, faculty rated 92-100% of the students’ protocols as ready for IRB submission. Intermediate and long-term follow-ups indicated that more than 92% of students were actively involved in patient-related research, and 66% had implemented five or more protocols. This NCI-sponsored MCCRW has had a major impact on the training of clinicians in their ability to design and implement clinical trials in cancer research.

Introduction

Although there have been recent advances against the many diseases collectively called cancer, there continues to be an urgent need for the development of new methods of early detection, prevention, and treatment. This need is even more urgent given the fact that a larger number of our population (the “baby boomers”) are now in their cancer-prone years.

Unfortunately, considerable concern has been expressed in the past and in more recent literature that there is a serious shortage of clinical/translational investigators (an “endangered species”) who can successfully design and conduct the clinical trials needed to determine if a particular new therapeutic, early detection, or prevention approach will help patients (1, 2, 3).

Since the time of those early reports, there continues to be a serious shortage of translational/clinical investigators who can design and conduct state-of-the-art clinical trials that match innovations in basic science with early drug development (4, 5). There continues to be a need for clinical/translational scientists with the ability to “build bridges across research’s Valley of Death” (i.e., clinical applications) (6-10). Simultaneously, there are relatively fewer biostatisticians focused on clinical trials specifically at a time when the methods for such trials have also evolved significantly to meet the challenges of new therapies and diagnostic methods.

For investigators involved in the design and conduct of clinical trials, advances in technology require continued training to take advantage of the latest investigative tools and methods, including molecular and imaging techniques (11-17) both in treatment and in prevention (18). Additionally, it is a major challenge for clinical investigators to maintain an understanding of regulatory agencies, such as the U.S. Food and Drug Administration (FDA) and European Medicines Agency (EMA) (19-23), in order to effectively utilize the various mechanisms available for expedited review and approval of novel therapies such as Breakthrough Designation (24-27). Finally, it is necessary for clinicians to understand effective clinical trial design and biostatistical principles, as well as the ethical principles and the need for transparency in disclosure of potential conflicts, in order to effectively communicate with patients regarding clinical trials and patients’ willingness to participate in them (28-37).

Although there has been a tremendous increase in the number of new potentially therapeutic agents introduced into clinical trials, a relatively small percentage of these agents produce positive clinical trial results. The reasons for this are multiple but certainly include (a) issues in the preclinical science which supports bringing a potential new therapeutic into clinical trials, and (b) some design issues in early clinical trials (38,39). Well-designed and well-implemented clinical trials are needed more than ever, given the limited number of patients eligible for trials, patients’ valuable time, the expense, and the time needed to conduct well-designed clinical trials (40,41).

Unfortunately, the success rate for phase III clinical trials remains unacceptably low (42-47). Given the costs and resources allocated to clinical trials, this is unacceptable from the perspective of cancer patients, clinical researchers, research supporters, and industry. Often not enough is being knowledgeable in phase I and II trials to be predictive of success (28, 40, 45). Potential remedies include: (a) better science in selecting new therapeutics to take forward into the clinic, (b) a better understanding of the reasons for the responses of patients in phase I and II trials; (c) design of new and better phase II trials that are more predictive of phase III success; and (d) better patient selection so that only those patients with the appropriate tumor molecular genotype/phenotype are entered into the trials.

The high demand for more clinical investigators is further compounded by the challenges of how to provide early-career oncologists with the most efficient and effective type of training in clinical/translational research, and how to afford them the time to devote to the training. Certainly, mentorship plays a major role (48, 49). There are excellent fellowship and training programs in many academic medical centers. However, so much needs to be accomplished to master the increasingly complex care of cancer patients, and little time remains for a concentrated effort in training young oncologists in clinical trial methods (9).

Continued education in the methodology, design, and implementation of clinical trials is needed to address the shortage of well-trained clinical investigators (5, 50, 51). Even well-trained clinical investigators beginning a career in the cancer field are deterred from clinical research by many obstacles including the lack of: funding; institutional support (e.g., training, certification, research time allocation); coordinated clinical research infrastructure; and mentorship (7, 10, 49, 52).

Taking the above background into consideration , the leadership of AACR, later joined by ASCO, felt that they had unique resources—expert investigators, teachers, and mentors—whom they could bring together to ensure that the next generation of clinical investigators received the best training possible. In an attempt to provide an intense period of training, and to set the stage for continued mentoring in the design and conduct of clinical trials, members of AACR and ASCO set out to develop and implement a one-week training workshop in clinical trial methods (S1). To ensure the rigor and quality of peer review needed for such a course, NCI funding was sought for assistance in supporting the workshop after a pilot period. The goal was to have the “Methods in Clinical Cancer Research Workshop” (MCCRW) provide knowledge, tools, mentorship, and peer-to-peer networking to overcome many of the daunting obstacles faced by young clinical investigators.

Overview

The initial training grant application was submitted to the NCI in 1996 but was not funded. The application was resubmitted after carefully addressing the critiques, and the first workshop grant was given a high priority for funding by the NCI that same year. Based on the requirement for the venue of the workshop to be relatively isolated, but accessible for major transportation and at a reasonable cost, the first event was held in Park City, Utah, and thereafter during the summer in Vail, Colorado.

Special Attributes of the MCCRW

As a result of several discussions among the organizers, it was decided that the workshop would incorporate the following attributes:

1. Teach the best possible methods for clinical trials so that patients would only be asked to participate in well-designed clinical trials that would answer important questions and move the field forward.

2. Require the student to produce a “product” at the end of the workshop. This would include a concept sheet, a full protocol, and a patient informed consent form that were ready for IRB submission at the student’s institution.

3. Conduct the workshop in an environment far from distractions in order to command the student’s full attention for as long as a physician could comfortably be away from patient care and other professional responsibilities.

4. Attract the best possible students through a competitive application process and by offering excellence in teaching. The applicant would need the support of a specific mentor/leader at his/her institution who would commit to the continued mentoring of the applicant.

5. Recruit a multidisciplinary faculty comprising biostatisticians, surgical oncologists, medical oncologists, radiation oncologists, pediatric oncologists, gynecologic oncologists, radiologists, pathologists, pharmacologists, bioethicists, patient advocates, and regulatory affairs experts with extensive experience in the design and conduct of clinical trials.

6. Maintain a low student/faculty ratio (preferably 2:1).

7. Utilize sound methods to evaluate the effectiveness of the workshop based on the following outcomes: (1) transfer of knowledge in clinical trial design and methodology; (2) implementation and completion of protocols designed at the workshop; (3) retention of workshop students in patient-oriented research; and (4) productivity of students in clinical trials after the workshop.

8. Put on display the necessary interactions among and between clinical investigators and their biostatistical colleagues working on trial methodology to provide a novel model of collaborations for beginning clinical investigators.

Specific Aims of the MCCRW

The specific aims of the workshop were to:

1. Provide training to a group of early-career clinical investigators in the methods, design, and conduct of clinical trials, including specialized designs for targeted and immunologic therapies, and methods to determine as rapidly as possible whether or not a particular design approach was effective.

2. Teach methods that will accelerate the development of better designed clinical trials that are early predictors of success. These designs would ensure that patients are not asked to participate in clinical trials that will not yield important insights.

3. Facilitate the development of attendees’ preferred networks and mentorships, both between the faculty and students and among the students themselves, and foster those networks.

4. Evaluate the workshop by the following criteria:

   1. Evaluation ratings of the workshop program, activities, and faculty by students and other participants
   2. Knowledge test scores before and after the workshop
   3. Students’ progress and advances during the workshop and development and completion of their protocol
   4. Percentage of protocols produced at the workshop that were activated after the workshop
   5. Percentage of students who reported staying in clinical trial research and implemented five or more clinical trials at 1 to 4 years (intermediate follow-up) and 5 years or more (long-term follow-up) after the workshop
   6. Comparisons of publication rates, citation impact, levels of clinical trial participation, collaboration, and collaborative networks of applicants selected versus those not selected for the MCCRW. It was felt that these objective analyses, performed by Thomson Reuters, would complement the follow-up self-report results.

Selection of the Workshop Directors, Faculty, Educational Evaluator, and Candidates

The criteria used for assigning workshop directors were outstanding accomplishments in clinical research and substantial teaching experience. In the first year, AACR and ASCO each selected an outstanding clinical investigator as their representative workshop (course) director. In year two, a third workshop director, a biostatistician, was added to address the critically important role of biostatisticians in the design and conduct of clinical trials. The biostatistician workshop director was nominated and agreed upon by both organizations. Via a request for proposal, an expert in educational evaluation was also selected and agreed upon by both organizations.

The Executive Committee for the workshop (workshop directors and administrative representatives from AACR and ASCO) met each year to set the curriculum and select the faculty members. In acknowledgment of their importance, patient advocates were added to the faculty when the workshop was in effect for about 5 years. Given the intensity of the workshop, as described in Supplemental Materials S1 and S2, the student to faculty ratio was set at approximately 2:1.

The workshop was announced to oncology training program directors and cancer center directors, and announcements were published by AACR and ASCO in their journals and on their websites. The application required a curriculum vitae, evidence of involvement in clinical research to date, a proposed project (later an initial concept sheet), and a written commitment of the student’s sponsor from the home institution for continued mentorship. A Selection Committee, comprising faculty members of the workshop for that year, selected the candidates, with the goal of 75% who were in their second or third year of subspecialty fellowship (or residency in the case of radiation oncologists) and 25% who were junior faculty members (for no more than 5 years).

In addition to the candidates, a limited number of corporate attendees were also invited to participate as observers during the workshop. Beginning in 2011, to increase the pool of qualified biostatisticians for subsequent workshops, a small number of junior biostatisticians was invited as non-faculty trainees to be mentored by faculty biostatisticians.

Workshop Outcomes

Short-term outcomes were established from different sources. The first was the participants’ ratings of the quality of the MCCRW program, activities, and faculty. The second was related to their performance during the workshop, namely (a) the incremental scores from the pre- to the post-workshop tests; (b) the listing of new concepts and processes that they acquired during the workshop; and (c) faculty ratings of the students’ progress in learning and completing their requisite research protocols in the Protocol Development Groups.

Intermediate and long-term outcomes were also measured. One measure consisted of determining from the students, through follow-up questionnaires one or several years after the workshop, (a) whether their participation in the workshop had been valuable to them in conducting clinical trials and in advancing their careers; (b) whether they had kept in contact and networked with the faculty; and (c) whether they had recommended the workshop to other clinical cancer researchers.

A second measure of intermediate and long-term outcomes focused on the students’ professional status, activities, and research performance. The latter was derived from two sources. One source was the follow-up questionnaires described above, in which the students were asked to report whether the protocol that they developed at the workshop had been submitted to the IRB, approved, and funded. In addition to these milestones, the students were asked to indicate the following:

1. Whether they had completed their training;

2. Their present professional position/title;

3. The focus and level of their research activity; and

4. The number of additional protocols they had developed and implemented.

The other source was the independent and objective evaluation commissioned by AACR and ASCO and conducted by Thomson Reuters. The outcomes assessed in that study include research productivity, citation impact, clinical trial involvement, and numbers of collaborations of applicants who were admitted to the workshop compared to those who were not accepted as students in the workshop.

Results

Demographic Data Over Time

Between 1996 and 2014, the number of applicants per year ranged from 142 to 279 (median of 195) with the number of students set at 100 per year (Figure 1). During this time period, a total of 1932 students had attended the workshop, a number sufficient for the purposes of definitive follow-up. It is noteworthy that there continues to be a considerable demand for the workshop, which has been quite steady over the entire period of the workshop.

Figure 1: Plot of Numbers of Students Selected Versus Number of Applicants.

The number of applicants and the number of students accepted each year (1996-2014).

The representation of women and minorities is needed for building clinical investigator capacity and is therefore an important aspect of this training opportunity. Accepted students were diverse, covering a variety of specialties in cancer medicine (Table 1) as well as gender and racial/ethnic categories (Tables 2a and 2b). In addition, the workshop has been attended by students from a broad spectrum of over 280 cancer centers and academic institutions across the country (S3). Throughout the years, there have been changes in faculty to reflect additional areas of expertise needed to teach the latest advances in clinical trial design (S4), e.g., immune oncology agents, special imaging techniques, and other rapidly evolving disciplines in clinical cancer research.

Table 1: Specialties of the Students (2007-2014).

Number of students selecting a given specialty for workshop years 2007-2014. Specialties of students showed a wide variety and range over the years of the workshop. Categories were self-selected.

	2007	2008	2009	2010	2011	2012	2013	2014	Total
Specialty	n	n	n	n	n	n	n	n	n
	%	%	%	%	%	%	%	%	%
Medical Oncology	57	51	56	53	58	46	50	49	420
	56.4%	51.0%	56.6%	53.0%	58.6%	46.0%	51.5%	49.5%	52.8%
Radiation Oncology	12	12	19	13	14	18	7	12	107
	11.9%	12.0%	19.2%	13.0%	14.1%	18.0%	7.2%	12.1%	13.5%
Hematology	9	11	9	14	3	13	12	11	82
	8.9%	11.0%	9.1%	14.0%	3.0%	13.0%	12.4%	11.1%	10.3%
Surgical Oncology	7	13	6	9	5	3	4	5	52
	6.9%	13.0%	6.1%	9.0%	5.1%	3.0%	4.1%	5.1%	6.5%
Pediatric Oncology	8	6	4	7	5	3	5	3	41
	7.9%	6.0%	4.0%	7.0%	5.1%	3.0%	5.2%	3.0%	5.2%
Gynecology	5	6	5	2	4	6	3	2	33
	5.0%	6.0%	5.1%	2.0%	4.0%	6.0%	3.1%	2.0%	4.2%
Neuro-oncology	3	1	0	1	6	4	4	3	22
	3.0%	1.0%	0.0%	1.0%	6.1%	4.0%	4.1%	3.0%	2.8%
Phase I Studies	0	0	0	0	0	6	4	6	16
	0.0%	0.0%	0.0%	0.0%	0.0%	6.0%	4.1%	6.1%	2.0%
Immunology	0	0	0	0	0	0	4	3	7
	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	4.1%	3.0%	0.9%
Cutaneous	0	0	0	1	1	0	1	1	4
	0.0%	0.0%	0.0%	1.0%	1.0%	0.0%	1.0%	1.0%	0.5%
Other	0	0	0	0	3	1	3	4	11
	0.0%	0.0%	0.0%	0.0%	3.0%	1.0%	3.1%	4.0%	1.4%
Total	101	100	99	100	99	100	97	99	795

Table 2a: Diversity of Students – Career Status and Gender.

Data on gender were only informally collected between 1996 and 2002. However, in 2003, we began to formally ask the students to check the appropriate demographic category on their application forms.

Year	Total # of Students	Students in Their Fellowships	Percentage Students in Their Fellowships	Students who are Junior Faculty	Percentage Students who are Junior Faculty	# Males	Percentage Male	# Females	Percentage Female
1996	105	*	*	*	*	66	64.10%	39	37.90%
1997	117	76	65.00%	41	35.00%	71	60.70%	46	39.30%
1998	104	75	72.10%	29	27.90%	84	80.80%	20	19.20%
1999	110	78+	70.90%	30+	27.30%	74	67.30%	36	32.70%
2000	100	79	79.00%	21	21.00%	49	49.00%	51	51.00%
2001	97	75	70.60%	22	21.10%	58	59.79%	39	40.21%
2002	100	75	75.00%	25	25.00%	61	61.00%	39	39.00%
2003	102	76	74.50%	26	25.50%	51	50.00%	51	50.00%
2004	100	76	76.00%	24	24.00%	53	53.00%	47	47.00%
2005	101	75	74.30%	26	25.70%	56	55.40%	45	44.60%
2006	101	76	75.20%	25	24.80%	60	59.40%	41	40.60%
2007	101	75	74.30%	26	25.70%	56	55.45%	45	44.55%
2008	100	75	75.00%	25	25.00%	56	56.00%	44	44.00%
2009	99	74	74.75%	25	25.25%	56	56.57%	43	43.43%
2010	100	75	75.00%	25	25.00%	53	53.00%	47	47.00%
2011	99	73	73.74%	26	26.26%	48	48.48%	51	51.52%
2012	100	77	77.00%	23	23.00%	50	50.00%	50	50.00%
2013	97	74	76.29%	23	23.71%	45	46.40%	52	53.60%
2014	99	74	74.75%	25	25.25%	46	46.46%	53	53.54%
Average	102.3	75.5	74.08%	26	25.36%	57.53	56.47%	44.8	43.64%

^*data not available

⁺only partial data available

Table 2b – Diversity of Students – Racial/Ethnic Categories.

Data on racial/ethnic categories were only informally collected between 1996 and 2002. However, in 2003, we began to formally ask the students to check the appropriate demographic category on their application forms.

Year	American Indians/ Alaskan Natives	Asian	Native Hawaiians or Pacific Islanders	Black or African Americans	White	Hispanic or Latino	Unknown/ Not reported	Total Jr. Faculty or Fellow	Total
2003	1	38	0	2	57	4	0	102	102
2004*	0	0	1	2	56	3	38	100	100
2005	0	33	0	5	52	6	5	101	101
2006	0	23	1	2	62	3	10	101	101
2007	0	25	1	2	60	1	12	101	101
2008	0	30	1	2	55	4	8	100	100
2009	1	31	0	3	55	4	5	99	99
2010	0	32	0	2	48	8	10	100	100
2011	0	26	0	1	58	8	6	99	99
2012	0	37	1	3	49	6	4	100	100
2013	0	38	0	4	45	2	8	97	97
2014	0	39	0	3	57	0	0	99	99

^*2004 Application did not include an option for "Asian."

Evaluation of Workshop Program, Learning Activities, and Faculty

Participants’ onsite evaluations of the workshop program had high rates of return (75% to 99%, mean = 92%). A large percentage of the respondents indicated that the objectives of the workshop were met (95.7 to 99.9%; Supplementary Table S1).

With a rating of 4 being Strongly Agree and 1 being Strongly Disagree, mean ratings ranged from 3.5 to 3.7 for the didactic lectures, 3.7 to 3.8 for the Protocol Development Groups, and 3.4 to 3.5 for the Small Group Discussion Sessions (Supplementary Table S2). Participants’ ratings of faculty in the Protocol Development Groups ranged from 3.7 to 3.9 with a mean rating of 3.8.

Evaluation of Students’ Performance at the Workshop

Paired t-tests comparisons of mean scores on the 50-60 multiple-choice-question examinations revealed significant increases (p < 0.001) from the pre- to the post-workshop test scores for all of the classes (Table 3).

Table 3: Pre- and Post-Workshop Mean Test Scores Comparisons (2001 to 2014).

There was a statistically significant increase each year from the pre to the post-workshop test score (Paired t-test, p < 0.001).

	2001	2002	2003	2004	2005	2006	2007	2008	2009	2010	2011	2012	2013	2014
# of Examinees	101 Ꝉ	101 Ꝉ	102	103 Ꝉ	101	101	100	101Ꝉ	98	101Ꝉ	99	99	95	99
Pre-workshop test mean score (SD)	49 (13)	52 (10)	47 (11)	58 (9)	55 (9)	57 (10)	54 (8)	52 (8)	56 (9)	51 (9)	55 (8)	58 (8)	55 (9)	58 (8)
Post-workshop test mean score (SD)	62 (10)	62 (8)	62 (9)	66 (9)	62 (10)	66 (9)	66 (9)	63 (8)	66 (9)	62 (9)	70 (9)	71 (8)	69 (9)	72 (8)

^ꝈTotal includes Jr Biostatistical and/or Corporate Attendees whose career status was neither as "Fellows" or "Jr. Faculty"

Faculty ratings of the students’ protocol development indicated that on average 92 to 100% (mean = 99%) of the final protocols, which included a protocol concept sheet, a full protocol, and the informed consent form, were judged acceptable and ready for IRB submission (Table 4). Regarding their individual performance during the Protocol Development Groups, 98 to 100% (mean = 99%) of students were rated as having made progress (Table 4).

Table 4: Faculty Evaluation of Students’ Final Clinical Trial Protocols and Progress in Clinical Trial Protocol Development Group (2002 to 2013).

The evaluation items introduced in 2002 are indicated by an asterisk. In 2014, the questions were changed to provide additional narrative on each student and their progression through the workshop.

Percentages of students at workshops held from 2002 to 2013 evaluated by the faculty as having an acceptable clinical trial protocol and having made progress
	2002*	2003	2004	2005	2006	2007	2008	2009	2010	2011	2012	2013	Mean
Protocol progress: Is the final protocol acceptable for submission to institutional review board?	92	98	98	97	96	97	100	97	100	95	98	93	97
Student’s progress: Did the student make progress on the protocol?	100	99	100	98	99	98	100	98	100	100	99	99	99

Evaluation of Students’ Performance after the Workshop

Intermediate (1-4 year) and long-term (≥5 years) follow-ups of workshop students had a return rate of 32 to 58%. With that limitation,: Supplementary Table S3 indicates the following: (a) 79 to 88% of the respondents remained in academia; (b) 92% or more spent some to substantial time in patient-related research; (c) 99 to 100% indicated that their participation in the workshop was valuable to conducting clinical trials and 89 to 93% to advancing their careers; and (d) 48 to 63% have maintained contact with workshop faculty and/or fellow workshop students. Forty-three percent to 52% of the respondents had their workshop protocols approved by the IRB and funded 2 to 5 years after the workshop. In addition, six to 50% of the respondents had 5 or more additional protocols implemented 2 to 5 years after the workshop, respectively.

Results of the Thomson Reuters Study

The complete and detailed Thomson Reuters report is provided in the supplemental material, titled “Evaluation of the American Association for Cancer Research (AACR) Workshop” (S5). Following is a summary of the key findings.

The report was based on the MCCRW applicants of the years 2002, 2004, 2006, 2008, and 2010. The sample used for the analyses included 480 students (i.e., selected applicants) and 520 unsuccessful applicants (i.e., applicants not selected for the workshop). As Thomson Reuters reported, “Searching of the Web of Science™ found a total of 31,261 publications between 1999 and 2013 which mapped to students and applicants…” (S5).

Overall, the yearly number of publications was higher for the students than for the unsuccessful applicants during the 3-year period after the workshop as compared to the 3-year period preceding it (Figure 2). “The average citation impact of the workshop participants increased between the before and after periods from around one and a half times the world average (1.52) to nearly twice the world average (1.95). The citation impact of unsuccessful applicants remained stable between the two periods (1.29)” (S5).

Figure 2 -. Total yearly number of publications, before and after workshop, of applicants who attended versus those who did not attend the MCCR workshop.

Plot courtesy of Dr. Yu Shyr.

Regarding clinical trials, 917 have been identified to which 346 workshop students and unsuccessful applicants had been linked. Of these, 205 are workshop students and 141 unsuccessful applicants (S5). It was found that (a) the students’ number of clinical trials increased significantly between the periods before and after the workshop, and (b) they collaborated with more coauthors and had larger collaborative networks than the unsuccessful applicants (S5).

SUMMARY AND FUTURE DIRECTIONS

This report presents the results of an NCI-sponsored Training Grant, titled “Methods in Clinical Cancer Research.” It has been well documented that there is a need for training more clinicians in clinical/translational research.

The workshop provides students with an intense week-long educational program that covers clinical trial methodology and the design and implementation of clinical trials through a mix of didactic lectures, Small Group Discussion Sessions, one-on-one mentoring, and Protocol Development Groups.

The students selected for the workshop are diverse in terms of gender, racial/ethnic backgrounds, oncology specialties, and training programs at various institutions. Even though this workshop has now been offered for over 20 years, there remains a continued “thirst” for it as evidenced by the fact that there are still many more well-qualified applicants than there are slots available to the student participants (Figure 1).

By the educational value parameters measured, (1) the didactic sessions have met their goal of improving the students’ level of knowledge based on pre- and post-workshop test scores; (2) the unique Protocol Development Groups, with their deliverable of an IRB-ready clinical trial protocol and patient informed consent form as an end product before the student leaves the venue, have also been assessed as a successful endeavor; and (3) the workshop outcomes as assessed by the students and faculty have all demonstrated the significant value of the workshop.

Students who completed the workshop and who responded to the follow-up surveys have largely remained in academic positions (79 to 88%); spent time ranging from “some” to “substantial” in clinical research (92 to 97% of respondents); and submitted their workshop protocols, as well as multiple other protocols to their IRBs, and implemented their protocols.

An independent and objective evaluation by an outside firm compared those applicants who were selected to attend the workshop as students to those not selected for the workshop (not inferring a causative relationship). The data indicate that the selected students had greater publication rates, increased citation impact, higher professional levels of working in clinical trials, and greater collaborations and larger collaborative networks.

The Protocol Development Groups are a unique feature of the workshop that cannot be provided at the students’ home institutions. With the current size of the workshop (100 students and about 45 faculty), there are 12 Protocol Development Groups to which individuals are assigned for the week. Faculty members guide and mentor students through the protocol development process each day, building on lecture session topics. Students have daily assignments that build upon the previous day’s work until the final assignment on the last day is submitted—i.e., a complete, IRB-ready protocol.

Throughout the Protocol Development Group sessions, a second tier of education and networking takes place among the students. In recent years, the Protocol Development Groups have been arranged, when possible, by disease site to encourage students to engage one another more openly and to build their own peer-to-peer networks that last well beyond the end of the workshop. Recent student evaluations have praised the Protocol Development Groups that were focused around similar disease sites instead of around phase I trials, phase II trials, biomarker trials, etc.

The faculty observed several important organizational points, including the following:

1. It is vitally important to have a biostatistician as one of the course directors.

2. The faculty roster should be frequently updated to reflect new focus areas such as patient advocacy, survivorship, biomarkers, novel imaging techniques, immunology, and many other timely topics. In particular, having patient advocates on the faculty has brought a very special and needed dimension to the workshop, especially the Protocol Development Groups.

3. Access to leaders in the field of clinical trial design and implementation enables students to leave the workshop with not only an expanded knowledge base, but also a core group of mentors and potential collaborators to enhance their careers. Students build relationships with the faculty and other students that prove to be invaluable for the success of their trials and, more broadly, for their professional careers in clinical cancer research.

Additionally, the following qualitative observations have been expressed by various course directors:

1. The protocols that are written and ultimately implemented reflect significant changes from the original concepts that the students bring to the workshop. This is the result of substantive input from the faculty and a direct measure of the learning that takes place during the workshop, all of which has led to improved clinical trial protocols.

2. Many important new concepts have been described/utilized in the course by the faculty, such as adaptive study designs, randomized discontinuation, N of 1, patient as own control, and multiple other designs.

3. Faculty enthusiasm is high year after year, with virtually no one wanting to decline an invitation to teach in the workshop. This is very encouraging given the intensity of the workshop, the amount of work involved in the teaching, the follow-up mentoring required, and the time that they have to spend away from their regular positions and families. The devotion and breadth of the faculty to this workshop have been exemplary (S4).

4. Although some have suggested putting the workshop online or publishing the syllabus, the faculty is of the strong opinion that these approaches will not capture the most important aspects of the course, e.g., the Protocol Development Groups and the one-on-one mentoring sessions with continued mentorship for years beyond the course. The faculty, students, and outside observers all agree that attending the workshop in person, with the opportunity for one-on-one interactions with the faculty, is essential to the training experience.

The MCCRW has been held since 1996, and over the course of nearly two decades it has inspired similar programs in Europe (Flims, Switzerland; now Zeist, The Netherlands) and Australia [the Australia & Asia Pacific Oncology Research Development (ACORD) Workshop], and India [the Collaboration for Research Methods Development in Oncology (the CReDO) Workshop], which have been repeated on an annual or biannual basis, as well as individual programs in South America, Asia, the Middle East, and Africa. The workshop in Vail has become highly respected in the field, and these related workshops are often referred to and advertised as “the Vail Course in Europe” or “Based on the Vail Workshop.” Others have written about these workshops in Europe (53,54). Comments on the Vail Workshop have also been published (55). Reischelman et al. (48) published a paper on MCCRW students from 1996-2004 and confirmed that mentorship was invaluable to oncologists in enhancing their research experience and expertise.

This workshop is one of the methods by which the NCI supports education and training. The NCI Career Development (K) award program, which supports investigators to develop their cancer research programs and achieve independence, has also been reviewed (56). That evaluation showed that the program had a positive impact “not only on participants’ biomedical research careers, but also on achieving outcomes significant to the scientific enterprise.”

In addition to the K award, the Clinical Translational Science Award program recognizes how important mentorship is to that program. This has also been reviewed (57). We feel that the MCCRW adds significantly to that mentorship approach; of note is that through 2014, 18 of the students at the MCCRW have returned to join the faculty including two students who have joined as co-directors.

To keep the workshop timely and on pace with rapidly advancing cancer science, there are now a number of potential modifications and additions under consideration for future workshops including: practical molecular tumor boards; transcriptomics; FDA Breakthrough Designation; and real-world evidence and AI approaches.

In summary, this one-week, focused, and intensive training workshop has been shown to increase students’ knowledge base, competence in clinical cancer research methods, and productivity in the clinical cancer research enterprise. Most of the students remain in patient-oriented clinical/ translational research. Students have proven to have superior publication history, wider networks of collaborators, and stronger clinical trial participation than their counterparts who were not selected to participate in the workshop. Participants report that the MCCRW has been a focus for multiple collaborations and a great source of lifelong, invaluable mentorship.

There continue to be concerns about having an insufficient number of clinical cancer investigators. Based on the data presented, it is believed that this workshop will continue to be an invaluable asset to help in the training of clinical cancer investigators.

Table 5: Average Number of Collaborators for Students and Applicants by Cohort Year.

Comparison of the average number of collaborators of students who attended the workshop vs. applicants who did not attend the workshop. These data were gathered by Thomson Reuters as part of a study commissioned by AACR and ASCO to provide independent evaluation of the workshop.

Cohort year	Students/ Applicants	Average number of collaborators before the workshop	Average number of collaborators after the workshop	Change
2002	Students	24.0	37.0	+13.0
	Applicants	14.7	20.2	+5.5
2004	Students	28.7	80.9	+52.2
	Applicants	30.2	34.0	+3.8
2006	Students	49.0	80.3	+31.3
	Applicants	18.0	25.3	+7.3
2008	Students	34.9	69.8	+34.9
	Applicants	18.7	32.4	+13.7
2010	Students	31.5	97.1	+65.6
	Applicants	17.4	35.6	+18.2