Career Outcomes of Graduates of R25E short-term Cancer Research Training Programs

Abstract

The efficacy of short-term cancer research educational programs in meeting its immediate goals and long-term cancer research career objectives has not been well studied. The purpose of this report is to describe the immediate impact on, and the long-term career outcomes of, 499 medical students and graduate students who completed the CaRES (Cancer Research Experiences for Students) program at the University of Alabama at Birmingham (UAB) from 1999 to 2013. In summer 2014 all 499 program alumni were located and 96.4% (481 of 499) agreed to complete a longitudinal tracking survey. About 23% of CaRES alumni (110 of 499) have published at least one cancer-related paper. Overall 238 cancer-related papers have been published by CaRES alumni, one-third of this number being first-authored publications. Nearly 15% (71 of 481 respondents) reported that their current professional activities include cancer research, primarily clinical research and outcomes research. Of these 71 individuals, 27 (38%) have completed their training and 44 (62%) remain in training. Of all respondents, 58% reported that they administered care to cancer patients and 30% reported other cancer-related professional responsibilities such as working with a health department or community group on cancer control activities. Of the 410 respondents not currently engaged in cancer research, 118 (29%) stated intentions to conduct cancer research in the next few years. Nearly all respondents (99.6%) recommended CaRES to today’s students. Challenging short-term educational cancer research programs for medical students and graduate health professional students can help them refine and solidify their career plans, with many program alumni choosing cancer research careers.

Introduction and Background

Despite increasing demand, the number of health professionals engaged in cancer research has declined [1]. In the early years of medical school and other graduate biomedical schools such as public health, dentistry, and nursing, it may be valuable to provide students the opportunity to participate in research that may shape their academic programs and better prepare them to make career choices. The purpose of this study is to evaluate the short- and long-term outcomes of 499 medical students, public health students, and other health professional students who participated in the CaRES (Cancer Research Experiences for Students) summer cancer research program at the University of Alabama at Birmingham (UAB) from 1999 to 2013.

CaRES was developed as an offspring of UAB’s Cancer Prevention and Control Training Program (CPCTP) R25 grant, originally funded by the National Cancer Institute (NCI) in 1988 [2, 3]. The CPCTP’s primary purpose is to support the study of post-doctoral fellows and doctoral students who are engaged in cancer prevention and control research. The early years of the CPCTP provided funds to support a few short-term students each summer. By 1998 it became apparent that the CPCTP could not take full advantage of the rich short-term cancer research opportunities available at the UAB Comprehensive Cancer Center (CCC). This realization led to development of the CaRES program and subsequent acquisition of funding to replace the summer interns portion of the CPCTP in 1999. Like the CPCTP, CaRES is funded by NCI’s R25 program. The CPCTP is in its 28^th year and CaRES is currently operating in its 15th year. Over the years CaRES has flourished as the UAB Comprehensive Cancer Center, School of Medicine, and School of Public Health have recruited many cancer research faculty members.

CaRES provides paid summer cancer research internships for UAB medical students, public health students, and other graduate biomedical students on the UAB campus or at UAB’s partner institutions. CaRES has supported over 540 students through 2014. The primary long-term goal of CaRES is to motivate participating students to pursue careers in cancer research. The short-term goals of CaRES are to provide each participating student with the support needed to collaborate on a cancer research project that could lead to a publication, abstract, poster, or presentation at a professional meeting. The purpose of this paper is to describe the results of our long-term tracking of CaRES alumni as well as our short-term CaRES intern and preceptor evaluation results.

Materials and Methods

Program Logistics

Each summer CaRES supports 40 student interns who have been chosen to undertake the cancer research projects proposed by members of the UAB-CCC faculty and posted on the CaRES website www.uab.edu/cares. About 60% of CaRES interns are medical students who are completing their first year of study, while most remaining interns are graduate public health students in their first or second year of study. Dental students, graduate nursing students, and graduate students from other health-related degree programs may participate. Recruitment of qualified minority students is emphasized. Undergraduate students can participate only if they hold a letter of acceptance from the UAB School of Medicine or UAB Graduate School.

The CaRES program provides each participating faculty preceptor a capable student who can accelerate his or her cancer research progress. CaRES preceptors are UAB-CCC faculty members who are actively engaged in cancer research. As of July 2014, 393 UAB faculty members held CCC appointments. A faculty member may propose more than one project as long as he or she has sufficient time available to supervise all interns recruited. Interested students schedule interviews with the preceptors of projects in which they have interest and for which they believe they are qualified. Preceptors interview as many students as they wish, then they select the students whose background, experience, and availability correspond with the demands of the project. Up to two interns may be selected for each project. Each selected student must provide his or her curriculum vitae or resume’ so that program eligibility (student status and proof of American citizenship or permanent residency) can be confirmed. Once a faculty member and a student have agreed to work with one another, e-mails are submitted to the Program Director to confirm their commitments to work together, and to claim one of the 40 internship slots. CaRES interns are paid at a rate set by NCI (maximum of $6000 for 15 weeks) and work from 8-12 weeks as negotiated with the preceptors, in light of the demands of each project.

The students chosen to participate are informed of the expectations of the program through meetings with the CaRES internship coordinator, who issues each student an award letter that details these expectations. A list of expectations is also posted on the program website, along with a document entitled “CaRES Tips for Success” that outlines how to succeed in the university environment. Typically all interns are selected by March 1 although the projects are performed in summer term (late May through August). Before starting an internship the student must meet with the preceptor to negotiate the work schedule and duties, which are recorded on the CaRES Intern/Preceptor Agreement Form (found on the CaRES website) and submitted to the Program Director. In mid-summer an all-day seminar is held with presentations by keynote speakers on the responsible conduct of research and career development issues. In addition each CaRES student presents the methods and preliminary results of his or her research, along with career plans. All interns are to present a poster at the university’s Medical Student Research Day, Public Health Research Day, Minority Health and Health Disparities Research Center annual conference, or at the UAB Comprehensive Cancer Center Annual Research Retreat, where a travel award is given to the CaRES student who presents the best poster as judged by CaRES program faculty. CaRES students are also encouraged to work with their preceptors to submit abstracts to professional meetings, and most importantly, to publish peer-reviewed papers.

Eligible students who interview for one or more CaRES projects but who are not offered a position are asked to contact the Program Director for assistance in finding summer employment on other training grants available at the university. In this way CaRES keeps track of the number of eligible and interested, but un-selected, students. In recent years the number of such contacts has ranged between two and six students, suggesting a close match between the number of available slots and the number of eligible and interested students who believe that they are qualified for selection. The CaRES program fills completely each summer, with 40-44 students supported, and on average only four students interviewed by at least one CaRES preceptor but not selected for a project.

Data collection and analysis

Evaluation of the CaRES program is necessary for continued receipt of funding from NCI and for continuous quality improvement. Each year data are collected from CaRES interns and preceptors in order to evaluate fulfillment of short-term goals. A brief, qualitative mid-summer evaluation instrument is administered to all participating interns in early July in order to identify any problems or issues that could be resolved quickly, thereby improving the quality of the internship for the remaining weeks. At the end of each summer all CaRES interns and preceptors complete exit evaluation surveys that measure satisfaction with their summer research experiences. The interns complete an evaluation survey consisting of objective items rated on Likert scales in the domains of preceptor performance, effectiveness of administrators and staff, knowledge gained, program logistics and career impact. The preceptors complete a survey instrument of similar format that includes items on administration of the CaRES program, benefits of the program, and their interns’ work ethic and contributions. Summary statistics are computed for each quantitative item and presented as summary means, and themes of satisfaction are gleaned from written comments. In addition, every five years CaRES alumni are tracked longitudinally in order to collect the data needed to evaluate the program for long-term outcomes, especially, career choices and publication of peer-reviewed papers. Data on cancer-related professional activities such as clinical care and oversight of screening programs are also collected. An in-depth longitudinal tracking methodology was developed for our alumni tracking in 2009 [4], while a new publication [5] presents the updated tracking methodology that we used in summer 2014.

One important indicator of cancer research activity is the number and quality of cancer-related, peer-reviewed publications. Based on responses to the items on our tracking questionnaire about publications, we conducted comprehensive literature searches on all CaRES program alumni to identify publications in two categories: 1) Publications directly related to CaRES internships, including both the intern and preceptor as authors; and 2) Later cancer publications authored by CaRES interns that provide evidence of cancer research activity. Publications in this second category may not directly relate to the CaRES project, and need not include the CaRES preceptor in the authorship. Although many CaRES interns present abstracts and posters on campus or at national meetings, these contributions are not peer-reviewed and therefore are not discussed here.

Results

Short-term Evaluation

Summary means for the objective items on the annual evaluation surveys completed by student interns for summers 2010-2013 are shown in Table 1. Estimates of variability are not shown to improve clarity. Interns were also invited to provide written comments. Overall satisfaction with the CaRES program is high and does not differ appreciably from year to year. The majority felt that the program was very well run, that their preceptors were of high caliber, and that their expectation had been met. Perceived strengths of the internships included learning new techniques, mentoring by the preceptors, working within team science, and gaining useful leadership skills. Most interns did not have the opportunity to write grant proposals and manuscripts, as reflected by lower scores in these areas. There was some inconsistency in scores for seminar satisfaction, likely related to the speakers and quality of student presentations that varied from year to year. A few interns suggested longer internships and improvements in communication, guidance, and clarification of responsibilities. Respondents were also asked to describe how CaRES impacted their career plans. Each student could select as many of six descriptor options as were applicable. The most common responses were that CaRES helped the students “refine” (selected by 40% of respondents), “identify” (30%), or “solidify” (28%) their career choices. Less than 20% stated that CaRES helped them “narrow,” “eliminate options from,” or “decide to change” their career plans, suggesting pre-existing interest in cancer research that we believe drew them to apply for a CaRES position. Many commented that CaRES expanded their views on the breadth of cancer research careers including clinical studies, community-based cancer prevention and early detection, and genetic research.

Table 1. Evaluation results of intern self-reported surveys conducted in 2010-2013.

Item	Content evaluated	Year
	CaRES Program Items: 1-5 scale, 5 is best	2010	2011	2012	2013	Mean
1	CaRES enhanced your knowledge of this research area	4.62	4.81	4.76	4.73	4.73
2	CaRES enhanced your knowledge of the research process and methods	4.65	4.75	4.76	4.68	4.71
3	CaRES staff communicated how CaRES works	4.46	4.86	4.57	4.66	4.64
4	CaRES system matched you to a good project	4.54	4.63	4.54	4.51	4.56
5	CaRES hours were flexible	4.85	4.89	4.65	4.66	4.76
6	CaRES stipend was adequate	4.50	4.50	4.51	4.63	4.54
7	CaRES staff responded to your questions	4.58	4.78	4.58	4.68	4.63
8	CaRES staff made expectations clear	4.65	4.92	4.76	4.65	4.75
9	CaRES effectively delivered short cancer research training experience	4.77	4.83	4.68	4.66	4.74
10	CaRES Seminar effective for communicating your research methods and results	3.19	4.39	3.96	4.37	3.98
11	Likelihood of pursuing cancer research career after graduation	3.50	3.78	3.39	3.66	3.58
12	Overall educational, research, and training experience provided by CaRES …	4.69	4.81	4.86	4.78	4.79
	Preceptor Items: 1-5 scale, 5 is best
13	Preceptor gave orientation and assistance	4.69	4.47	4.54	4.71	4.60
14	Preceptor showed interest in your progress	4.88	4.67	4.65	4.78	4.75
15	Preceptor understood goals of CaRES	4.58	4.47	4.58	4.63	4.57
16	Preceptor available to assist and advise you	4.54	4.42	4.32	4.66	4.49
17	Preceptor made clear his / her expectations	4.73	4.47	4.46	4.68	4.59
18	Positive research relationship with preceptor formed	4.85	4.81	4.85	4.68	4.80
19	Recommend preceptor to do CaRES again	4.85	4.69	4.76	4.80	4.78
20	Interacted with possible future collaborators	4.04	4.36	4.19	4.22	4.20
	Skills Learned Items: 1-3 scale, 3 is best
21	Enhanced general writing skills	1.38	1.33	1.41	1.48	1.40
22	Enhanced scientific writing skills	1.81	1.78	1.65	1.75	1.75
23	Enhanced laboratory procedures	2.00	1.97	2.30	2.36	2.16
24	Enhanced research design	2.23	2.31	2.51	2.38	2.36
25	Enhanced data analysis skills	1.96	2.29	2.24	2.03	2.13
26	Doing research in a university setting	2.31	2.58	2.46	2.40	2.44
27	Acquiring grants and funding	1.69	1.64	1.65	1.93	1.73

Note: The n completing this survey / N possible respondents is: 26/32 for 2010; 36/43 for 2011; 37/43 for 2012; and 41/43 for 2013, for an overall response rate of 140/161 (87%).

Summary means for the objective questions on the annual evaluation survey completed by preceptors for summers 2010-2013 are shown in Table 2. Preceptors were also invited to respond to open-ended questions regarding the program. As was true for the intern results, most aspects of the program were rated high each year. The preceptors engaged the interns in data collection and entry, experimentation, literature reviews, patient recruitment, and data analysis. The preceptors evaluated the interns favorably with regard to hours worked, tasks completed, and deadlines met. Most preceptors stated no problems although a few requested longer internships.

Table 2. Evaluation results of preceptor self-reported surveys conducted in 2010-2013.

Item	Content	Year
	Intern Quality Items #1-3: 1-5 scale, 5 is best	2010	2011	2012	2013	Mean
1	You had strong applicants to interview	4.35	4.70	4.52	4.70	4.57
2	Interviews were effective	4.12	4.70	4.56	4.59	4.49
3	Interns allow acceleration of research	4.88	4.64	4.51	4.68	4.68
4	Number of applicants interviewed (average)	2.63	3.67	3.14	2.82	3.07
5	Number of interns mentored (average)	1.63	1.81	1.81	1.46	1.68
	CaRES Program Items: 1-3 scale, 3 is best
6	Communication with CaRES program faculty	2.76	2.91	3.00	3.00	2.92
7	CaRES allows interns a flexible schedule	2.88	2.95	2.93	2.92	2.92
8	CaRES faculty / staff respond to questions	2.88	2.95	3.00	3.00	2.96
9	Stipend support was adequate	2.76	2.90	2.93	3.00	2.90
10	Expectations of preceptors and interns clear	2.82	2.91	2.85	3.00	2.90
11	Administration of CaRES program	2.82	2.81	3.00	3.00	2.91
	Preceptor Benefits Items: 1-4 scale, 4 is best
12	Help conceptualizing research methods	3.12	2.71	2.76	2.64	2.81
13	Help designing research plan	2.47	2.53	2.67	2.81	2.62
14	Help analyzing data	3.00	2.88	2.87	2.84	2.90
15	Help interpreting results	3.00	2.64	2.80	2.74	2.80
16	Help writing final paper / report	2.53	2.53	2.83	3.00	2.72
	Intern Performance Items: 1-4 scale, 4 is best
17	Hours worked vs. promised	3.58	3.56	3.77	3.64	3.64
18	Research tasks done	3.71	3.53	3.79	3.62	3.66
19	Deadlines met	3.63	3.44	3.67	3.57	3.58
20	Conceptualizing and designing research	3.46	3.33	3.05	3.24	3.27
21	Writing quality	2.54	3.28	3.32	3.48	3.16
22	Understanding methodology	3.63	3.56	3.57	3.45	3.55
23	Analyzing and interpreting data	2.79	3.23	3.26	3.28	3.14
24	% Willing to be CaRES Preceptor again	100	95	100	96	98

Note: The n completing this survey / N possible respondents is: 17/24 for 2010; 22/26 for 2011; 28/28 for 2012; and 28/29 for 2013, for an overall response rate of 95/107 (89%).

Another short-term outcome is the number and quality of publications directly related to CaRES projects, typically published within a few years of the CaRES summer, and jointly authored by CaRES interns with their preceptors. Over 30 peer-reviewed papers based on CaRES research and co-authored by CaRES students and their preceptors have been published in the first four years of our current grant (July 2010 - July 2014), reflecting short-term cancer research productivity. These publications are listed in Online Resource 1.

Long-term Outcomes

In July and August 2014 we conducted detailed and comprehensive longitudinal tracking of the 499 interns who participated in CaRES in any year(s) from 1999 through 2013 in order to gather data on their career choices and to document their cancer research achievements. We located all 499 individuals and 96.4% (481 of 499) completed our tracking survey [5]. Based on self-reports, 20% of respondents (96 of 481) published cancer-related, peer-reviewed papers, not necessarily relating to their CaRES projects, and not necessarily including CaRES preceptors in the authorship. Such papers provide evidence of cancer research careers. Our subsequent PubMed searches revealed that from 1999 to the present, overall 22% (110 of 499 alumni) have published a total of 238 cancer-related, peer-reviewed papers (Figure 1). A list of these 238 papers according to year of publication, with CaRES student authors indicated in bold font, is Online Resource 2. For one-third (32%) of these papers the CaRES alumnus was the first author. Of these 110 published authors, the majority (65%) authored one paper, 17% authored two papers, 15% authored three to ten papers, and 3% authored eleven or more papers. These papers have appeared in over140 journals, including two dozen cancer-focused journals. Journals having a high impact factor [6] in which at least two such papers have appeared are Cancer Research (impact factor 9.3), Clinical Cancer Research (8.2), Molecular Cancer Therapeutics (6.1), Cancer Letters (5.0), Journal of Thoracic Cardiovascular Surgery (4.0), and Annals of Thoracic Surgery (3.6). Among papers that are cancer site-specific, most numerous are papers about cancers of the lung, head and neck, pancreas, breast, cervix, ovary, and prostate, in that order. Other CaRES alumni papers focus on cancer staging, survivorship, education, and communications.

Figure 1. Cumulative distribution of cancer-related, peer-reviewed papers by CaRES alumni.

Our tracking survey also requested information on career choices and accomplishments. Key results are shown in Table 3. Forty-three percent (208 of 481) of respondents remain in training as students, residents, or fellows; while 57% (273) have completed their training. Forty-five percent are employed in academic research institutions (while still in training or thereafter). About 15% (71 of 481) stated that their current professional responsibilities include cancer research (44/208=21% of individuals still in training and 27/273=10% of individuals who have completed their training). Of the 71 individuals engaged in cancer research, 56% stated that they do clinical research, 35% do outcomes research, 31% do translational research, 22% do basic laboratory research, and 22% do cancer prevention and control research, with multiple responses permitted per respondent. Because the career paths of physicians and public health professional differ, we stratified the 71 self-declared cancer researchers by degree program in order to see whether they were medical students, public health students, dental students, or other types of biomedical graduate students, when they did CaRES. We found that the percentages of medical students and public health students who did CaRES and declared current cancer research activity were essentially the same: 15.3% (44 of 287) of medical students and 16.1% (22 of 137) of public health students. None of the dental students who did CaRES (0 of 30) stated current cancer research activity. Of the “other” students, 18.5% (5 of 27) declared current cancer research activity. This figure is difficult to interpret due to the variety of students (e.g., graduate students in nursing, psychology, sociology, medical technology) classified in the “other” category. Twenty-nine percent (118) of the 410 respondents not currently engaged in cancer research stated their intention to conduct cancer research in upcoming years.

Table 3. Self-reported data on career outcomes by n=481 CaRES alumni from 1999 - 2013.

Cancer-related outcomes	N (% yes)
Currently in training (student, trainee, resident, fellow)	208 (43.2)
Currently employed in academic or research institution	216 (44.9)
Current responsibilities include cancer research	71 (14.8)
If not currently engaged, plan to do cancer research in upcoming years	118 (28.8)
Publication of cancer-related papers in peer-reviewed journals	96 (20.0)
Clinical care (e.g., medical, nursing, dental, psychological) of cancer patients	278 (57.8)
Non-clinical cancer-related activities (e.g., administration, advocacy)	142 (29.5)
Member of any cancer-related organizations	47 (9.8)
Recommend CaRES to current students	479 (99.6)

Regarding cancer activities other than research (Table 3), 58% of respondents reported that they administered care to cancer patients, and 30% reported cancer-related professional responsibilities such as advocacy or administration of a cancer control program. Only 10% of respondents are members of a cancer-related professional organization. Virtually all respondents recommended CaRES to today’s students. Comments (edited for brevity and clarity) from several satisfied CaRES alumni are presented here with the students’ permission.

“As a result of the great experience I had conducting oncology based research through CaRES, I applied and was accepted to the Center for Clinical and Translational Science (CCTS) 1-year master’s program, to take a year out of medical school to continue my research I started as a CaRES student. Once I complete medical school, I plan to specialize in Surgical Oncology, and continue my focus on cancer research in the lab.”

“The CaRES program provided me with lab experience, gave me perspective. It’s something that I still use on my resume’ and see it as one of its strengths. I believed it helped me when I applied for residency and when I got one of my jobs. The program was one of the most memorable experiences in my life.”

“This internship gave me hands-on data analysis [experience] and got me hired for 2 years at NIH. It opened so many doors career wise for me.”

“Thanks for the opportunity for me to give back to CaRES by participating in the survey. I submitted it. CaRES was an amazing program and I hope it continues to grow.”

“CaRES was a great program for pre-med and medical students to explore their options in health care as it relates to cancer. There is a wide variety of issues surrounding the care of cancer patients including the care of the patient and caretaker issues that involve both psychological and medical issues.”

“The CaRES internship was a fantastic experience! Not only did it allow me to collect data for my thesis, but it also provided the experience that was beneficial to me in eventually securing cancer work as a contractor to CDC.”

“CaRES opened the door to doing one year of cancer research for the NIH in Washington, D.C.”

Discussion

It is important to differentiate between the short-term effects of a brief research training program on the perceptions of participants about research (immediate impact) and the long-term outcomes (engagement in research careers). Several authors have reported results of their cancer education programs. The Partners in Research program in Arkansas provides research experience for medical and pharmacy students, and for African-American undergraduate students [7]. This program had an uplifting effect on perceptions of cancer research. Presentations or publications were generated by 21 of 155 (13.6%) participants. Two students pursued further advancement in cancer research through education and by securing funding. The long-term outcomes used as benchmarks were authorship of scientific papers, presentations at national meetings, and graduation with honors from medical school.

A program designed to increase ethnic diversity in cancer control included a 5-day summer institute, Careers in Cancer Control Research, at the heart of its program [8]. The design of the institute featured components from social learning theory, self-efficacy, role modeling, and the theory of reasoned action to influence each participant’s intention to apply to a doctoral program. The impact evaluation measured an individual’s intentions to apply for a doctoral program before and after the institute experience, with continued follow-up for several years. The outcome of interest was enrollment in a doctoral program. Early data indicated that 30.6% had enrolled in a doctoral program or reported plans to do so within the next two years. Research conducted on satisfaction with the program found that several psychosocial and behavioral dimensions suggested readiness to apply for a doctoral program. Longer term follow-up of this 5-day summer cohort was published by Yancey in 2006 and indicated that 22.5% of participants from UCSF and 10% of participants at UCLA were enrolled, accepted into, or completed a doctoral program [9]. Of these alumni 21 of 31 at USCF and three of five at UCLA planned to conduct their doctoral research in cancer control. Other major themes included the importance of the program’s social and emotional support, networking opportunities, and exposure to minority leaders in cancer prevention and control.

Some educational research findings have been published on the early career choices of high school and college students introduced to careers in oncology and cancer research through summer research programs [10]. A program at Roswell Park Cancer Institute found that among college participants only 6% entered a non-scientific field. An early report by Legardeur described the effect of short research experiences for medical and undergraduate students at the Louisiana State University Medical School [11]. Although the short-term impact of the program was favorable (79% responded that the experience exceeded their expectations), the majority was undecided about whether they would pursue a career in cancer research (84%) or in oncology (62%). Long-term follow-up of the cohort was not available. The investigators state that the most favorable time during a student’s biomedical career to introduce a cancer research program is unknown.

Our analyses characterize the impact of the CaRES program and add to the knowledge base of evaluation of health education programs. Strengths of our study include the reliability of our short-term intern and preceptor evaluation survey statistics due to completion rates that were consistently high from year to year; and the use of our effective longitudinal tracking methodology developed in 2009 [4] and refined in 2014 [5] that allowed us to collect data from over 96% of 499 individuals who completed the CaRES program, thereby minimizing non-response bias.

Our investigation is unable to evaluate what percentages of students who do not participate in CaRES later select cancer research careers. The lack of a control group could be considered a limitation to the generalizability of our findings. But CaRES was not designed as a research study; it is a training grant designed to propel medical students and graduate public health students into cancer research careers. As such there is no control group, and we believe that no control group that we could choose would be valid. Had CaRES been an experiment in which we randomized some medical and public health students to CaRES, leaving their fellow students to engage in other summer activities, we could then longitudinally follow the CaRES alumni and their fellow students to ascertain any differences in professional accomplishments and cancer research activities. But CaRES is not a randomized trial. The students chosen by faculty preceptors to do CaRES are those students having the strongest preparation for and experience in cancer research, those students whose CVs most closely match the preceptors’ descriptions of the “ideal interns” for the CaRES projects. If in 2014 we chose “controls” from the medical and public health student cohorts of 1999 through 2013, such “controls” would not be valid because most such students did not apply for a CaRES internship, while a few others applied but were not chosen. Either scenario suggests that before the CaRES summer began, these potential controls were not comparable in interest or in research background or experience to the students chosen for CaRES. This is an example of “confounding by indication,” meaning that the students chosen by preceptors to do CaRES were the students most prepared to do CaRES, and thus the students most likely to engage in cancer research careers even if they did not participate in CaRES (12).

Still, CaRES has great impact, keeping in mind that it is a program of short duration (10-12 weeks) that costs NCI less than $5000 per student. We view CaRES as a vehicle for re-engaging cancer-interested medical students and public health students in cancer research, helping them advance their career decisions. Based on our extremely positive short-term evaluation results, we believe that CaRES has a very positive impact, increasing the likelihood that the students’ career paths will continue toward cancer research. Not surprisingly CaRES caused very few students to “decide to change” their career plans, and helped many students to “refine,” “identify,” or “solidify” their career plans which were evidently already research-oriented. Feedback from CaRES students and alumni suggests enthusiastic support for the program and the sense that CaRES focused and heightened their interest in cancer research.

In order to estimate the absolute effect of an R25E program on career selection, a large cohort of students, of a common academic background, degree of research experience, and cancer interest, would need to be identified. After some members of this cohort are enrolled in the R25E program, the remaining students would form the control group (13). In such a study the exposure is defined as ‘graduate summer research experience in cancer’ and the lack of exposure is defined as the absence of such an experience among comparable, cancer-interested, program-eligible individuals. This study could not be done within our CaRES program because our applicant pool of strong students closely matches the number of available slots, leaving only a handful of students interviewed but unchosen, presumably because they were less qualified than the students chosen. Even if we believed it was valid, we do not have enough un-awarded CaRES applicants from 1999-2013 to form a control group. An alternative approach for collecting some relevant information to address the issue of long-term impact would be to add pre/post interviews to R25E trainee evaluation procedures, although the logistics of so doing may not be feasible (14).

CaRES supports only medical students and graduate students, students who are mature enough to make career decisions with confidence, based on their academic track record and previous research experiences. We believe that a successful CaRES experience by such students who are oriented toward cancer research confirms and underscores their inclinations to choose a cancer research career. We believe that research experience at this critical time in career development (i.e., early in medical school or early in graduate school) is a strong determinant of career selection. We will continue tracking our CaRES alumni to monitor their career choices. In light of our improved tracking methodology we can explore patterns of career trajectories over time, the effect of previous research experience on selection for the CaRES program, and institutional factors that can impact long-term career choices.