Approaches for Longitudinally Tracking Graduates of NCI-Funded Short-Term Cancer Research Training Programs

Abstract

Short-term cancer research programs for health professions students typically state goals of encouraging and fostering interest in pursuing cancer research-based careers. Evaluating career choices and professional achievements of these trainees has been problematic. Well-designed program evaluation is a key element in determining successful training experiences, and program-specific outcome measures are typically used to assess the effectiveness of each short-term cancer research experience. We describe evaluation approaches for longitudinal tracking of participants of our National Cancer Institute-sponsored Cancer Research Experiences for Students (CaRES) program at the University of Alabama at Birmingham, emphasizing those approaches that demonstrated greatest effectiveness. We also evaluate the various methods we used in terms of cost, efficiency, and effectiveness. Recommendations are provided to assist comparable training programs facing the challenges of longitudinal tracking and program evaluation.

Introduction

A. Training for Healthcare Professionals in Research

As progress is made in the fields of technology and biomedicine, the demand for healthcare professionals who are actively engaged in research has increased [1,2]. Healthcare professionals engage in research to educate the public on the significance of research trials, to increase the number of studies implemented, and to foster advancements in disease control and prevention [1, 3]. However, the number of health professionals engaging in cancer research has sharply declined [2, 4–5]. Possible explanations for this abatement include debt due to steepening graduate and health professions school expenses and the aging of the current cohort of researchers, who are not being replaced by equal numbers of new researchers [2]. To counteract this trend, several incentives have been offered to physicians such as loan repayment programs funded by the National Institutes of Health (NIH), joint MD/PhD programs, research opportunities in sub-specialty fellowships, and other research options offered later in training [2, 5]. Providing research opportunities to individuals pursuing advanced degrees in the health profession has been frequently noted as a successful approach to impact their career decisions [1, 4–7].

B. Short-term Research Programs for Students

There has been recent concern with respect to low enrollment of health professions graduate students pursuing research careers. Programs that offer short-term research experiences for graduate students expose them to scientific career opportunities, and familiarize them with new areas of research [4, 7]. Short-term student research programs are typically conducted during the summer, and are offered by many universities and institutions nationwide [8–11]. Several of these programs have received funding by the National Cancer Institute (NCI) through its R-25 training programs that may lead students to consider careers in cancer research [12–14]. One key attribute required to demonstrate the effectiveness of these programs is that they must include longitudinal follow-up of program participants as a program evaluation element. Because the long-term goal of these short-term research programs is to encourage students to consider careers in research, data are needed in the years following their participation to determine whether these students chose to pursue scientific careers, and if so, in what arenas. This information correlated with other outcomes measure program success. Many programs struggle with the challenge of appropriate and effective methods for longitudinal follow-up and evaluation.

The Cancer Research Experiences for Students (CaRES) program at the University of Alabama at Birmingham (UAB) has operated for over ten years and has been highly successful with regard to short-term endpoints such as student satisfaction, publications, and presentations. This paper focuses on the longitudinal tracking methods used to determine how effectively CaRES facilitates the choice of a cancer research career.

C. The CaRES Program

The CaRES program began at UAB in summer 1999 as an offshoot of a program already in progress, the NCI-funded Cancer Prevention and Control Training Program (CPCTP) that supported study by post-doctoral fellows and doctoral students, and included funding for short-term summer research experiences for a small number of interns. The CPCTP had been funded by the NCI via an R-25T grant since 1988 [15–17]. When it became evident that the UAB Comprehensive Cancer Center (CCC) could provide many additional short-term cancer research experiences for students at UAB, NCI funded an R-25E grant that became the CaRES program. From 1999 to 2008, CaRES supported 320 health sciences students to do short-term cancer research projects directed by CCC faculty. The projects and opportunities available through CaRES are diverse, with approximately 40% of projects in basic research, 40% in clinical research, 10% in community research, and 10% in statistical/analytic research. Each project must have a clear cancer focus. CaRES is open to all incoming and first-year medical, dental, and public health students at UAB. The primary goal of the program is to introduce health professions students to cancer research. The secondary goal of CaRES is for alumni of the program to become engaged in cancer research or in clinical cancer care after graduation.

Methods and Results

Our longitudinal follow-up and evaluation efforts were directed at two primary goals: locating all previous participants of the CaRES program (“CaRES alumni”) and updating their current contact information (addresses, phone numbers, email addresses), and second, updating each alumnus’s current activities (employment, further education) in relation to cancer research. In a later publication, we will detail our findings; here we describe the tracking methods used and evaluate the effectiveness of each method.

We used various methods to find our CaRES alumni. Upon locating them, we sent them a brief email survey requesting current employment/student status, contact information, and information on any activities related to cancer patient care or cancer research. Other data elements included were the year(s) of CaRES participation, the name(s) of former preceptors, and the academic unit in which they were enrolled at the time of their CaRES participation. A current curriculum vita was also requested. The survey in its entirety is displayed in Table 1. All information collected was housed in a Microsoft Access database.

Table 1.

Sample questionnaire sent to CaRES alumni

Sample Questionnaire Sent to CaRES Alumni
Last name used while a CaRES intern:
Current last name (if different):
First name and middle initial:
Current position and title:
Current mailing address:
Email address:
Phone number(s):
Are you currently a student or trainee?	Yes	No
If yes, name of institution and program?
Are you currently employed in an academic/training or research institution?	Yes	No
Do your professional responsibilities involve cancer related activities?	Yes	No
Do your current responsibilities involve cancer research?	Yes	No
If yes, what is your cancer research focus? Check all of the following that apply:
Basic or laboratory cancer research
Clinical research related to cancer
Translational research (basic to clinical)
Cancer prevention and control research
Cancer survivorship research
Palliative care research
Cancer quality of life
Cancer outcomes research
Other cancer related research
Do your professional responsibilities involve clinical care of cancer patients?	Yes	No
Do your professional activities include leadership/management responsibilities?	Yes	No
Do your volunteer activities involve cancer related organizations?	Yes	No
Will you please send us a copy of your CV/Resume?	Yes	No
If yes, how will it be sent?
Email
Fax
US Mail
Other (please indicate mode)

Below we share our experience with each tracking method and recommend certain approaches that were especially productive. The methods are listed in the order in which we tried them for CaRES longitudinal tracking.

A. Alumni Records

We utilized existing records from the alumni offices in the Schools of Medicine, Dentistry, and Public Health because the vast majority of students in the CaRES program were drawn from these three schools. For some individuals, this information included their last known location and employer. However for many individuals, insufficient information was available.

Although the information from these files served as a useful starting point in the tracking process, the variability in quality of information necessitated follow-up with each participant to confirm or invalidate the information given. This extensive follow-up proved to be a daunting and extremely time-consuming task. This method can be useful if no prior information is known; however it must be used in conjunction with other methods and therefore is not highly recommended.

B. Home Institution Electronic Directory

Another technique used was the electronic campus directory of UAB. Because the vast majority of participants in CaRES were at one time or continued to be UAB students, we utilized the directory, searching by name or variations of name. The electronic university directory demonstrated the greatest success in locating participants, providing accurate information, and with ease of navigation and use. This tool typically offered useful information such as email addresses. It also indicated whether individuals were employed by UAB or if they were a current student or an alumnus, their course of study and year of graduation. Such comprehensive information was extremely helpful. Also we frequently were able to find some piece of contact information (if not an email address), such as a telephone number or middle name or initial that provided further opportunities for tracking. However, the contact information for CaRES alumni from the earliest years of the program tended to be outdated or invalid.

Overall, this method demonstrated an extremely high rate of information return and ease of navigation, and thus is highly recommended if available. This particular method may not generalize to other institutions, though, due to variability across their directories which may not be electronic, easy to navigate, or as comprehensive as the one described here.

C. Outside Institution Websites

For individuals listed in alumni records as either currently enrolled, in residency, or employed at a particular university or hospital, we visited the web address for the specific institution. We searched in detail for the individual in question via search tools available at the institution’s site using the site’s directory, student/resident/employee listings, and any other available options.

This technique rarely resulted in contact information for CaRES alumni. Websites for different institutions can be vague in their search terms, difficult to navigate, and confusing. One must acquaint oneself with each particular site individually in order to successfully search. This method was extremely time consuming and generally inefficient. It should be noted, that for the few CaRES alumni located using this method, there was substantial accuracy in the information, and these individuals often responded upon contact. However, because this technique resulted in so few CaRES alumni locations, it is not highly recommended.

D. Social Networking Sites

Social networking sites were also used to search for former CaRES participants. These sites are consistently gaining in popularity, particularly among the age range (20’s and early 30’s) of our targeted population. We used the social networking website “Facebook”, typing in the names of the participants. If results appeared to match the search criteria (name, student or alumnus, listed location in Birmingham, and/or listed location where last known to reside), the individual was sent a message via the “Facebook” messaging tool explaining the longitudinal tracking process and requesting his or her email address.

The use of social networking websites, in this case, “Facebook”, resulted in a moderate response rate from those contacted. This website was chosen because it is the world’s largest and most popular social on-line network, with an estimated 500 million users [18]. Although the website is easy to navigate, this method was extremely time-consuming for several reasons: dozens of results may be found for each search, and the only way to confirm that you have found the correct person is through Facebook’s messaging system. Sending numerous individual messages is cumbersome, especially when including personalization (although personalization such as names and year(s) of participation demonstrated increased responses).

“Facebook” is popular among persons in their twenties, offering high potential to reach many CaRES alumni; however, it cannot locate those who are not on social networking sites and may be limited to specific demographic groups. Higher response and success rates might have been found through this method if multiple social networking sites had been explored, however given the limited results acquired with the trial of “Facebook”, learning to navigate and search other networking sites was determined to be too time consuming and inefficient. Depending on the time availability and age range of individuals being searched for, this method is moderately recommended. However, users should be aware of the limitations and concerns discussed here.

E. Personal Contacts

Another method used was requesting CaRES preceptors and others who knew CaRES alumni to contact the alumni and explain our tracking efforts. This technique provided some information. When approached directly by a personal contact, CaRES alumni almost always responded quickly and provided the desired information. This finding is extremely important and useful when a program has the resources to personally contact participants. Those who made the contacts were former preceptors, professors, colleagues, and friends. Limitations include preceptors and professors having lost touch with individuals, particularly those from the earliest years of the program, as well as the evaluation team’s substantial effort needed to establish contact with the former preceptors or professors. Depending on the size of the institution, this may or may not be a plausible method to employ. There is considerable time involved in sending emails and making telephone calls, but if these tasks are shared among various individuals they are not overwhelming. If feasible, considering the minimal input, quick return, and high response, this technique is highly recommended.

F. Search Engines

Search engines such as the website “Google” were employed in the basics of confirming the last reported location of an individual and obtaining contact information. We used these sites by entering the person’s name, graduate degree, and last known location. Often multiple attempts were made to find an individual, using variations of potential locations, maiden/married names, and other identifiers to maximize the possibility of locating individuals.

This technique was time consuming and frequently returned multiple results for the same name. However, using university alumni records for comparison, this method was somewhat effective in locating CaRES alumni. Often we were able to obtain a telephone number from these searches and a subsequent phone call could be made to request an email address. This method is moderately recommended, but only if other information about alumni is available for cross-reference.

A summary of the methods discussed above is given in Table 2. “Technique” describes what was actually done; “Time Spent” is classified as little, moderate, or extensive and defined as the time spent using that method for each individual search; “Usability” is categorized as easy, moderate, or difficult and defined as the ease of using that method; “Results” is classified as low, moderate, or high, and defined for each method as the estimated ratio of CaRES alumni successfully found to those searched for; “Comments” offers the researcher’s notes on the method; and “Recommendations” is categorized as low, moderate, or high. The table is arranged in order of effectiveness, with the most effective methods listed first.

Table 2.

Summary of methods used for longitudinal tracking of alumni of the short-term cancer research program CaRES at the University of Alabama at Birmingham

Method	Technique	Time Spent	Usability	Results	Comments	Recommendations
Home Institution Electronic Directory	Entered names of CaRES alumni	Little	Easy	High	Most efficient and effective method	High
Personal Contacts	Looked through records; contacted mentors, professors and requested they contact CaRES alumni	Extensive	Moderate	High	Time consuming but offers high return	High
Search Engines	Entered names and degrees of CaRES alumni to “Google”	Little	Easy	Moderate	Best if used in conjunction with alumni records	Moderate
Social Networking Sites	Entered names of CaRES alumni; sent messages to potential matches	Moderate	Moderate	Moderate	Time consuming with only moderate return	Moderate
Outside Institution Websites	Using information from alumni records, visited websites searching for CaRES alumni	Extensive	Difficult	Moderate	Time consuming and should be used in conjunction with alumni records	Low
Alumni Records	Searched university alumni records for CaRES alumni	Moderate	Moderate	Low	Not advised for use on their own, but provide helpful starting point and a good cross-reference tool	Low

Discussion

Longitudinal tracking is very important for short-term research programs. Tracking is essential for evaluation, determining long-term outcomes, and obtaining further funding to continue the program. Short-term cancer research training programs must maintain accurate and comprehensive records of program participants from the start of the program, including phone numbers, addresses, and email addresses. Periodic updates should be performed to ensure possession of program alumni’s most up-to-date information. Various techniques exist to aid in tracking former participants. In this paper six methods for tracking are presented. However, the recommended course of action varies according to a particular program’s needs, resources, and time.

If time constraints are an issue, we recommend focusing on the home institution directory and search engines. These methods were the least time-consuming and provided a substantial return rate. If time allows, search engine results can be compared with alumni records to provide further confirmation. However if time is abundant and one is looking for the absolute highest return possible, we recommend employing the home institution directory, searching other institution sites (as directed by alumni records), and using personal contacts, all of which demonstrated success in contacting program alumni. Some institutions will be limited by lack of resources such as a comprehensive university directory or detailed alumni records. In this event, we recommend that trackers use search engines and personal contacts, which will provide the most comprehensive approach to locate program participants of years past, employing both a broad and a tailored method.

The results of this study must be considered in light of some limitations. Our searches were performed under a time deadline, and therefore some methods that could have been prolific may have been dropped prematurely. Also, because not originally designed as a study, the searches were not performed in a uniform manner, which may have resulted in some inconsistency. Once a CaRES alumnus was located, no further attempts were made to search for him or her using other methods; therefore, it cannot be determined whether some alumni could or would have been located using other methods. However, the outcomes reported here remain an excellent indication of the nature of these tracking methods, their usability, and a general measure of their success in locating program alumni.

This study sheds light on one of the most pressing issues in sustaining research training programs for students—how to perform longitudinal follow-up and determine long-term outcomes of the program. This follow-up can be vital to the sustainability of such programs. Future work evaluation research would ideally focus on the potential of creating and maintaining feedback systems for program alumni to continually inform, providing updates of employment, research activities, and other information. The results and information provided here may be of interest to administrators currently running these programs, those in the process of development, and program alumni who wish to foster the tradition of student research training by one day directing their own cancer training program.