An Evaluation Methodology for Longitudinal Studies of Short Term Cancer Research Training Programs

Abstract

The need to familiarize medical students and graduate health professional students with research training opportunities that cultivate the appeal of research careers is vital to the future of research. Comprehensive evaluation of a cancer research training program can be achieved through longitudinal tracking of program alumni to assess the program’s impact on each participant’s career path and professional achievements. With advances in technology and smarter means of communication, effective ways to track alumni have changed. In order to collect data on the career outcomes and achievements of nearly 500 short-term cancer research training program alumni from 1999–2013, we sought to contact each alumnus to request completion of a survey instrument online, or by means of a telephone interview. The effectiveness of each contact method that we used was quantified according to ease of use and time required. The most reliable source of contact information for tracking alumni from the early years of the program was previous tracking results; and for alumni from the later years, the most important source of contact information was university alumni records that provided email addresses and telephone numbers. Personal contacts with former preceptors were sometimes helpful, as were generic search engines and people search engines. Social networking was of little value for most searches. Using information from two or more sources in combination was most effective in tracking alumni. These results provide insights and tools for other research training programs that wish to track their alumni for long-term program evaluation.

BACKGROUND

The importance of familiarizing rising medical students and other healthcare professional students with research careers is essential for research progress. The number of healthcare students who follow a research path has declined [1, 2]. Some authors characterize physician-scientists as an endangered species [3–4]. In some universities research projects are unavailable, while medical students at other universities are unaware of ongoing research activities at their institutions. Barriers that may discourage student involvement in research include faculty disinterest, uncooperative mentors, minimal incentives, lack of funds, time constraints, perception of an overwhelming career workload, and concerns about challenges faced while engaged in research during academic training years [5–6]. Further, some students believe that research training has no value for achieving career goals, and that it will detract from clinical practice experiences [7]. A study by Nikkar et al conducted in the United Kingdom found that many students who engage in research do so only to improve their curriculum vitae, believing that research experience will increase their chances of selection for competitive clinical training programs [8]. These attitudes and beliefs may arise from a poor understanding of the contributions that research makes to career development and to the practice of medicine and public health.

Contrary to widely-held beliefs that research experience does not offer advantages for students in clinical training programs, engagement in research has numerous and varied benefits [9]. Medical students may not understand translational science and believe that “research” implies basic science laboratory research. Discouraging student interest in research may hinder them from comprehending and applying medical advances that would help their patients. Healthcare professionals who have participated in scientific programs during their student years have an advantage over their peers when making decisions based on research results, as they can make more accurate diagnoses and better professional decisions [10]. Having research skills not only makes a student more competitive when applying for further training; it also increases options for future employment [11, 12]. Medical research enhances critical thinking, improves the quality of training, provides new skills, and sensitizes students to health-related social issues [13].

Early exposure to research can impact the career paths of students [14]. Federal and privately funded scholarships, fellowships and student loan repayment programs are incentives to engage students in research [2, 15]. Summer research scholar programs offered by academic and federal institutions may inspire students to pursue careers in basic, clinical, translational, community, global, epidemiology, policy, and humanities research [16]. However, additional efforts must be made to elevate research productivity through mentored student research training programs, a proven approach to increasing the number of research-oriented healthcare professionals [17].

The National Institutes of Health provides funding for research training programs through R25 grants available in 27 of its institutes including the National Cancer Institute (NCI) [18]. In 1988 the University of Alabama at Birmingham (UAB) joined in the effort to produce more cancer research opportunities for students with the funding of an R25T grant called the Cancer Prevention and Control Training Program (CPCTP) which has supported over 120 post-doctoral fellows and doctoral students [19, 20]. Through 1998 the CPCTP also supported a few short-term internships each summer but it could not meet the demand. In 1999 UAB’s Cancer Research Experiences for Students (CaRES) program was funded by NCI through its R25E program in order to meet the demand for short-term cancer research projects at UAB [21].

CaRES has awarded over 580 summer internships in cancer research for UAB medical students and graduate students from public health, dentistry and nursing who are American citizens or permanent residents of the U.S. The CaRES program’s primary goal is to introduce eligible students to cancer research and to motivate them to consider cancer research as a career option. Each CaRES project lasts from eight to twelve weeks, according to the demands of the project and the availability of the preceptor and student. A mid-summer evaluation is completed by the interns to identify any issues of concern so that corrective actions can be taken promptly. At the end of the summer these students complete a detailed exit evaluation with Likert scale questions and the opportunity for commentary. Preceptors submit a parallel questionnaire to complete the short-term evaluation of the program for that year. The short-term evaluation results for the past several years are available on our program website, www.uab.edu/cares. Since its inception, CaRES has met its short-term goals as documented by outstanding exit evaluations by students and preceptors; and by the number and quality of publications, abstracts, and posters co-authored by CaRES interns with their preceptors.

An equally important question is whether CaRES is meeting its long-term goal of motivating students to pursue cancer research careers. Such a determination can be made only by means of longitudinal tracking of program graduates (“alumni”) to assess their career choices and to catalogue their professional achievements such as cancer research publications, clinical care of cancer patients, or other cancer-related activities such as advocacy and administration of cancer prevention and control programs. For some training programs longitudinal tracking is performed every five years in connection with submission of five-year training grant renewal proposals. Alternatively, tracking can be done on a continuous basis, but this can be especially time-consuming and tedious. Program alumni from recent years are typically easy to locate, whereas tracking alumni from many years ago can be challenging. Another challenge is that, once located, alumni must be persuaded to complete and submit the desired information on career status and achievements.

With emerging technology and consistently evolving means of communication, the most effective ways to track alumni have changed over time. In 2009 CaRES tracked its program alumni who participated from 1999 to 2008 [21]. In preparation for tracking in 2014, we reviewed our 2009 methods, but with additional consideration of social changes in media use and technology and resources not widely available in 2009. Our data collection in 2009 was quite complete (97% response rate) but required two part-time staff members working over a six-month period to acquire the desired data on about 320 CaRES alumni. Are better methods available today? The purpose of this paper is to describe and evaluate the methods we used in summer 2014 to longitudinally track CaRES alumni who participated from 1999 to 2013, and to describe our means of data collection on their current career activities and outcomes.

METHODS

Description of the cohort

Over a period of six weeks in summer 2014 we attempted to longitudinally track all 499 alumni who participated in our CaRES program in any summer from 1999 to 2013. The proportions of CaRES alumni according to academic program were 57% from the UAB School of Medicine, 29% from Public Health, 7% from Dentistry, and 7% from Nursing and other graduate health professional programs. The first goal, and the focus of this paper, was to obtain current contact information (i.e., an email address, telephone number, or physical address) for these alumni that could be used to solicit data from them. The second goal was to collect data on career outcomes (most important, engagement in cancer research) from these alumni in order to assess our program’s effectiveness in motivating students to engage in cancer research. The outcome data we collected are presented in a separate publication [22].

Survey design

CaRES program faculty and staff convened to decide what type of career outcome information was most valuable for program evaluation purposes, and how to most effectively acquire this information. Obtaining the Curriculum Vita (CV) of each program alumnus was considered as a first option. However, because CV’s may be incomplete and outdated, and may not include the specific information desired, we decided not to use this approach as a primary data source. The working group decided to design a customized survey instrument for collection of evaluation metrics of interest, with the goals that the instrument be clear, concise, and targeted. The instrument includes demographic information, contact information, current place of employment and job title, education and training status, cancer research activity, cancer publication citations, clinical care activities, and descriptions of cancer-related service activities such as advocacy and community presentations (Figure 1). This questionnaire is very similar to the questionnaire utilized for tracking in 2009, with the addition of items about experiences with the CaRES program and attitudes toward recommending CaRES to current students.

Figure 1.

CaRES Tracking Questionnaire

Note. The tabulated questions were conditional and would be asked based on the answers form the alumni

Webform data entry

The decision was made to use a webform posted on the CaRES program website as the primary means of data collection and data entry. The platform used was Webform Module (with MCRYPT encryption) for the Drupal Content Management System, and the database engine was MySQL [23]. Once contacted, an alumnus would be directed to our website to click on a link to access the tracking questionnaire. Once completed, survey responses would be directly uploaded into our secure, encrypted database. This approach eliminates the data entry step and associated sources of error. The website manager could query the database in real time for statistical analyses and overall results such as percentage of completion for each survey item as well as the distribution of item responses.

Telephone interviews for webform data entry

An alternate means of completing the tracking questionnaire was made available for individuals who did not wish to access the web-based version. A telephone version of the instrument allowed a program staff member to interview the CaRES alumnus and enter responses directly into the web-based version (Figure 2). Prompts were added to the questionnaire to aid the interviewer and to make the process user-friendly. Program staff members conducted pilot interviews to assess the flow of questions and answers and to ensure that each interview would require only a few minutes. Algorithms were devised in order to extract details about any items answered “Yes,” whereas items answered “No” allowed the interviewer to skip to the next question. The telephone-collected responses were entered directly into the online questionnaire webform by the program staff interviewer, allowing telephone interview data to merge with the electronically collected data. The ongoing monitoring of questionnaire completion allowed our website manager to discern which respondents submitted their data on the webform, and which responses were submitted from staff members’ computers, implying that a telephone interview was conducted. This approach allowed comparisons of webform versus telephone responses and assessment of any systematic differences in responses.

Figure 2.

Telephone Adaptation of CaRES Tracking Questionnaire

Tracking approaches

The 2009 methodology for tracking our program alumni was used as a basis for our tracking study in 2014; but additional methods, utilizing newer technology, were employed. The sources of contact information that we used were previous tracking records, alumni society records, personal contacts, search engines, people search engines, and social media. All contact information collected, regardless of source, was accessible only by program faculty and staff from secure computers and was not shared with individuals outside the CaRES program. Institutional Review Board (IRB) approval was not required for accessing contact information on alumni for the purpose of gathering data on their professional roles and accomplishments.

Statistical analyses

The primary endpoint used in this study was the overall tracking response rate. Secondary endpoints typically used in survey research are the proportion of usable responses for each survey item. For our study such calculations are unnecessary because the response to every survey item (other than our optional requests for a CV and for information about other CaRES alumni) was required for the electronic submission to be accepted. A qualitative assessment on time spent and ease of use was performed for each tracking algorithm. We classified time spent as “extensive” (slow progress requiring repeated attempts, typically over many days), “little” (quick and straightforward results, taking seconds or only a few minutes) and “moderate” as results in between these endpoints (hours, not days or minutes). For ease or difficulty of using each data source, we used a scale of “difficult” (data unclear or contradictory, difficult to link, or difficult to acquire), “easy” (information of interest immediately accessible with minimal mouse clicks), and “moderate” as results in between these extremes.

RESULTS

By a combination of methods we obtained current contact information for 100% of the 499 CaRES alumni who participated in the program in 1999–2013; and we received complete tracking questionnaire data from 481 (96.4%) of these individuals [22]. Three-quarters of the successful contacts were made by email, and the remainder made by telephone. Our email messages provided step-by-step instructions about how to visit the CaRES website and where the questionnaire was located within it, in order to reassure the recipient of our identity and reason for contacting them. Our contact information was included in our email message, in the event that respondents needed technical support or wished to submit information over the telephone. As we began tracking, email requests for questionnaire completion were sent every three to five days and were not personalized. As time progressed and we had fewer and fewer untracked alumni remaining, we personalized our email messages, resulting in faster responses.

Despite repeated requests, 18 alumni failed to complete the online questionnaire or grant a telephone interview. These individuals may have chosen not to respond to our email messages because of distrust of chain emails or fear of contracting a software virus. For some recipients our emails may have been labeled as spam. We know that two of the 18 individuals accessed the online survey, had difficulty submitting it, and then emailed CaRES program staff for assistance. Despite our prompt response to their requests for assistance they did not return to the survey to complete it and submit it, even after further email prompts on our part.

Of the 481 (of 499) program alumni who completed the tracking survey either online or by telephone, the proportion of usable responses per survey item was 100% (by definition) for required items indicated by a red asterisk in the on-ine survey. Even for the 24% of alumni interviewed over the telephone, the CaRES staff interviewer completed the survey online while on the telephone, and could not submit it unless all required questions had been answered.

When compared with our previous tracking percentage (97.8% of 320 alumni tracked in 2009), our 2014 results are virtually identical. However our 2014 tracking study took only six weeks of part-time effort by two staff members, whereas the 2009 tracking study took almost six months of part-time effort by two staff members. The utility of our methods for tracking are summarized in Table 1 and the strengths and weaknesses of the methods are detailed below.

Table 1.

Sources of Tracking Data and Their Utility

Source of Contact Information	Technique	Time Spent	Easy or Difficult?	Yield of Results	Comments	Recommendations
Previous tracking records	Abstract data from spreadsheets and searchable tracking databases	Little	Easy	Great for email addresses; moderate for phone numbers	Efficient and effective for the period of previous tracking	Good starting point
Alumni society records	Obtain access to alumni databases for each school involved	Extensive	Easy	Great for email addresses and phone numbers, especially for recent graduates	Access may be limited by university policy; contact information from early years may be outdated	Good second step if access is permitted; time-consuming but worthwhile
Personal contacts	Ask former preceptors and instructors for contact information for those alumni with whom they have remained in touch	Little	Easy	Great for email addresses; moderate for phone numbers	Great when it works, but former preceptors and instructors may be hard to contact and many have lost touch with their former students	Worthwhile when preceptors and instructors still work at the home institution or are otherwise accessible; typically contact information they have is current
Search engines	Enter names and degrees of alumni into “Google” and “Hipaaspace”	Extensive	Moderate	Moderate for email addresses; low for phone numbers	Best if used in conjunction with alumni record data on middle name or employer	Do not use unless approaches above have failed; time-consuming and tedious
People search engines	Visit websites of probably employers to find alumni contact information	Extensive	Difficult	Low for email addresses; moderate for phone numbers	Alumni record data needed in order to know where to search; the correct website may not have contact information	Do not use other than as a last resort; time- consuming and lots of time may be spent finding the wrong person
Social networking sites	Enter names of alumni and send message to potential matches	Extensive	Moderate	Low for email addresses and for phone numbers	Time-intensive; frustrating for common names that have many matches	Can be entertaining but correct results unlikely in most cases; suggest avoiding this method

Previous Tracking Records

Although a major advantage for 2014 tracking was that we had records from the 2009 study, current contact information was scarce for alumni from the earliest years of the program. An email list was extracted from the 2009 alumni tracking spreadsheet and an email message requesting participation was sent to each alumnus. This method was effective in that active and inactive email addresses were quickly identified. Many email addresses for older alumni were outdated, but telephone numbers extracted from this source tended to be current. Searching through the previous tracking data was time-consuming but somewhat valuable because the data were current and easy to use. The alumni society records (described below) made this method more effective.

Alumni Society Records

Meetings were held with Student Alumni Affairs directors from the Schools of Medicine and Public Health to request access to their alumni databases. This action did not require IRB approval because the purpose was to locate alumni and collect data on their current positions and career achievements. The directors were cooperative and provided us with the contact information that they had on record for our CaRES alumni. Email addresses were compiled and an initial confirmation email was sent with an invitation to complete the online questionnaire. After several submissions software was used to filter our email address database that was updated as responses were submitted. Because university-based student email addresses remain active for only one year after graduation, personal or workplace email addresses were needed for contacting alumni from the early years. In addition to email addresses we had previously collected telephone numbers on CaRES students and stored them in our database. For many alumni these numbers were no longer in service, but this could be determined only after making a telephone call, with only about 40 calls per day manageable for a part-time staff member. For CaRES alumni who were medical students, residency matching results and clinical specialty information provided to us by the School of Medicine was valuable because it contained leads to institutions or employers that helped us to track our alumni by means of search engines (as described below).

Personal Contacts

The personal contacts of greatest value were those involving former CaRES preceptors who had continuing relationships with their former CaRES students. Our database of past CaRES projects identifies each preceptor and a check of our University electronic directory identifies whether a faculty member is still employed at the university. Many preceptors had no information about their former CaRES students, whereas other preceptors knew exactly where their former students were and gladly provided us with contact information. Former academic advisors and faculty acquaintances were contacted ad hoc if we knew of the professional relationships the alumni had when they were students. This method was not time-consuming and was effective immediately for select CaRES alumni.

Search Engines

A search engine generally provides only a lead to a possible contact, and rarely a viable telephone number or email address. Google (http://www.google.com) is the most commonly used search engine on the world-wide web, handling more than three billion searches each day [24]. Google provided a massive amount of information that was broad and tedious to sort through. However, any additional piece of information it provided, such as a middle name or middle initial, narrowed down our search and allowed us to find some alumni. Following each lead was an added step. Reaching an alumnus by telephone at their workplace typically required a call to the main office of a university, clinic, or a company, with a request to be connected to the alumnus. This process was time-consuming and effective only in instances when “gatekeepers” connected our call or passed on our message. We also utilized a website called Hipaaspace (Http://www.Hipaaspace.com) that has limited information on all licensed healthcare professionals, with the location of their practice, contact information, and sometimes a personal telephone number [25]. This was a useful search engine for tracking physicians and dentists, accounting for about two-thirds of CaRES alumni. But determination of location alone did not guarantee personal contact or completion of the tracking questionnaire.

People Search Engines

We tried using specific “people search” engines to track our alumni. In descending order of their utility they are: zabasearch.com, switchboard, ZoomInfo, Yellowpages, USSearch.com, SpyFly.com, Petrieve, Spock, Pipl, Wink, yoName, and Yahoo! People search. A key to obtaining correct results is knowledge of the current state of residence, information that we typically did not have. This was a barrier to using this method. The free information available typically included telephone numbers that required repeated callbacks to verify whether our alumnus was indeed there. If the subject of our search was female, it was useful to know the spouse’s name, because the telephone number of the current residence was typically registered in the husband’s name. Most of these engines claimed to have additional information, but only for a cost of $1 to $40. We did not pay to obtain information from these sources, as our funds were limited, and we were not confident of usable results. These sites sometimes provide contact information for relatives (e.g., parents or siblings) who may be willing to provide contact information for the alumnus, or who may be willing to ask the alumnus to contact our training program or access our website. This method is not time-efficient, but could produce some usable leads if time and funds are available for maximum tracking efforts.

Social Media

Social media have become very popular. The social networks we used were Facebook and LinkedIn. As of 2012 Facebook has one billion active users and the numbers continue to increase [26]. Finding our alumni via Facebook was difficult because a username is not always the individual’s given name; profiles may be listed by nicknames; and it often remains uncertain that you are contacting the intended person. Social media searches are difficult and unreliable because social media users are world-wide and many have similar usernames. Even if we found a subject, we would typically have no connections or “friends” in common. This is problematic because Facebook does not permit a “non-friend” to freely inbox or message the user. To avoid spam and fraudulent contacts, there is a charge to message subjects with whom you have no common contacts. Because the cost was nominal ($1.00) we paid to inbox about 40 suspected CaRES alumni through this method. Once one dollar is paid, sending additional messages is free. Messenger on Facebook will verify whether the subject saw a message, and at what time and date. We received few responses to messages that we placed. This may be because Facebook is viewed as an informal means of communication, making messages from a university appear as spam to some recipients. In summary, Facebook searches were time-consuming and cost-ineffective. LinkedIn is a professional social network that invites users to create a profile to build professional relationships. It has 259 million users as of 2013 [27]. Compared to Facebook, it is more likely that the subject will have the same username as their given name. But LinkedIn provided too many false results to be usable. Because we had no “connections” in common with the subjects of our searches, their privacy settings determined whether they would allow us to send them an inbox message. In summary, LinkedIn was time consuming and generally ineffective, but provided a few leads to employers or other contacts. It may be more useful for finding non-physician health professionals such as public health graduates.

DISCUSSION

Following former trainees (“alumni”) of a training program for evaluation purposes is a time-consuming task and a resource-intensive process. Tracking requires several individuals working together and comparing notes, striving for consistency in methodology to locate, and then to motivate, individuals to submit their information electronically or answer questions by telephone. Our best sources of contact information for tracking CaRES alumni from the early years was our previous tracking results; and for later years, the institutional alumni records that provided email addresses and contact phone numbers. Personal contacts (generally, former preceptors) were sometimes helpful, as were search engines and people search engines. Social networking methods remained of only low to moderate value (as was true in 2009), even though the number of users has increased steadily over the years. In order to increase response rates, for some individuals several methods of contact were employed. For those alumni who received requests through more than one approach, the most effective method cannot easily be identified. We were unable to discern which method finally motivated the alumnus to access the website and complete the tracking questionnaire.

Female alumni were harder to track than male alumni because of name changes due to marriage or divorce. Public health students were the most difficult type of student to track as they have a wider variety of employers than do physicians, dentists, and nurses, and they have no clinical license. This should be taken into consideration for programs wanting to track alumni who are not licensed in their professions. Clinicians are easy to locate if their workplace telephone number can be found; however not all alumni who are found will return phone calls or agree to furnish data. Typically messages left with nurses or receptionists were not returned.

Posting our tracking questionnaire on our website was a key to our high and rapid 2014 response rate. In 2009 we had sent questionnaires via email with the request for completion and return to us via email. With the “attachment to email” method, respondents tended to skip some questions, scan their form and email it back to us, often with incomplete answers. With the online questionnaire approach, critical fields are required or the questionnaire will not electronically submit. The 2014 tracking was 96.4% complete for 499 individuals, virtually identical to our 2009 tracking rate of 97.8% for only about 320 individuals (many of whom were still in school). This rate far surpasses standards for the minimally acceptable response rates of 70% for an online survey and 80% for telephone survey [28]. On-line respondents and telephone interview respondents did not differ in the nature of their responses about the effect that CaRES had on their career choices. However, online survey respondents provided more CVs than did telephone respondents. It should be appreciated that for any large group to be tracked, a 100% completion rate for any survey may be unrealistic, in that some individuals are too busy to respond, while a few others may have negative feelings that they choose not to share. However we have shown that response rates as high as 96% are possible for a group as large as 500 individuals.

There are limitations to the generalizability of our results. The process used to obtain alumni information was exclusively designed for the CaRES program at UAB and may not apply to other institutions or their training programs. Each institution should assess their tracking needs and resources, and consider the population to be tracked, according to age, gender and professional field. Other imitations of this study are a very limited budget and time constraints that ruled out the use of additional tracking methods that were costly or protracted.

CONCLUSION

Evaluating the long-term success of a cancer research training program can be undertaken by longitudinal tracking of alumni that provides data on career choices and professional productivity. For efficiency, the program faculty and staff in charge of tracking should carefully consider the characteristics of the population to be tracked, and prioritize their tracking methods in order maximize return rates in the shortest possible time. Tracking at regular intervals, using metrics appropriate for program planning and evaluation takes advantage of new technology and builds a sustainable database. The methods that proved effective for CaRES may not be universally useful, but may provide insights, efficiencies, and tools for other training programs that wish to track their alumni for long-term program evaluation. We encourage training programs to continuously update their alumni information, in order to adequately document career outcomes and achievements, and make progress reports and renewal proposals easier to assemble.