Improving clinical and translational research training: a qualitative evaluation of the Atlanta Clinical and Translational Science Institute KL2-mentored research scholars program

Abstract

A major impediment to improving the health of communities is the lack of qualified clinical and translational research (CTR) investigators. To address this workforce shortage, the National Institutes of Health (NIH) developed mechanisms to enhance the career development of CTR physician, PhD, and other doctoral junior faculty scientists including the CTR-focused K12 program and, subsequently, the KL2-mentored CTR career development program supported through the Clinical and Translational Science Awards (CTSAs). Our evaluation explores the impact of the K12/KL2 program embedded within the Atlanta Clinical and Translational Science Institute (ACTSI), a consortium linking Emory University, Morehouse School of Medicine and the Georgia Institute of Technology. We conducted qualitative interviews with program participants to evaluate the impact of the program on career development and collected data on traditional metrics (number of grants, publications). 46 combined K12/KL2 scholars were supported between 2002 and 2016. 30 (65%) of the 46 K12/KL2 scholars are women; 24 (52%) of the trainees are minorities, including 10 (22%) scholars who are members of an underrepresented minority group. Scholars reported increased research skills, strong mentorship experiences, and positive impact on their career trajectory. Among the 43 scholars who have completed the program, 39 (91%) remain engaged in CTR and received over $89 000 000 as principal investigators on federally funded awards. The K12/KL2 funding provided the training and protected time for successful career development of CTR scientists. These data highlight the need for continued support for CTR training programs for junior faculty.

INTRODUCTION

The number of junior faculty scientists including physicians, PhD, and other doctoral-trained faculty performing clinical and translational research (CTR) has diminished in recent decades.^1–3 Multiple obstacles to conduct CTR exist; these include fragmented institutional infrastructure; lack of qualified CTR investigators and senior mentoring; poor collaboration between clinical, translational and basic science investigators; limited implementation of interdisciplinary research, including lack of team science credentials in the academic promotion process, insufficient research funding, and challenges related to the balance of clinical care and research responsibilities among physician scientists.^4–6 Translation of scientific advances to improve health in the community is unacceptably slow.^3,7 For example, a recent review of the literature showed that studies report an average of 17 years is required for specific positive research evidence to be implemented into clinical practice.⁸ Several translational blocks have been well described, including the lack of an adequate number of well-trained CTR physician and other doctoral investigators.^7,9

Significance of this study.

What is already known about this subject?

• A major impediment to improving the health of communities is the lack of qualified clinical and translational research (CTR) investigators.

• National Institutes of Health (NIH) developed mechanisms to enhance the career development of physician, PhD, and other doctoral junior faculty CTR scientists including the CTR-focused KL2 program.

• The KL2 program provides protected time for mentored and didactic research training for junior faculty members in order to increase the number of CTR investigators.

• Disseminating program evaluation results from the existing KL2 programs can influence and improve subsequent initiatives to support junior faculty scientists.

What are the new findings?

• The KL2 provision of protected time allows junior faculty in the program to successfully focus their career development on CTR.

• Participants in the Atlanta Clinical and Translational Science Institute (ACTSI) KL2 program gain a range of critical skills and knowledge through didactic coursework and a mentored research project.

• Mentors play an influential role in the scholars’ idea generation for research studies, study planning and design, and review of grant drafts and manuscripts.

• Participants from the ACTSI KL2 program are successful at becoming independent investigators: 91% remain engaged in CTR and the scholars have received over $89 000 000 as principal investigators on federal grants.

How might these results change the focus of research or clinical practice?

• The results from this study inform the development of training programs for junior faculty who are clinical and translational scientists. The evaluation of the ACTSI KL2 program demonstrates the need for funding mechanisms such as these to provide protected time for junior faculty to acquire research skills, complete a mentored research project, and apply for additional funding. The KL2 program builds important research capacity in CTR and supports new, innovative investigators working towards improving local, national, and global health.

To address this workforce shortage, the National Institutes of Health (NIH) has provided funding mechanisms to help stimulate training of clinical and translational scientists. In 1998, the NIH created the K30 programs, which provided support to academic institutions to develop and maintain the infrastructure necessary to carry out mentored CTR training and also responded to the need to provide more sophisticated didactic and mentored training for clinical investigators.¹⁰ The NIH National Center for Research Resources (NCRR) Mentored Clinical Research Scholar (CRS) Program Award (K12) initiative was started to provide support to academic institutions for mentored CTR training, especially for physician scientist investigators. The goal of this nascent program was to increase the number of trained clinical and translational investigators. In 2006, the Clinical and Translational Science Awards (CTSAs) were established to create an academic home for CTR and to train the future cadre of transdisciplinary scientists.^11,12 The K30 and K12 programs were incorporated into the CTSAs; the CTSA program to date has mandated the KL2 scholars program, which has similar goals as the K12, although the target population was expanded within the KL2 program to include postdoctoral trainees and junior faculty with PhD and other doctoral degrees in addition to physicians. The CTSA program is designed to strengthen and support the entire spectrum of CTR from scientific discovery to improved patient care. CTSAs are issued by the NIH National Center for Advancing Translational Science (NCATS), which currently supports a national consortium of about 62 medical research institutions that work to speed the translation of research discovery into improved patient care.¹³ The Research Education, Training and Career Development (RETCD) oriented cores of the CTSAs are designed to build the research workforce by training qualified clinical and translational investigators, reducing the barriers for obtaining training, and improving the mentoring of junior clinical researchers.^4,7,14,15

Several studies have previously addressed the evaluation of CTSAs.^16–20 However, there are a limited number of reports that focus on the impact and accomplishments of the CTR training programs for junior faculty^21–23 and few studies to date have specifically focused on the evaluation of the accomplishments of the K12 or KL2 program.²³ The KL2 program addresses the recommendations of the 2000 National Research Council report, Addressing the Nation’s Changing Needs for Biomedical and Behavioral Scientists, which recommends intensifying efforts to train and retain clinical and translational researchers in order to reverse the dramatic decline of clinical and/or translational investigators entering the research workforce.

The purpose of our study was to evaluate the impact of the NIH National Center for Research Resources (NCRR) Mentored Clinical Research Scholar (CRS) Program Award K12 and its successor, the ACTSI KL2-Mentored Clinical and Translational Research Scholars program. Further evaluation of the CTSA KL2 program is warranted to assess the value of this clinical research training initiative and determine the strengths and weaknesses in the structure and implementation of the program. This evaluation assesses the impact of the ACTSI K12/KL2 program on the careers of participating scholars, including determining skills gained throughout the program, assessing the mentoring relationships, investigating the program’s strengths and areas for improvement, and culling lessons learned that might be applicable to other K programs.

Program description

The NIH NCRR Mentored CRS Program Award K12 program was initiated at Emory University (Emory) in September 2002 and focused on didactic and mentored clinical research training for junior faculty physicians at Emory University interested in careers that encompassed clinical and translational science. In September 2007, Emory University was awarded a CTSA, entitled the ACTSI, in a consortium with Morehouse School of Medicine (MSM) and Georgia Institute of Technology (GT). The free standing NIH K30 award (which supports the ACTSI Master of Science in Clinical Research (MSCR) program that provides didactic training for K12 and KL2 scholars) and the K12 were incorporated into the ACTSI RETCD program which was one of nine components or ACTSI cores. The ACTSI KL2-Mentored CTR Scholars program provides didactic and mentored CTR training to ACTSI junior faculty with a doctoral or medical degree (MD or PhD) who are committed to a career in clinical and/or translational research. The ACTSI KL2 program provides salary support to protect 75% professional effort for research and research training for junior faculty and a $25 000 technical budget for research costs including tuition for the MSCR. KL2 scholars are selected through a competitive application process. They submit an NIH-style K23 or K01 research proposal, have a lead mentor who is a federally funded, established, and a successful clinical and/or translational investigator, and most importantly, committed to be their mentor. Support in the program is provided for up to 2 years.

METHODS

The ACTSI KL2 evaluation measures program goals and outcomes using a mixed-methods approach that includes quantitative measures (KL2 scholar demographics, academic appointment, publications, grants received, and a semiannual progress report during the 2 years of formal training) in addition to a qualitative component that consists of interviews with KL2 participants following completion of the program. Metrics are inventoried in a program database from all former and current K12/KL2 scholars. Publications by former and current K12/KL2 scholars are tracked by monthly searches in PubMed. NIH funding as a principal investigator and/or program director is tracked by review of Research Portfolio Online Reporting Tools (RePORT) Expenditures and Results Tool²⁴ on a 6-month basis. Non-NIH federal funding and non-federal funding is tracked by obtaining updated curriculum vitae from former trainees at the end of their formal K12 or KL2 support and thereafter on an annual basis.

For this study, qualitative interviews were conducted with former scholars to assess the impact of the K12/KL2 program on their career path. The purpose of qualitative research is to capture rich description and context about participant experiences as it allows for more in-depth exploration of important evaluation themes and the results can be transferable to similar study populations.^25,26 Qualitative data are not intended to be generalizable but rather provide meaning about experiences that are lost in quantitative measures.^27,28 Scholars who completed the Emory-based K12 or ACTSI-based KL2 program were recruited for one-on-one interviews (n=20). The evaluators used purposive sampling to recruit participants from each cohort to collect data about the program over time.^29,30 The RETCD program evaluators (CE, AF, DLC) conducted the qualitative interviews either in person or by phone, using a standardized interview guide.

Instrument

The interview guide was developed to collect information about the attainment of program goals and objectives (figure 1). After the evaluators developed the interview guide, the K12/KL2 program directors (HMB, TRZ) reviewed the guide and provided feedback. Revisions were made to capture data about important evaluation domains. The guide included both open-ended and close-ended questions to assess knowledge and skills gained, mentoring relationship, the MSCR coursework, the impact of the K12/KL2 program on the scholars’ career development, and areas for program improvement. Interviews were conducted from June 2007 to December 2014. Interviews were digitally audio-recorded and transcribed verbatim.

Figure 1.

Qualitative interview guide for the evaluation of the Atlanta Clinical and Translational Science Institute (ACTSI) K12/KL2 scholars program, sample questions.

Analysis

For the qualitative interviews, the researchers independently reviewed the transcripts and identified codes. Codes were terms that labeled issues, topics, and ideas that emerged in the data.²⁵ Deductive codes were known a priori to the analysis based on themes from previously published studies on similar evaluations and were derived from the topics included in the interview guide. Inductive codes captured new themes that emerged from the data during the analysis process.²⁵ Two authors (AF and DLC) reviewed the transcripts for major themes and selected exemplar quotes to address the evaluation goals. The researchers compared coded passages and reached a consensus about the content. For quantitative data (eg, publications, funded grants, etc) descriptive statistics were used to summarize the scholars’ accomplishments. The Emory Institutional Review Board (IRB) issued a letter of determination stating that the evaluation did not require formal IRB review because data were collected as part of standard quality assurance procedures.

RESULTS

To date, 46 junior faculty members have been supported by the K12 and KL2 programs including the current 3 KL2 scholars (Emory=40, MSM=4, GT=2, and 1 of the 2 GT trainees matriculated within the joint GT-Emory Department of Biomedical Engineering program). Thirty (65%) of the 46 K12/KL2 scholars are women; 24 (52%) of trainees are minorities, 10 (22%) of which are members of an underrepresented minority group as defined by the NIH (table 1). Beginning with the KL2 program, PhD faculty were eligible for program support. Eight (29%) of 28 KL2 scholars have been PhD-level faculty including 1 scholar with both MD and PhD degrees. The 46 K12 and KL2 scholars derived from diverse academic departments: biomedical engineering, epidemiology, human genetics, internal medicine (eight divisions: cardiology, endocrinology, general medicine, geriatrics, hospital medicine, infectious diseases, pulmonary and critical care medicine, rheumatology), neurology, otolaryngology, pediatrics (five divisions: gastroenterology, general pediatrics hematology/oncology, pediatric infectious diseases, pulmonary and critical care medicine), psychiatry, radiology, and surgery.

Table 1.

Demographics of ACTSI KL2/K12 scholars, 2002–2016 (n=46, including current scholars)

Characteristics	n (%)
Gender
Female	30 (65)
Male	16 (35)
Race/ethnicity*
White	22 (48)
Minority	24 (52)
Underrepresented minorities†	10 (22)

^*Total equals >100% because underrepresented minorities are included in the minority category.

^†As defined by the NIH this includes Blacks or African-Americans, Hispanics or Latinos, American Indians or Alaska Natives, Native Hawaiians and other Pacific Islanders.

ACTSI, Atlanta Clinical and Translational Science Institute; NIH, National Institutes of Health.

Forty-three K12/KL2 scholars have completed the program or no longer receive support. Among the 43 K12/KL2 scholars, 39 (91%) remain engaged in CTR. This includes 36 scholars at academic institutions (professor=5, associate professor=18, assistant professor=16), 2 at the Centers for Disease Control and Prevention (CDC) with joint appointments at Emory University and 1 engaged in a leadership position in CTR in industry. Thirty-two (74%) scholars have received federal funding as a principal investigator (including three scholars who have fundable scores for a K award, funding pending; table 2). For the majority of the people in the program, the most appropriate federal award is a K but there are some scholars who are more advanced and prepared to go from the KL2 to an R series award. Twenty-four of the 32 scholars completed the program and moved directly to principal investigator on a NIH K series award (K23=19, K08=2, K01=2, K25=1). The mean number of days from completion of the program to receiving a K series award was 118 (range 1–1158 days, median=15 days). To date, 16 scholars have completed their K award and 6 of these scholars have since received an R award. Eight of the 32 scholars completed the program and moved directly to principal investigator on a larger award (R series=4, CDC=2, National Science Foundation (NSF)=1, Department of Defense (DoD)=1). Of the four scholars who went straight to an R series award after completion of the program, the mean number of days to receive the award was 404 (range 1–1188 days). Eleven scholars have received a total of 23 R series or R series equivalent awards (R01=13, R21=5, R equivalent/U awards=5). The range of R series awards per scholar is 1–6. All of the scholars who are a principal investigator on a U series award are also principal investigators on an R series award. In total, five scholars have received eight non-NIH federal awards (DoD=3, CDC=3, NSF=2). Overall, the NIH government-funded awards exceed $72 000 000 (total costs) and the non-NIH federally funded awards exceed $17 000 000 for a total of over $89 000 000 in federal funding (three K awards are pending funding and not included in this calculation). In addition, scholars have secured ~$13 million in non-federally funded awards through foundations and internal grants. The K12/KL2 graduates have published over 1137 articles in peer-reviewed journals (first author=386, last author=226) with an average of 27 articles published per scholar since the time they began the program through the current date, and an average of 5 articles per year per scholar. In addition, the K12/KL2 scholars have published 1 book and 84 book chapters. Several former K12/KL2 scholars have gone on to take leadership roles in the ACTSI RETCD program that includes the ACTSI/MSCR, KL2, and predoctoral TL1-funded programs. Seven former K12/KL2 scholars serve on the RETCD Executive Committee and/or as course director for one of the MSCR courses.

Table 2.

Number and type of federally funded grants awarded to ACTSI K12/KL2 scholars as principal investigators, 2002–2016

Award type	Number of grants awarded*	Number of unique scholars (PIs) who received award
Total K series†	24	24
K23	19	19
K08	2	2
K01	2	2
K25	1	1
Total R series (or equivalent)	26	11
R01	13	8
R21	6	6
R24	2	2
R56	1	1
U01	3	3
U10	2	2
Total non-NIH federal awards	8	5
Department of Defense	3	1
Centers for Disease Control	3	3
National Science Foundation	2	1

^*The total amount of federally funded awards exceeds $89 000 000 (total costs).

^†Includes three K series grants with fundable scores, pending funding.

ACTSI, Atlanta Clinical and Translational Science Institute; NIH, National Institutes of Health; PI, principal investigator.

Knowledge and skills gained in the program

The qualitative evaluation included interviews with 20 junior faculty who completed the K12 program or the subsequent ACTSI KL2 program. Through individual interviews, K12/KL2 scholars were asked to reflect on the knowledge and skills gained in the program, specifically from the mentored research training and didactic research training components of the K12/KL2 program which included the Emory-based MSCR curriculum. These MSCR courses included biostatistics and data management; bioinformatics; epidemiology; community engagement and health disparities in clinical research; clinical trial design and analysis; ethical, legal, and social issues in the responsible conduct of CTR; analysis of clinical research data; and scientific and grant writing.

Several respondents noted that the Emory MSCR program carefully tailored coursework to properly train scientists in CTR. Nearly all participants interviewed reported an increase in research skills, such as designing clinical studies, developing research questions, and critiquing the literature and methodology. Many trainees commented on the high quality of training through their coursework, and how it provided them with an opportunity to gain critical research skills not taught in prior medical or doctoral training. One scholar stated, the program “provides great skills in terms of epidemiology fundamentals, statistical fundamentals and how to utilize evidenced-based medicine. It really gives you all of the other building blocks that you need to be a clinical researcher/epidemiologist.” Another scholar explained, “I went [into the KL2 program] with the impression, ‘oh, I’m going to know this stuff’ but it was a different way of thinking about it, so that was good.” Several participants reported that the training prompted new perspectives on their own research. One scholar explained: “It helped me be more aware and insightful to my work and it gave me a whole new skill set that I was lacking.” Furthermore, the curriculum presented an opportunity to understand research study design across the continuum. One participant noted newfound expertise in their own research skills and the evaluation of a variety of study designs. They said, “One of the major things I gained was just general methodological critiquing skills…the general ability to look at a study methodology and propose alternatives.” Several respondents mentioned that course-work provided the knowledge and confidence to discuss research with a wide variety of senior colleagues. This included learning from the expertise of faculty across the three institutions. For example, one scholar commented, “we were able to get input from a diverse group of individuals…because it’s through the ACTSI…we had a person from Morehouse, Emory as well as Georgia Tech involved in our training…we were able to benefit from the strengths of other institutions.”

Important research-related skills learned through the program included grant writing, grant management, development of a research timeline, and creation of budgets. Trainees lauded the MSCR course on scientific and grant writing for providing these skills. Several mentioned that this course was the foundation for successful grant submissions in their second year of K12/KL2 training. Several K12/KL2 scholars mentioned that the instructor’s feedback was essential to the development of their research study design in their grant application. Others commented that the course provided helpful information about funding sources. One trainee noted that the process of submitting a grant was an unknown before their time as a scholar. They commented, “[the course] gave me an insight into what the process was all about, and the practical things that need to be done to accomplish the funding and setting up research projects.”

Many scholars commented on developing skills related to team science and interdisciplinary work. In particular, scholars noted the value of the exposure to scientists and resources across the university that occurs through courses, colloquiums, and special events. For example, one scholar explained, the program “introduces you to a wider aspect of the university” and explained that knowing “who is doing what and where…helps to build teams in terms of collaborative efforts with other people in other departments.” Another scholar stated, “One strength [of the program] was that they made us actually meet with all the key people involved in clinical research—with the IRB official people, the people from the clinical trial office, people from the legal office. So when I had specific question, I actually [knew] who to call.” Several scholars noted the importance of networking with scientists across various methodological and content areas. For example, a scholar stated, “It gave me access to biostatisticians and epidemiologists.”

Experiences throughout the mentoring process

Respondents were generally positive when asked about interactions with their mentor and the role the mentor played in their training. There was a range in frequency of meetings and communication, from multiple times per week to monthly meetings. Most mentors played a significant role in idea generation for research studies, study planning and design, and review of grant drafts and manuscripts. Mentors also provided general professional advice, encouragement, and feedback. For example, one scholar noted:

I would say that [mentoring] has been the key part in propelling my career forward because [my] mentors are very experienced in terms of…successful grant applications and conducting research themselves. And so as I went through the whole process of designing my study, writing and applying for grants during the course of the program, I got a lot of feedback from them and also they were key in terms of making recommendations as far as collaborative relationships have been concerned and I think that these things have helped me in terms of advancing my career.

One scholar summarized a positive mentoring experience as follows:

I have a great mentor…The success of my individual K12 mentoring experience is directly tied into her ability to mentor. We meet once a month and go through whatever I’m working on and she redlines it…I also sit in with her at weekly research team meetings and [executive] committee meetings for one of her grants. That’s helped me learn how to manage, handle people.

Another scholar noted their mentor’s influence as the scholar put together an NIH K23 award application. They explained, “I think the biggest impact [my mentor] had on me was the letter she wrote for my K23…I still have that, looking at it every now and then…I think that letter made a big difference for the K23.”

Several trainees described assembling a mentoring team rather than relying only on a single mentor. Although their lead mentor was their formal mentor and most likely most influential in their work, other mentors were critical to long-term career success. The scholars described diverse teams of mentors that provided guidance on research study design, publication, professional and ethical conduct, and funding opportunities. One scholar explained, “I had a mentoring team, a variety of individuals that I could go talk to about different aspects of clinical research, and I found that to be a very useful and helpful way of doing it…It’s nice to get a bunch of different opinions and ways of tackling programs.” Another scholar stated, “I felt like there was a huge group of mentors. I didn’t have just one.” And another explained, “I had a mosaic of mentoring…it gave me other ways to get mentored, so it broadened my mentorship.” Many scholars also mentioned the importance of the program directors as secondary or co-mentors. They noted the directors’ availability to answer questions about the program, the mentoring relationship, the course-work and assignments, and professional development. This was particularly important when a scholar felt hesitant or unsure about how to approach their K12/KL2 mentor about a problem or challenge. The K12/KL2 program directors offered advice and guidance on how to communicate effectively, resolve differences and move forward with career plans. A few scholars requested the inclusion of more formalized workshops designed to assist mentees with maximizing their relationship with their mentors. Overall, scholars reported that the mentoring component of their experiences was critical to their success in the program.

Impact of the K12/KL2 program on scholars

The majority of scholars felt strongly that the K12/KL2 program advanced their career in a multitude of ways: through a funding mechanism, protection of time, and the quality of training provided. Scholars noted the critical timing of receiving the K award and the role it played in the formation of their identity as a physician scientist. For example, one scholar stated, “When this [the KL2] came through, it was literally like an amazing thing that happened for me and in a very critical, formative time…It helped me solidify my vision and what my research career needs to look like.”

Several scholars indicated that the program assisted them in their efforts to obtain federal grants including NIH K23 award, specifically that it helped to acquire the award more quickly than if they lacked support. One scholar stated:

I think [the K12 has] been kind of like a rocket. It’s just zoomed me into clinical research. Because of the MSCR and also the amount of time [the K12/KL2 program] protected for me, I now have an American Heart Association Grant and [NIH] K23 grant. So basically I’m independently funded and I couldn’t have done that without K12 for sure.

Most importantly, the protected time for research was extremely valuable in providing sufficient time to conduct studies and gain practical experience in applying knowledge gained from the classroom. The scholar below describes the necessity of the K award protected time so that clinicians are able to justify research to administrators.

It actually freed up time that I would not have had otherwise. Having the protected time is very important…With the demands of clinical work…it would be really easy for the people in charge to just dump a lot of clinical work on us that would take away from our career development from a research perspective.

Another scholar credited the KL2 award with the ability to publish: “I’ve had 14 publications…and I will say that alone is directly attributed to the KL2 because it gave me protected time.” Some scholars acknowledged having a naïve perception of the skills needed to conduct CTR before entering the program.

I couldn’t have done any of this [research] before completing the program. I think I was delusional, frankly, because I thought I could do this without training. People had told me, ‘You don’t need any formal training,’ but I think that is absolutely wrong. Anyone who wants to do clinical research well really needs training.

One scholar summed up the impact of the program as follows: “There’s so many things…I had time to do publications. I had time to submit grants. I gained a skill set. I’m a better reviewer of journals. I was applying [everything I learned] every day.” Another scholar stated, “I think my entire research career moving forward, I owe it to the KL2 program.”

Scholars noted that the K12/KL2 award had impact beyond the careers of individual scholars. The program diversified the interests of faculty and other trainees at the institution. In particular, the addition of clinical skills broadened the school’s focus. When asked about the strengths of the program, one scholar described the importance of building more CTR teams:

One success goes back to the idea of an integrated research infrastructure…the School in Medicine is so focused on basic science research and this has been a very, very welcome addition to the overall spectrum of work that’s being done here. It’s really encouraged the growth of clinical research.

At the end of the interview, each scholar was asked two close-ended questions about their satisfaction with the program and the program’s future: ‘How satisfied are you with your experience in the K12/KL2 program?’ (response options were ‘very unsatisfied’, ‘unsatisfied’, ‘neutral’, ‘satisfied’, ‘very satisfied’) and ‘Should the program be continued?’ (response options were ‘yes’ or ‘no’). All of the respondents (100%, n=20) answered they were ‘very satisfied’ with the program and 100% (n=20) reported that the program should continue.

Recommendations for program improvement

Although the overall feedback about the program was positive, K12/KL2 scholars offered suggestions for improvement. In particular, some scholars discussed the desire for more diverse classes in research methods or additional training in professional skills. Several students requested opportunities to learn about other software programs including ‘R’, depending on their areas of study and available resources. One person explained, “I’m a big fan of R. I like it because it’s free, so when people leave here and don’t have access to SAS anymore, they can still do work… I mean SAS is great, but I can’t afford that.” One participant suggested that a larger selection of electives could provide trainees with the opportunity to tailor coursework to their specialty area. A few participants requested more opportunities to gain practical professional skills, such as giving feedback, structuring conversations with one’s supervisor, or negotiating employment requirements. For example, one scholar noted that more coaching in some basic career areas would be helpful: “I think it would be very helpful…to discuss…how [to] negotiate salary, just very basic sort of basic things.”

Currently, KL2 scholars provide feedback about the program through course evaluations that cover course content, quality of instruction, and relevance of course-work to career trajectory. They also provide feedback to the program through semiannual progress reports and an exit interview with the KL2 program directors (HMB, TRZ) and their lead mentor. Selected K12/KL2 trainees also undergo a confidential exit interview (one-on-one) with the KL2 program evaluator (DLC). In addition, as part of the MSCR program, trainees complete a pretest/post-test assessment that captures their overall experience with the program and mentoring. During the interviews, scholars suggested new mechanisms to debrief with the MSCR program administration during the semester. For example, one scholar explained, “I think the only change would be if somebody could meet with the students in the middle of the course and ask them how it is going and see how satisfied or dissatisfied they are and what they would like to change, that would help the current students.” More timely feedback would allow scholars to share concerns early on in the training process and seek productive solutions. It should be noted that several scholars described taking the initiative to provide such feedback. The supportive response on behalf of the program directors built relationships and contributed to the sense of having multiple mentors.

DISCUSSION

Overall, the junior faculty trainees at Emory University in the K12/KL2 program were extremely positive about their experiences during the didactic and mentored CTR training program. Most felt that the program provided opportunities to gain knowledge and skills about CTR, and therefore jumpstarted the development of their research program and careers. Among the major strengths identified were the didactic curriculum provided via required MSCR courses, protected time for research, mentorship, and the creation of an interdisciplinary network of new research colleagues. Scholars described increased confidence in their skills in study design and CTR implementation, grant and scholarly writing, statistical analysis and the constructive critique of other research studies. The high percentage of scholars who have attained extramural funding since completing training confirms the success of the program. The greatest number of scholars received NIH K23 Mentored Patient-Oriented Research Career Development Awards. These awards are well suited for the KL2 scholars because they support career development of junior faculty who are committed to patient-oriented research with the potential to become successful clinical investigators—the focus of our training. The scholars received fewer grants like the K08 because this funding mechanism is focused on laboratory research and might not be as applicable to our trainees if they are not conducting a laboratory-based project. A similar explanation holds true for the K01 (Mentored Research Scientist Career Development Award, biomedical, behavioral, or clinical sciences) and the K25 (Mentored Quantitative Research Career Development Award) which offer support on more focused areas of research that are not always applicable to all of the scholars in the program.

The KL2 program helps achieve the goals of the NIH NCATS CTSA program to increase the cadre of well-trained multidisciplinary and interdisciplinary CTR investigators, many of whom will lead research teams. Programs such as these provide protected time for comprehensive, didactic and mentored research training which K12/KL2 scholars who were interviewed felt was essential to their success. The lack of sufficient mentoring and protected time is cited as a barrier to efficient CTR.⁴ Our Emory K12 and ACTSI KL2 scholars highly valued mentoring within the program. Mentors played critical roles in providing feedback on grants and research manuscripts, access to resources and other scientists, and general career guidance. Mentoring is critical in academic medicine and a key component for the development of young physician scientists^31,32 and enhances research productivity in terms of publications and grant success.³³ Recently, the ACTSI KL2 program has implemented individual development plans (IDPs) as a required component of the program. Scholars complete their IDP within the first 2 months of the program, and the IDPs are approved by their mentors and reviewed by a RETCD Executive Committee member. This process assists with deliberate career decisions about research training, applying for funding, and publication. Training programs like the ACTSI KL2 should continue to require IDPs and provide structured mentoring, mentor training, and mentee training. Moreover, programs should recognize the value of multiple mentors and facilitate these relationships when appropriate. It is also evident that the role of the mentors and program directors is critical to program success. As senior faculty, they offer scientific expertise as well as career guidance.

The research interests of KL2 scholars are diverse. However, the scholars in this study described coursework (usually related to the Emory/ACTSI MSCR program) that provided a solid foundation for their independent investigations. Trainees recommended some additional training in other software and data management programs such as ‘R’, database design and management, more advanced statistics, and grants management. KL2 programs could offer this additional training experience through electives or incorporation into existing courses. This would enable individual scholars to pursue focused and cutting-edge research methodologies. One way in which leadership can be sure to stay abreast of scholars’ needs is to implement mid-semester reviews. This would provide a formal structure for scholars to report challenges with the program. Currently, the ACTSI KL2 program mandates 6-month reviews with the scholar, mentor, and the KL2 program directors. These meetings have been effective forums to keep scholars on track; however, additional efforts could be made to allow reciprocal feedback.

An important goal of the ACTSI-mentored CTR KL2 program has been to increase the number of clinical investigators who have the knowledge and skills to be successfully funded and productive investigators. A goal of the ACTSI KL2 program is to assist efforts to overcome translational blocks in moving biomedical discoveries from the bench to the bedside and community.⁷ The KL2 program minimizes some of the barriers to translational research by increasing the number of qualified clinical researchers, improving institutional infrastructure, and enhancing collaboration between researchers. Moreover, the KL2 funding mechanism serves as a catalyst for future independent investigators, and the translation of bench science into population-level health.

It is critical to evaluate the clinical and translational science training programs within the CTSAs to learn the value of the program and the impact on future translational science and workforce development. The data from this study have been important to share back with NIH but more importantly have been used by ACTSI to make changes to improve the program over the years. Evaluation guidelines, logic models, and metrics for CTSAs have been developed from numerous CTSA sites across the country. To the best of our knowledge, this is the first study to report on a qualitative program evaluation from the perspective of the participants. Future evaluations can incorporate these qualitative evaluation tools with quantitative metrics for evaluations at the end of training programs as well as track the long-term career trajectory of trainees to learn about distal impact on academic career success.^6,31,34 Furthermore, common evaluation methodologies could be recommended across CTSAs to be able to pool data about the impact of all of the training programs and provide for more standardization of these training evaluations.³⁵

This study has several limitations. Unlike NIH funding, which was assessed using NIH Reporter, the data on non-federal grants not included in Reporter are self-reported. Not all K12/KL2 scholars were included in the qualitative interviews, and thus there is the possibility of selection bias among the 20 former trainees who agreed to be interviewed. Our data are reflective of graduates of the ACTSI K12/KL2 program at Emory (between 2002 and 2016) and may not be generalizable to other scholars of translational science training programs. Further evaluation, with a mixed-method approach, and across CTSAs, will provide a more comprehensive measurement of long-term impact of KL2 programs.

CONCLUSION

These qualitative interviews and metrics on grants and publications indicate that the KL2 program is meeting its goals. The program is viewed as positive in the areas of clinical and translational science coursework, research mentoring, interdisciplinary networking, and grant writing. K12/KL2 graduates report strong program impact on obtaining early career awards, and increasing the number of publications. Furthermore, many matriculated scholars pursue CTR through extramural grant funding and clinical research careers. This study contributes to an important body of literature about the implementation and assessment of clinical research training programs.