NIH Grant Awards as a Metric of Clinical and Translational Research Training Effectiveness

Abstract

The number of clinical research training programs has increased over the last 5–10 years, but few studies have quantitatively evaluated the effectiveness of these programs. The goal of this study was to evaluate the clinical and translational research training program at the University of Cincinnati by comparing the number of National Institutes of Health grants awarded to pediatric fellows who graduated from the MS degree program between 1995 and 2013 vs. fellows who did not pursue an MS degree. Among 394 pediatric fellows, 16 of 81 (20%) MS alumni were awarded at least 1 NIH grant, as compared with 28 of 313 (9%) fellows who did not obtain an MS degree (p < 0.02). In multivariable analysis, MS alumni were more than 3 times as likely to have received at least 1 grant than were non-MS fellows (OR=3.5, 95% CI [1.7–7.2]; C-statistic=0.71) and MS alumni were more likely to obtain at least 1 K-series (OR=4.1, 95% CI [1.6–10.2]; C-statistic=0.74), M-series (OR=11.8, 95% CI [3.4–41.4]; C-statistic=0.81), or R-series (OR=10.1, 95% CI [2.4–42.8]; C-statistic=0.74) grant than were non-MS fellows. These findings suggest that graduate training in clinical and translational research prepares graduates for the highly competitive field of clinical and translational research.

Introduction

Reluctance among young physicians to undertake a career in clinical and translational research, and high attrition rates among those that do, have long been concerns within the field of academic medicine [1 2]. A major contributor to the low appeal of clinical and translational research as a career is the perceived difficulty in obtaining grant funding. Unless the situation is remedied soon, a whole generation of potential researchers may be lost. As recently as 2013, Gottesman [3] wrote, “The shortfall of new physician-researchers is a national, if not global, concern, and its remedy requires a coherent and cooperative approach among biomedical research and teaching institutions.” In an effort to improve the appeal of a clinical and translational research career for young investigators, several initiatives have been developed since the mid-1990s, including relieving educational debt [4–7], creating a clear clinical and translational research career path [3 7–11], improving mentorship [12–18], investing in clinical and translational research institutes across the country [19–21], and emphasizing training and education programs across the spectrum of clinical and translational research [4 22–24]. Early studies on the effects of those investments show promising signs of improvement [25], although there is still a known gap in the field between translating bench research to clinical application [26].

As the National Institutes of Health (NIH) and institutional investments in clinical research have increased, graduate training programs in clinical and translational research have been formalized across the United States. The training component of the Clinical and Translational Science Award (CTSA) program involves immersing investigators from diverse disciplines such as medicine, pediatrics, surgery, dentistry, nursing and pharmacology in a series of relevant clinical and translational science courses and training experiences [23]. The University of Cincinnati (UC) College of Medicine Department of Environmental Health has been training physicians in epidemiology, biostatistics, research design, and ethics since the mid-1980s, primarily through its MS and PhD programs in Epidemiology. The training program for physician-scientists was formalized in the mid-2000s, culminating with a Master of Science in Clinical and Translational Research (MSCTR) degree, formally approved in 2009. The educational objective of the MSCTR is to train clinical professionals (physicians, nurses and other terminal degree clinicians) to become independent investigators. One of its key goals is to equip graduates to prepare successful career development and independent investigator award applications.

Few studies have quantitatively evaluated the success of training provided through a clinical and translational research degree program. In one study, using an alumni survey, Goldhamer and colleagues (2009) found that success in obtaining NIH grant funding is associated with starting a training program at a younger age, being a generalist, and successfully publishing projects emanating from coursework. Recently, we reported findings from a study that compared publication track records of pediatric fellows who graduated from our MS program vs. comparable pediatric fellows who did not get the MS degree, showing that MS graduates publish more first-authored articles, and more articles overall, than their counterparts [27 28]. Additionally, men in the non-MS group out-published their women colleagues, but the gender gap was absent in the MS group.

The purpose of this study was to extend our empirical program evaluation of the effectiveness of the UC MSCTR program by using NIH grant awards of graduates as a metric. We compared grant awards data of pediatric fellows who completed the MSCTR program vs. pediatric fellows who did not pursue a Master’s degree during their fellowship.

Methods

Study Sample

The study sample included physicians who completed a pediatric fellowship program at Cincinnati Children’s Hospital Medical Center (CCHMC) between 1995 and 2013. Pediatric fellows were excluded if they did not complete their fellowship or if they enrolled in a Master of Science training program other than the UC MSCTR. For the purposes of our prior analysis of publication rates, and also for the current analysis from the same cohort, we combined our 2 physician training programs (the historical MS in Epidemiology program from 1995–2009 and the MS in Clinical and Translational Research [MSCTR] program from 2009–2013) into one, herein referred to as the MSCTR [27]. Demographic information for each fellow was obtained from CCHMC records, including the fellow’s sex and age and the beginning and end date of each subject’s fellowship. This study was reviewed by the UC Institutional Review Board and determined to be exempt.

Grant Award Data Collection

We searched enGrant Scientific (http://www.engrant.com/) and the NIH Research Portfolio Online Reporting Tools (RePORT) database to retrieve grant award information for each fellow. Grant awardees were cross-referenced with CCHMC division/fellowship affiliation and known current position to ensure that the investigator was indeed the fellow in this study and not someone with a similar name. Any grants awarded before beginning the MS and/or fellowship program were excluded from the analysis (n=2), as were any non-NIH grants (n=2). We subdivided grant awards into grant types: career development [K-series], research grant [R-series], and General Clinical Research Center [M-series]. We also calculated the total grant money awarded, including facilities and administrative costs.

Data Analysis

Descriptive statistics were used to characterize the sample. We used t-tests for continuous variables and chi-squares for categorical variables to compare differences between MSCTR fellows and non-MSCTR fellows. We used logistic regression to evaluate the effect of graduating from the MSCTR on fellows’ likelihood of receiving at least 1 grant award. We also evaluated the effect of the MSCTR on fellows’ likelihood of obtaining at least 1 grant award by different type (K-, R-, and M-series), and included age and sex as covariates, since past research has found these can be significant [27 29–34]. We analyzed the amount of time to obtaining one’s first grant by using a Kaplan-Meier plot and a Cox proportional hazards regression model with age and sex as covariates. We used SAS for Windows, version 9.4 (SAS Institute, Cary, NC) to carry out all statistical analyses, and a 5% significance level was assumed.

Results

Subjects

Of the 537 fellows at CCHMC between 1995 and 2013, 394 met the inclusion criteria: 81 (21%) were MSCTR fellows and 313 (79%) non-MSCTR fellows. When comparing the age and sex of the included and excluded fellows, sex ratios were similar, but, as expected by virtue of having completed a fellowship, fellows included in our analyses were significantly older (mean=38.7 years) than those who were excluded (mean=32.9 years; p<0.0001). The analytic sample was almost evenly divided between women (n=195) and men (n=199). The MSCTR fellows group included more women (n=50 [62%]) than men (n=31 [38%]), whereas the non-MSCTR fellows group was comprised of more men (n=168 [54%]) than women (n=145 [46%]; p<0.01; Table 1). Non-MSCTR fellows were on average 2 years older than MSCTR fellows (p<0.005; Table 1).

Table 1.

Descriptive Characteristics of Former Fellows with an MSCTR Degree vs. Former Fellows without an MSCTR Degree

Characteristics	MSCTR Degree (n=81)	No MSCTR Degree (n=313)	p-value
Female, n (%)	50 (63)	145 (46)	0.02*
Age, mean, SD (range)	37 (30–50)	39 (25–60)	0.005
Total People with a Grant, n (%)†	16 (20)	28 (9)	0.01*
Total People with a K Grant, n (%)	10 (12)	15 (5)	0.02*
Total People with an M Grant, n (%)	8 (10)	5 (2)	0.001*
Total People with an R Grant, n (%)	5 (6)	5 (2)	0.03*
Years to First Grant, mean (range)	5.1 (2–12)	6.0 (1–13)	0.31*
Total Grant Money, GM (range)	$295,933 ($4,549–$5,540,386)	$251,841 ($1,056–$2,844,398)	0.78

MSCTR: Master of Science in Clinical and Translational Research; GM: geometric mean

^*Chi-square

^†K+M+R>Total because some fellows had >1 grant

Funding Success Rates

In univariate analyses, 16 (20%) graduates of the MSCTR program had been awarded at least 1 NIH grant, as compared with 28 (9%) fellows who did not obtain an MS degree (p<0.02; Table 1). MSCTR fellows were also significantly more likely to have been awarded at least 1 M-series grant (p<0.02), but there was no significant difference in the proportion of MSCTR and non-MSCTR fellows with at least 1 K-series or R-series grant (Table 1).

In multivariable analysis controlling for age and sex, MSCTR fellows were more than 3 times as likely to have received at least 1 NIH grant than were non-MSCTR fellows (OR=3.5, 95% CI [1.7–7.2]; C-statistic=0.71; Table 2). In addition, MSCTR fellows were more likely to obtain at least 1 K-series (OR=4.1, 95% CI [1.6–10.2]; C-statistic=0.74; Table 2), M-series (OR=11.8, 95% CI [3.4–41.4]; C-statistic=0.81; Table 2), or R-series (OR=10.1, 95% CI [2.4–42.8]; C-statistic=0.74; Table 2) grant than were non-MSCTR fellows. Age was a significant covariate in all models, and sex was insignificant in all models except for R-series grants, with men being more likely than women to have been awarded at least 1 R-series grant (OR=13.0, 95% CI [1.5–110.9]; Table 2).

Table 2.

Multivariable Regression Odds Ratios of MSCTR Graduates Receiving NIH Grants

Grant Type	Variable	Odds Ratio	95% Confidence Interval	p-value	C Statistic
Total Grants	MSCTR Status	3.5	1.7–7.2	<0.001	0.71
	Age	1.1	1.1–1.2	<0.001
	Sex	1.3	0.7–2.5	0.49
K Grants	MSCTR Status	4.1	1.6–10.2	0.002	0.74
	Age	1.1	1.1–1.2	<0.001
	Sex	1.1	0.5–2.7	0.77
M Grants	MSCTR Status	11.8	3.4–41.4	<0.001	0.81
	Age	1.1	1.0–1.3	0.02
	Sex	3.1	0.9–10.8	0.08
R Grants	MSCTR Status	10.1	2.4–42.8	0.002	0.87
	Age	1.2	1.1–1.4	0.003
	Sex	13.0	1.5–110.9	0.02

MSCTR: Master of Science in Clinical and Translational Research

Time to First Grant

Of the fellows who received grants, MSCTR fellows tended to be more likely to get a grant almost a year earlier (mean=5.1 years, SD=2.6) than were non-MSCTR fellows (mean=6.0 years, SD=2.4), but this association was not statistically significant (Table 1; Figure 1). The Cox proportional hazards regression model yielded a hazard ratio of 0.68, (non-MSCTR reference group), indicating that MSCTR graduates tended to get funded more quickly, although the hazard ratio was not statistically significant (p=0.31). When comparing the geometric means of total grant dollars awarded, we did not find a significant difference between the 2 groups (Table 1).

Figure 1.

Kaplan-Meier Curve of Years to First Grant

MSCTR: Master of Science in Clinical and Translational Research

Discussion

As clinical and translational research training programs have proliferated in the last 5–10 years and curriculum guidelines and training competencies have been formalized, the emphasis has started to shift from planning and development to evaluation and improvement. While traditional forms of program evaluation (course evaluations, exit surveys, and student focus groups) indicate high levels of satisfaction from our UC MSCTR graduates, the goal of this study was to evaluate the effectiveness of our training program in terms of one of its primary objectives: to prepare clinical professionals to conduct independent clinical and translational research as manifested by obtaining external funding for their research. We began tracking alumni grant awards years ago, but this analysis was our first attempt to compare our alumni rates with those of an appropriate comparison group. Pediatric fellows from CCHMC comprise approximately 70% of the MS student body. The remaining 30% are UC fellows or junior faculty from UC, CCHMC, or an external organization, and we did not include them in this analysis because an appropriate comparison group was not available.

Our analyses indicate that pediatric fellows who completed the MSCTR are more than 3 times as likely to have been awarded at least 1 NIH grant than fellows who did not complete the MSCTR. We also found that graduates of the MSCTR program were more likely to have been awarded at least 1 K-, M-, or R-series grant award than fellows who did not pursue the MSCTR. Although not statistically significant, MSCTR alumni were also awarded their first grant, on average, almost a full year sooner than were non-alumni, even though MSCTR fellows were significantly younger than non-MSCTR fellows. Further studies with larger sample sizes would be needed to confirm this association.

Reasons for greater success in obtaining grants among MSCTR alumni are likely related to the curriculum: fellows who pursue the MSCTR are immersed in didactic coursework and mentored research projects throughout the 2 or more years during which they earn the degree, and they complete a capstone course that requires them to develop a hypothesis and specific aims, create a plan of investigation, and prepare an NIH R21-like grant proposal. The MSCTR also connects trainees with a network of research resources, including faculty mentors with shared research interests, large existing databases, biostatistician and study design colleagues, and student assistants for data entry and other research tasks. MSCTR students are strongly encouraged to become members of the Center for Clinical and Translational Science and Training (CCTST; http://cctst.uc.edu), where they can gain access to a myriad of research services.

In our previous study examining publication rates of MSCTR fellows and non-MSCTR fellows, we found that men in the non-alumni group publish more papers than do their female colleagues, but that alumni men and women publish at similar rates [27]. Interestingly, sex was not a significant covariate in any of this study’s models, except for the R-award analysis. Among both alumni and non-alumni, men were more likely to have received at least 1 R-grant award than women. The R-award is generally viewed as the funding mechanism that marks one’s passage from a mentored researcher to an independent researcher [35]. Given the literature showing that careers of women clinical and translational researchers lag behind those of men [29 31–34 36], it is perhaps not surprising that we found that women are less likely to attain R-level funding. This disparity could be explained by the dearth of high-quality female mentors and/or by part-time employment being more common among women, particularly during child-rearing years [37–39].

This study has several limitations. We focused on pediatric fellows from a single institution, so the results may not be generalizable. Even though we analyzed nearly 20 years of data, the relatively small number of fellows with grants, and the small number of grants, was a limiting factor in our analyses. Unfortunately, because of the small number of fellows having grants, we did not have statistical power to analyze additional subgroups, such as race or ethnicity. The fellows included in this study hailed from approximately 14 different pediatric general and subspecialty fellowship programs. Fellowships have different emphases and expectations in terms of research and education, so some fellows are more strongly encouraged than others (or required) to pursue graduate training in research. Nevertheless, all of the fellowships included in this analysis are ACGME-accredited, meaning they all emphasize research education to some degree. This study was also unable to control for the multitude of other factors that contribute to physicians’ decisions to pursue research funding after fellowship, such as mentorship, educational debt, values, and other personal factors [17 40].

Conclusions

Except in very rare cases, a clinical and translational researcher cannot be successful without securing funding for his or her research. When compared with non-MSCTR alumni, our MSCTR alumni have been more likely to have secured NIH grant funding. This finding indicates that the MSCTR program has contributed to preparing graduates for the highly competitive field of clinical and translational research. Directions for future research include evaluating success in terms of team science awards and academic leadership positions. A larger-scale research project across several MS training programs would also provide more data and a more comprehensive picture of the effects of clinical and translational research training on physician-scientist research productivity.