Publication Track Records as a Metric of Clinical Research Training Effectiveness

Abstract

Clinical research training programs exist across the country, but no quantitative studies have been performed to evaluate the effectiveness of these programs. The goal of this study was to evaluate the success of the clinical research training program at the University of Cincinnati by comparing the publication histories of pediatric fellows who graduated from the clinical and translational research Master of Science (MS) degree programs between 1995 and 2011 with fellows who did not pursue an MS degree. Among 296 pediatric fellows, 44 of 54 graduates (81%) published at least 1 first‐authored paper, as compared with 149 of 242 (62%) fellows who did not obtain an MS degree (P < 0.01). In multivariable analysis, 3–4 years after program completion, MS graduates published more papers overall (R ² = 0.10) and more first‐authored papers than did non‐MS graduates (R ² = 0.04). These findings suggest that graduate training in clinical and translational research is related to an increase in research productivity as assessed by publication rates.

Introduction

For several decades, the academic medical community has been concerned about the reluctance of young physicians to prepare for and undertake careers in clinical research.1, 2 As the demand at academic health centers for physician‐faculty members to provide clinical care to patients increases, clinicians have less time to focus on a research career.3 Equally troubling is the high attrition rate of accomplished clinical investigators within academic medical centers.3, 4, 5 Numerous factors have contributed to the dearth of clinician‐investigators: increased competition for National Institutes of Health (NIH) grant funding (particularly from non‐MD applicants),3 deficient research training,6 ineffective mentorship,7 inefficient institutional infrastructure,7 and outmoded promotion and tenure guidelines that do not reward team science.8 In an era characterized by potentially enormous biomedical discoveries, all of these factors limit progress.

In response, the federal government has taken strides over the past several decades to try to bolster the numbers of well‐trained clinical researchers. In 1995, the Director of the NIH organized the “NIH Director's Panel on Clinical Research” to assess the current state of clinical research and offer suggestions for improving it.10 The panel's recommendations instigated many new changes to the clinical research environment. In 1999, new patient‐oriented research career development awards were introduced (K23 and K24 grants), supporting both young investigators and their mentors.9, 10 In 2000, Congress passed a bill to offer educational loan relief to clinical researchers who fulfill eligibility requirements,10 aiming to reduce the financial strain that prevents some physicians from pursuing a research career. The NIH also began investing more in clinical research training programs, instituting the Clinical Research Curriculum Award, or K30, in 1998 to improve the quality of training in clinical research by funding educational programs at many U.S. academic institutions.2, 11 The K30 program supports developing and/or improving didactic curricula in clinical research theory, methodology, application, and ethics.12

In addition, in 2006, the NIH initiated the Institutional Clinical and Translational Science Award (CTSA) program. A key component of the CTSA program involves fostering graduate and postgraduate programs in clinical and translational science to provide a knowledge base for clinical and translational researchers. The training component essentially carries forth the original goal of the K30 program, which is to train investigators from diverse disciplines such as medicine, pediatrics, surgery, dentistry, nursing and pharmacology in a series of relevant clinical and translational science courses and training experiences.13 In addition, as new training programs began to emerge at more institutions around the country, the NIH National Center for Research Resources, which has since been supplanted by the National Center for Advancing Translational Sciences, developed core clinical research educational competencies in 2008.14 The core competencies provide guidelines for training in a variety of thematic areas, including research question formulation, literature critique, study design, research implementation, identification of sources of error, statistical approaches, informatics, scientific communication, diversity, translational teamwork, leadership, cross‐disciplinary training, and community engagement in research.14

Few studies have quantitatively evaluated the success of physician‐scientist training programs. The Program in Clinical Effectiveness at the Harvard School of Public Health surveyed its alumni and showed that an individual's likelihood of receiving NIH grant funding is correlated with starting the training program at a younger age, being a generalist, and successfully publishing projects from coursework.15 At least two more qualitative tools and frameworks for assessing clinical researchers’ success after completing a training program have been developed over the last 5 years. The first, called the Clinical Research Appraisal Inventory (CRAI), aims to measure trainees’ perceived self‐efficacy in a variety of conceptual areas such as conceiving and designing a study, funding and managing a study, collaborating with peers, conducting research responsibly, collecting and interpreting data, and reporting study results.11, 16 More recently, the CTSA program's Evaluation Key Function Committee published a comprehensive model for career success that can be used to “theoretically explore determinants of career success among physician‐scientists”.17 That model includes both extrinsic (financial success, promotion, grants, and publications) and intrinsic (satisfaction with job, career, and life) factors in its conceptualization of overall career success.

Although assessment tools such as the CRAI and the comprehensive career‐success model are important and helpful approaches for program directors to consider, neither offers pragmatic methods for empirically evaluating the productivity of investigators who are trained through postdoctoral research training programs. Although self‐efficacy surveys can provide a gestalt of applied knowledge,13, 18, 19 assessing trainees’ perceived self‐efficacy by using the CRAI is not necessarily directly correlated with their actual or potential success in a research career.20, 21, 22 Clinical researchers might feel very confident about their skills posttraining without having actually mastered those skills. Hence, the CRAI is typically viewed as a “short‐term indicator of program impact,” rather than a long‐term indicator.16 The personal and organizational factors described in the career‐success model are intended primarily for theoretical use, not as “empiric validations.”17 As stated above, the comprehensive career‐success model does include a few quantifiable metrics of “extrinsic” success indicators—including financial success, promotion, leadership positions, grants, and publications—but the first three of those are difficult to measure, given that data on trainees’ current salary levels and career trajectories are not readily available. Publications, on the other hand, can be found through open access databases available online.

The University of Cincinnati (UC) has offered master's‐level training in clinical research methods since the early 1990s, originally through the Master of Science (MS) in Epidemiology and Biostatistics and then through the MS in Clinical and Translational Research. UC received a K30 award in 2005 and a CTSA grant in 2009. The curriculum has been updated regularly, both as a result of K30/CTSA funding and to meet current need and student interest, but the fundamental focus on epidemiology, biostatistics, ethics, and study design and management have remained the same. For the purposes of this study, we have combined these MS physician training programs, herein referred to as the UC Clinical Research Training Program (CRTP).

The purpose of this study was to evaluate the effectiveness of the UC CRTP by using publication data as a metric. We compared publication productivity of pediatrics fellows who graduated from the CRTP program with pediatrics fellows who did not matriculate in the program.

Methods

Participants

Participants in the study were physicians who completed pediatrics fellowship programs at Cincinnati Children's Hospital Medical Center (CCHMC) between 1995 and 2011 (n = 443). A total of 147 fellows were excluded because they either did not complete their fellowship, they enrolled in the CRTP without graduating, or they enrolled in a training program other than the CRTP. The remaining 296 participants comprised the study sample: 54 who completed the CRTP and 242 who did not enroll in the CRTP or any other similar program. Demographic information for each fellow was obtained from CCHMC records, including sex, age, and the beginning and end date of each subject's fellowship. This study was reviewed by the UC Institutional Review Board and determined to be exempt.

Publication data collection

We searched PubMed and Scopus to retrieve publications for each fellow.23 Publication authors were cross‐referenced with CCHMC division/fellowship affiliation and known current position to ensure that the author was indeed the participant in the study and not someone with a similar name. The total number of publications for each fellow was subdivided into first‐authored publications, publications before fellowship, publications during fellowship, and publications after fellowship.

Data analysis

We calculated means (with range and standard error) or counts (with percents of total) to describe continuous and discrete characteristics, respectively. Wilcoxon two‐sample tests were used to compare differences between CRTP graduates and nonmatriculants. We used analysis of variance to evaluate the effect of graduating on the number of publications. Using log transformation to normalize the data, we built two different models, one to assess the effect of CRTP training on the total number of first‐authored publications and another to assess the effect of CRTP training on the total number of publications. In each model, variables such as age, sex, and time after fellowship were included as covariates in the analyses. Time after the fellowship was defined as the difference between year 2011 and the fellowship ending year. We used SAS for Windows, version 9.3 (SAS Institute, Cary, NC, USA) to carry out all statistical analyses, and a 5% significance level was assumed.

Results

We identified 296 fellows who met the inclusion criteria: 54 (18%) were CRTP alumni and 242 (82%) had completed their pediatrics fellowship program but not the CRTP. The average age of fellows who completed the CRTP was 39, compared with 40 for fellows who did not (P = NS). There was no difference in length of fellowship training between the two groups; graduates of the CRTP averaged 3.1 (range, 2–4) years to complete fellowship versus 2.9 (range 1–5) years for the comparison group (P = NS). Our study sample was almost evenly divided between women (n = 147) and men (n = 149). The MS alumni included more women (n = 33) than men (n = 21), whereas the group of fellows who did not enroll in a graduate training program included more men (n = 128) than women (n = 114) (Table 1).

Table 1.

Descriptive characteristics of CRTP graduates versus nongraduates

Characteristics	CRTP Graduates (N = 54)	Non‐CRTP Graduates (N = 242)	P value
Female, N (%)	34 (63)	114 (47)	0.05
Age, mean (range)	39 (32–50)	40 (32–60)	0.24
Number of years to finish fellowship, mean (range)	3.1 (2–4)	2.9 (1–5)	0.23
Number of years since the end of fellowship, mean (range)	5.2 (0–11)	5.8 (1–11)	0.01

CRTP = Clinical Research Training Program.

First‐authored publications

Among graduates of the CRTP, 44 (81%) published at least 1 first‐authored paper, as compared with 149 (62%) fellows who did not obtain an MS degree (P < 0.01). The effect that the CRTP had on the number of first‐authored publications was modified by the time elapsed since fellowship. Within the first 3–4 years after program completion, the CRTP had no significant effect on the number of publications. However, approximately 3–4 years after program completion, CRTP graduates had published significantly more first‐authored papers (P < 0.04; R ² = 0.04; Table 2; Figure 1). Overall, men were significantly more likely to have first‐authored publications (P < 0.01; Table 2), but the sex difference in publication rate was significant only among non‐alumni (P < 0.01) and not among alumni (P = NS).

Table 2.

Multivariable predictors of the number of first‐authored publications

Characteristics	Estimate (beta)	SE	P value
Masters*	0.06	0.10	0.70
Masters†	0.42	0.20	0.04
Sex‡	0.39	0.14	0.01
Age	−0.02	0.02	0.33

*Effect of training program within the first 3–4 years after the fellowship;

†Effect of training program beyond the first 3–4 years after the fellowship, Model R ² = 0.04.

‡Females are the reference group.

Figure 1.

Log‐transformed number of first authored publications over time by masters degree status.

Total number of publications

The mean (SE) of publications by fellows who graduated from the CRTP was 9.7 (19.3), and the mean (SE) of publications by fellows who did not graduate from the CRTP was 5.8 (8.0; P = 0.02). In multivariable analysis, we found that completing the CRTP was also significantly associated with the total number of publications, first‐author or otherwise, but only 3–4 years after program completion. Similar to the results found for first‐authored publications within the first 3–4 years after fellowship completion, the CRTP had no significant effect on the number of publications. However, the effect of the CRTP became significant approximately 3–4 years after fellowship completion and beyond (P < 0.03; R ² = 0.10; Table 3; Figure 2). In addition, in the full sample, men had significantly more publications than their women colleagues (P < 0.02; Table 3)—nonalumni men published significantly more papers than women (P < 0.0001), but there was no significant sex difference in publications in the alumni group.

Table 3.

Multivariable predictors of the number of total publications

Characteristics	Estimate (beta)	SE	P value
Masters*	0.07	0.21	0.75
Masters†	0.44	0.20	0.03
Sex‡	0.45	0.18	0.02
Age	−0.08	0.04	0.06

*Effect of training program within the first 3–4 years after the fellowship;

†Effect of training program beyond 3–4 years after the fellowship, Model R ² = 0.10.

‡Females are the reference group.

Figure 2.

Log‐transformed number of publications over time by masters degree status.

Discussion

Thanks to decades of efforts, clinical research training programs are now widespread. As with any other curriculum, however, the effectiveness of clinical research training programs should be evaluated. Whereas traditional methods of training program evaluation typically include alumni survey data that attempt to measure everything from course and curriculum quality to faculty and mentor interaction to perceived usefulness of program content, our goal was to consider the success of our clinical research training program from a pure productivity standpoint. The CRTP at UC has tracked alumni publications for many years as a measure of training success. Until recently, though, we did not evaluate an appropriate comparison group comprised of clinicians who share similar backgrounds with our CRTP trainees, but who did not pursue a research‐specific master's degree. Our pediatric fellowship programs are accredited by the Accreditation Council for Graduate Medical Education (ACGME); in its “Common Program Requirements” document, the ACGME emphasizes a research education component to training and a commitment to scholarly activity.24 Because of this, at least in principle, fellows in our ACGME‐accredited programs who do not pursue formal research training share a similar educational focus and research‐oriented career path with those fellows who do pursue research training in the CRTP.

Our analyses indicate that pediatric fellows who complete the UC CRTP, on average, have greater numbers of first‐authored publications and greater numbers of publications, overall, than fellows who do not complete the CRTP. This effect becomes particularly apparent 3–4 years after program completion. The 3–4‐year lag in increased publications could be explained by the typical career trajectory of a clinical investigator. Clinical post‐doctoral fellows spend much of their fellowships seeing patients. Those who also enroll in master's programs need time to learn research methodology before (or perhaps while) beginning to apply it, and then months to years to complete all steps of the publication process. Depending on the type of project, fellows may also need to obtain approval from an institutional review board or obtain funding, or both, before embarking on their research.25 The publication process itself can also be lengthy, delaying the actual publication of research results for months or even years after a project ends.26, 27

We also found that overall, men are significantly more likely than women to publish, both during and after fellowship training. This finding could perhaps be explained by maternity leave, lack of high‐quality female mentors, and/or part‐time employment among women.28, 29, 30 When we compared publication rates of men and women among alumni versus non‐alumni, men in the nonalumni group published significantly more papers than females, both overall and first‐authored. However, there was no significant difference in publication rates by sex among MS graduates, overall or first‐authored—suggesting that completing the CRTP eliminates the gender gap in publication productivity for women.

Our study has several limitations. First, it was a retrospective cohort study that focused on fellows at a single institution, so the results cannot be generalized. The cultures of the various pediatrics fellowship programs within CCHMC vary; some require fellows to complete different levels of training programs (MS, Certificate, or none at all), whereas others require that fellows spend more time performing clinical duties, perhaps at the expense of research training. Students who enroll in our CRTP program are potentially self‐selecting, as their enrollment in a graduate degree in clinical and translational research demonstrates a strong interest in a career in academic medicine. Such differences can affect fellows’ decisions to complete a master's degree and pursue research careers—and yet, as discussed above, all of the CCHMC fellowships require scholarly activity during fellowship. In addition, our analysis did not include other possible variables that affect fellows’ publication productivity: previous research experience, research mentoring,31, 32 demographic factors that impact professional goals, and opportunities in fellows’ particular fields of interest.

Conclusions

Research success and publication productivity are inextricably linked in academic medicine;33, 34, 35, 36 the increased likelihood of our alumni publishing their research findings indicates that on this measure, the clinical and translational research training program at UC has succeeded in boosting scholarly output, as measured by publication rates. Before generalizing these findings to other physician‐scientist training programs, further evaluation is needed to assess consistency of program outcomes and effectiveness. Other measures of success should also be evaluated, such as grant awards and leadership positions, as well as the intrinsic rewards of a successful research career. Finally, further investigation is needed to determine whether completion of a graduate research training program does eliminate the gender gap in publication rates for women.

Sources of Funding

The project described was supported by the National Center for Research Resources and the National Center for Advancing Translational Sciences, NIH, through Grant 8 UL1 TR000077–05. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

Ethical Approval

This study was reviewed by the UC Institutional Review Board and determined to be exempt.

Disclaimers

None.