An Institutional Postdoctoral Research Training Program: Increasing Productivity of Postdoctoral Trainees

Abstract

Objective

Postdoctoral training is a critical stage of career development, and there has been a national effort to increase the consistency and quality of postdoctoral experiences. However, much of the effort has gone towards improving the process of training with less effort focusing on the content of what should be achieved during postdoctoral training, primarily because of a lack of empirical evidence in this area. One possible predictor of later scientific productivity is the number of peer-reviewed papers published during postdoctoral training. This manuscript reports on efforts to increase postdoctoral productivity.

Method

A single institution made postdoctoral training program changes designed to increase postdoctoral publication productivity. Postdoctoral publication productivity was compared between 114 trainees who matriculated prior to the changes and 20 trainees who matriculated after the changes.

Results

Postdoctoral trainees who matriculated after program changes had higher publication rates than postdoctoral trainees who matriculated prior to program changes [χ²(df= 15)=31.4, p=.002]. Four or more postdoctoral publications are associated with the greatest likelihood of sustained posttraining publications; postdocs matriculating after the program changes were more than twice as likely to have four or more publications (55 vs 26 %).

Conclusions

Postdoctoral program changes designed to increase postdoctoral publication rates can be successful. Defining, for each postdoc, a minimal postdoctoral publication rate may be an appropriate component of individualized development plans.

Postdoctoral training has long been considered an integral part of research training, providing the additional skills necessary for transition to an independent investigator. As recognition of the importance of this stage in career development has grown, there has been increasing scrutiny as to what defines an optimal postdoctoral training experience. A combined effort of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine produced a seminal 2000 report aimed at increasing the consistency and quality of postdoctoral experiences; recommendations included standardizing salaries, defining and improving benefits, limiting duration (to ensure appropriate career advancement), creating institutional postdoctoral training offices, and writing career development plans early in postdoctoral training [1]. While success in this area has been incomplete [2], there have been strong efforts and a fair amount of success, at both the institutional and national level, implementing these recommendations. However, one limitation of the recommendations is that they focus more on process than on content. For example, while one recommendation states “Advisers should discuss goals with the postdoc at the outset” [1], there are no recommendations on what goals should be.

We, the Developmental Psychobiology Research Group at the University of Colorado School of Medicine, began several years ago a process to address this gap, attempting to identify what milestones within a postdoctoral training experience were most predictive of successful career outcomes. Different career goals likely require different milestones; we focused our efforts on milestones predictive of later success as a sustained contributor to the scientific literature. While we considered a number of definitions of success, we defined success as a postdoctoral trainee who, after completion of postdoctoral training, showed sustained scientific contributions by averaging at least one published paper per year. Details of other definitions, including considering author order on published papers and success at obtaining independence, have been published elsewhere [3].

In previous work, in order to identify predictors of sustained scientific success, publication records from 92 former postdoctoral trainees of a long-standing National Institute of Mental Health (NIMH)-funded Postdoctoral Institutional Research training program were evaluated relative to several potential predictors, including factors specific to the postdoctoral trainee (age, gender, terminal degree, year(s) in the program), factors specific to the mentor (gender, terminal degree, publication history, impact factor [4], R01 funding), factors associated with the interaction between the mentor and mentee (gender match, terminal degree match), and performance during the postdoctoral training program. In a multiple regression analysis, most factors—including trainee terminal degree (MD, PhD), trainee gender, mentor funding history, and mentor’s impact factor—were not predictive of sustained scientific success. Only two factors independently predicted scientific success. The best predictor was the number of publications published as a postdoctoral fellow; success at achieving individual training funding that persisted beyond involvement in the institutional training program was a smaller but also significant predictor. Figure 1 summarizes the results from this previously published analysis [3]. While not reviewed here, the same two factors were the strongest predictors for other definitions of sustained academic success [3].

Fig. 1.

Relationship between the numbers of papers published while a postdoctoral trainee and likelihood of averaging at least one published paper per year after completion of the postdoctoral training (left). Relationship, for institutional postdoctoral trainees, between successfully competing for an individual NIH training award (NRSA or K-Award) and likelihood of averaging at least one published paper per year after completion of the postdoctoral training (right). Circled numbers represent the number of trainees in each group

Based on the results of these analyses, the postdoctoral program decided to focus its efforts on increasing publication rates of postdoctoral trainees, aiming for maximizing the percentage of trainees who published at least three and preferably four manuscripts during a 2-year postdoctoral experience. This manuscript evaluates the success of that effort as well as considers three alternative explanations for any changes in postdoctoral trainee publication rates.

Method

The sample consists of 134 individuals who matriculated into postdoctoral training as part of the University of Colorado’s Institutional NRSA “Developmental Psychobiology, Psychopathology and Behavior” (T32 MH015442) between September 1, 1978, and September 1, 2013. The sample includes four current postdoctoral trainees plus 130 individuals who completed an average±S.D. of 19±7 months in the postdoctoral program. The sample is 67 % female, 89 % Caucasian non-Hispanic, who entered the program at a mean age±S.D. of 34 ±6 years and includes 31 MDs, 101 PhDs, 1 MD-PhD, and 1 PsyD. The Colorado Multiple Institutional Review Board exempted this study from review.

Intervention to Increase Postdoctoral Publications

This training program is a research training program; clinical and teaching responsibilities have been limited throughout the duration of the program. In 2006, this program’s postdoctoral training faculty reviewed the initial findings and agreed to a goal of increasing publication productivity. Over the next 18 months, the faculty developed an approach which was initiated in 2008.

The earliest change was conceptual. The majority of faculty encouraged postdoctoral development in a sequential process similar to development of a specific research project: familiarization with the field, development of a research plan, skill development and data collection, analytic techniques, and then dissemination. This generally results in a limited number of publications which occur late in a postdoctoral training experience.

In contrast to the sequential nature of the educational process, most active researchers’ careers do not follow this model, instead engaging in all components: concurrently writing a paper from one study, collecting data on the next, learning new skills for a planned project, and expanding knowledge to be ready for the next project. While the sequential model may be appropriate for trainees earlier in their career development, to facilitate transition to independence, the program switched to a concurrent model, where trainees are expected to be doing all aspects at all times. The program hypothesized that this would result in a consistent publication rate throughout the postdoctoral experience. For publications early in the postdoctoral training experience, data can come from unpublished work done in graduate school or from not-yet-published data already available within the mentors’ research program. Because we believe that postdoctoral trainees and their mentors should not be competing for publication rights, we limit our postdoctoral faculty to senior faculty with well-established careers, national reputations, and a successful history of mentoring trainees, including graduate students, medical students, and residents. Thus, most faculty (a) have available unpublished data and (b) do not feel a risk to their own career by allowing the postdoctoral trainee to first-author publish the mentor’s data. A corollary of this approach is that mentors understand, at least in the context of an institutional training program, that postdocs may devote a portion of their time to writing papers related to data collected prior to joining the postdoctoral program.

In addition to the conceptual approach, program alterations included infrastructure and requirement changes. Changes were made incrementally for feasibility assessment. With the exception of the Writing Group (described below), all changes were announced 14–15 months prior to implementation to allow both mentors and postdoctoral trainees time to effectively prepare for this program change.

Beginning in 2010, the program’s application was adjusted to match these program goals. All applicants must have a mentor sponsor, and the application represents an initial collaborative effort. In addition to standard demographic and past training information, the application requires answers to four questions: (a) what is the topic and data to be used for a paper submission within 6 months of starting the program; (b) what are the skills to be learned as a postdoctoral trainee and what ongoing project can the trainee join immediately upon starting the postdoctoral program to learn/apply those skills; (c) what is the topic area in which the postdoctoral trainee will develop their own project; and (d, added in 2012) how do they match all these efforts to the NIMH strategic plan? The deadline for the program’s application is December 1 of each year, often 8–10 months prior to the matriculation date. While the goal of this application process was to facilitate trainees starting the program with an already developed plan, the co-development between postdoctoral applicant and mentor many months before matriculation is increasing the likelihood of individual training grant submission prior to matriculation.

While infrastructure support mechanisms are critical, outcomes are often improved when support is matched by an increase in consequences for failure to meet goals. Beginning in 2009, all postdoctoral trainees enter the program knowing that a minimum of three publications, including one by the end of the first year, are required during a 2-year fellowship. Beginning in 2013, a recommended timeline was introduced recommending four papers during a 2-year fellowship (one each 6 months). In an effort to reinforce the concept that trainees are responsible for their own careers [1], requirements and recommendations focus on papers first-authored by the postdoctoral fellow. Quarterly reviews with the trainees of progress and plans towards publication goals also occur.

To reinforce the importance of writing as an ongoing scientific activity, a postdoctoral Writing Group was initiated in 2008. To encourage independence, the group is self-administered by the postdoctoral trainees with the only requirement that it meet 4 h per month (generally either an hour weekly or 2 h twice per month) and that it focus on postdoctoral trainees reading and providing critical feedback on each other’s writing. Writing contributions can be papers or grants, but each postdoc is expected to be working on a scholarly product at all times. Faculty participate on 25 % of sessions; faculty rotate and equally share in this responsibility.

Outcome: Postdoctoral Publications

A postdoctoral publication was defined as any article with a publication date in the same years as the individual was enrolled in this postdoctoral program. Papers had to be either published or “in press”; submitted or in preparation articles were not included. For the four trainees currently enrolled in the program, the number of postdoctoral publications is estimated based on publication rate to date and estimated date of completing the postdoctoral program: if a trainee was 12 months through a projected 24-month postdoctoral training experience, the number of peer publications to date was doubled. Thus, if a postdoctoral trainee has not published in the first 12 months, his or her projected publications for the next 12 months is 0, irrespective of number of papers in preparation or under review.

Analysis

This postdoctoral training program, currently in its 36th year of funding, has generally been funded for six to eight postdoctoral trainees per year. With a modal postdoctoral training duration of 24 months, each year has an average matriculating cohort of three to four postdocs. In order to obtain sufficient numbers for analysis, postdocs were collapsed into decade matriculation periods (1978–1987, 1988–1997, 1998–2007) plus a final period (2008–2013) which corresponds to the period during which the program focused on increasing postdoctoral publication productivity. As previous work had suggested that there was no increase in sustained posttraining productivity associated with achieving more than four postdoctoral publications, postdoctoral publications rates were used both as a continuous measure and a measure where trainees with four or more postdoctoral publications were collapsed into a single 4+ publication category. Matriculation-period groups were compared using an ANOVA/ANCOVA for continuous measures and chi-square analyses for ordinal publication categories (SPSS version 22 [5]).

Two potential confounds, changes in mentor and changes in trainee preparation, were assessed using mentor publication impact factors (h-index) based on papers published up to the trainees’ matriculation date [4, 6] and the number of predoctoral publications, respectively.

Results

As of October 1, 2014, the 134 postdoctoral participants have published 308 publications while a postdoctoral trainee including 140 (42 %) as first author; when projected publications from current trainees are included, this increases to 337 publications or an average of 2.5 postdoctoral publications per postdoctoral fellow (range 0–10 postdoctoral publications).

There is a significant effect of matriculation period on postdoctoral publication productivity [F(3,130)=5.478, p=.001; Table 1]. Postdoctoral publication rates were higher for trainees who matriculated between 2008 and 2013 than for any of the three decade-long periods prior to 2008; the three prior decades were not significantly different from each other. The percentage of postdoctoral papers that had the postdoctoral trainee listed as first author was fairly consistent across matriculation periods (χ²=1.36, p=0.715). The 20 postdoctoral trainees who matriculated in 2008 or later are more than twice as likely to have published four or more (55 vs 26 %) and less than half as likely (25 vs 65 %) to publish two or less postdoctoral papers than are trainees who matriculated prior to 2008 (Fig. 2).

Table 1.

Relationship of matriculation period to number of pre- and postdoctoral publications, number and percentage of postdoctoral publications which are first-authored by the postdoctoral trainee, and primary research mentor impact factor (h-index) at the time of postdoctoral trainee matriculation

	Overall	Year of matriculation				Statistic	p	Post-hoc
		1978–1987	1988–1997	1998–2007	2008–2013
	n=134	n=29	n=42	n=43	n=20
# predoctoral publications (mean±S.D.)	420 3.1±3.8	76 2.6±3.4	101 2.4±4.3	156 3.6±3.5	87 4.4±3.3	F(3,130)=1.698	.171
# postdoctoral publications (mean±S.D.)	337 2.5±2.4	55 1.9±1.9	80 1.9±2.0	118 2.6±2.3	84 4.2±3.0	F(3,130)=5.478	.001	2008–2013>1978–1987, 1988–1997, 1998–2007
# (%) 1st author	140 (42 %)	20 (36 %)	36 (45 %)	47 (40 %)	37 (44 %)	Χ2 (3)=1.36	.715
Mentor h-index	25±16	19±12	27±18	26±16	29±20	F(3,130)=1.782	.154

Fig. 2.

For the University of Colorado Developmental Psychobiology Postdoctoral Research Training Program, by decade of the program, the percentage of trainees who publish zero, one, two, three, or four or more papers while a postdoctoral trainee

Confounding Factors

There was no significant effect of matriculation period on mentor publication impact factors nor on the number of trainees’ predoctoral publications (Table 1). While there was no relationship between trainee publication rate and mentor impact factor [t(134)=−.052; p=.551], predoctoral publication rates are predictive of postdoctoral publications rates [t(134)= −.416; p<.001]. The relationship between matriculation period and postdoctoral publication rates remains significant after controlling for predoctoral publications [F(3,129)= 4.014, p=.009].

Discussion

One primary goal for postdoctoral training programs is the development of individuals who will provide sustained contributions to the scientific community. Although institutional training programs account for only a portion of all postdoctoral training options, they provide consistency across a larger number of trainees and can therefore be useful situations in which to explore mechanisms for optimizing postdoctoral training experiences. The University of Colorado School of Medicine Developmental Psychobiology, Psychopathology, and Behavior is in its 37th year of continuous NIMH funding and trains basic, translational, clinical, and epidemiological postdoctoral scientists. A previous evaluation of this program, completed in 2006, identified the number of peer-reviewed papers published as a postdoc as a predictor of long-term sustained publication productivity; while increasing postdoctoral productivity from zero to two publications was associated with no increase in long-term productivity, three and, even more notably, four or more postdoctoral publications were a positive predictor. Given these results, this program initiated training changes designed to increase postdoctoral productivity.

For the first decade of the program (1978–1987), only 28 % of postdoctoral trainees published four or more papers. That number was stable over the next 20 years, with 24 % of trainees in the second decade (1988–1997) and 28 % of trainees in the third decade (1998–2007) publishing four or more papers during postdoctoral training years. However, after beginning a process to alter the program in a way designed to increase postdoctoral publication productivity, the percentage of postdoctoral trainees with four or more postdoctoral publications has doubled to 55 %. It appears that publication rates climb even within this most recent 6-year span; since 2011, only one trainee has not published at least three peer-reviewed papers; however, numbers are too small to be confident of this ongoing improvement.

The percentage of publications where the postdoctoral trainee is the first author does not significantly differ across matriculation periods and has remained remarkably stable over the entire duration of the program. This suggests that the increased productivity seen in the final matriculation period is reflective of true increased productivity and not due to external pressure to place trainees as more distal authors on papers they would not have been included on in the past. The lack of a change in mentor publication impact scores over time and the lack of a relationship between mentor impact scores and trainee postdoctoral publications argue against improved postdoctoral publication rates as being due to any changes in mentor quality over time. While an increase in the average number of predoctoral papers reported by entering trainees might suggest trainees are more prepared for publication by predoctoral programs, the effect is not significant and, even when analyses are covaried for predoctoral publication rates, the relationship between matriculation period and postdoctoral publication rates remains strong. Thus, better predoctoral training does not appear to be the reason behind improved productivity in more recent postdoctoral trainees. The last few years have seen increased difficulty in obtaining research funding, and it is possible that increases in postdoctoral productivity are due to postdoctoral trainees feeling increased competition for career advancement. However, there have been other difficult funding periods (e.g., the early 1990s), and no similar increase was observed during that time period. Thus, we conclude that changes in the postdoctoral training are the most likely etiologic factor in the recent increase in postdoctoral productivity.

While we have previously reported a correlation between postdoctoral publication rate and sustained scientific contribution, it is unclear if the relationship is causal and it is possible that increasing postdoctoral publication rates will have no impact on long-term scientific involvement. Additional follow-up work will be necessary to determine distal benefits. Another limitation of this analysis is the focus on performance of trainees from a single postdoctoral program. Several program variables have remained consistent over the duration of the program, including the use of developmental psychobiology to study developmental psychopathology; the inclusion of both physician and non-physician trainees; the multi-specialty, multi-departmental, and multi-institutional nature of the faculty; and independence from departmental administration (minimizing outside pressure to promote non-research academic development or fill service needs). While holding these variables constant improves the ability to assess individual differences, it is unclear if results will generalize to other programs. Future work comparing this program to similar programs is necessary to address program variables.

Comments on the Process of Program Changes

Even though all postdoctoral faculty were involved with and voted for changes in the program designed to increase postdoctoral publication productivity, change is always difficult. Concerns raised by faculty during the change process followed one of three themes. First, most mentors experienced a sequential training model in their own training and, since it worked for them, were uncomfortable changing to a new model. Second, a concurrent training model includes the idea that, early in the postdoctoral experience, a trainee is writing manuscripts on already collected data. In many cases, this already collected data was generated by the trainee during predoctoral training. Some faculty expressed reservations that a trainee be working on something from outside the mentor’s lab during the postdoctoral experience believing this would diminish the time and therefore benefit of exposure to the mentor’s work. Third, and most commonly, mentors quickly endorsed the concept, but then noted that their field of research was unique and should be the only field excused from the requirement. This concern was raised by mentors across content and methodological approaches: basic scientists were convinced that their efforts often required a series of experiments while clinical scientists only needed one simple study to publish; clinical scientists reported that, unlike basic scientists who could grow tissue or order animals, for clinical science, it took a long time to recruit and retain a sufficiently large population, particularly for longitudinal studies; neuroimagers and statistical geneticists described both the cost and highly collaborative nature of their work; epidemiology and other large dataset investigators noted the ongoing logistical difficulties with collecting, cleaning, and even getting access to large data sets. In short, while mentors generally had no problems understanding the cognitive skill necessary and quality of mentors who utilized other methodologies, each was also highly aware of the hurdles in their own research while less aware of hurdles involved in other methodological approaches.

Programmatic change focused on changes in requirements for trainees. For example, the program application was changed to include three separate questions requiring the applicant to concretely state (a) what papers on what data would be written in the first year, (b) what skills would be learned as part of ongoing projects in the mentor’s lab, and (c) what independent project would be developed during the postdoctoral period. In our program, applicants to the program must identify a mentor prior to application and the mentor’s input is necessary for a successful application; thus, the application change not only introduced the idea of concurrent training to the applicant but required the mentor to conceptualize the trainee’s experience in the same way.

Consequences were associated with concrete milestones. For example, trainees who failed to meet the first year’s requirements, which include an accepted publication not just a submission, within 10 months after starting the program were not guaranteed a second year of funding. Similarly, we formalized the possibility of a third year of postdoctoral funding and required trainees to have appropriately met milestones during their second year, including in the number of papers published, to be eligible to apply.

By focusing programmatic change on trainee requirements, we projected that the strong desire of mentors to help their trainees meet the new requirements would reshape mentor beliefs and behavior. The evidence provided in this manuscript suggests that belief was accurate.

Summary

After identifying postdoctoral publication rate as an important predictor of sustained scientific contribution, this postdoctoral program initiated program changes designed to enhance postdoctoral publication rates. In the first 6 years after initiating a program change process, 75 % of trainees published three or more postdoctoral papers; the percentage of postdoctoral trainees who published four or more papers during their postdoctoral experience has doubled, to 55 %. In previous work, three, and even more notably, four, or more postdoctoral publications were predictive of greater likelihood of sustained scientific publication activity. Program attention to postdoctoral publications can increase postdoctoral publication rates, hopefully leading to more sustained scientific productivity by its graduates.

Implications for Educators.

• Recommendations for postdoctoral training programs include the inclusion of stated objectives and goals.

• Postdoctoral trainees who publish three articles are more likely and those who publish four or more articles are even more likely to sustain publication activity after completing the postdoctoral training.

• Program attention to postdoctoral publications can increase postdoctoral publication rates, hopefully leading to more sustained scientific productivity by its graduates.