Abstract
Training of doctoral students as part of the next generation of the biomedical workforce is essential for sustaining the scientific enterprise in the United States. Training primarily occurs at institutions of higher education, and these trainees comprise an important part of the workforce at these institutions. Federal investment in the support of doctoral students in the biological and biomedical sciences is distributed differently than the distribution of students across different types of institutions, for example, public vs private. Institutions in states that historically receive less federal support for research also receive less support for doctoral student training. Doctorates at different types of institution exhibit little difference in research productivity, with the exception of citations, and subsequent receipt of additional NIH awards. Thus, training outcomes, which are related to the quality of the student and training environment, are similar across different institutions. Research productivity of doctoral students does not correlate with the number of F31s awarded to an institution. Factors that correlate with F31 funding include R01 funding levels and program size. The findings suggest strategies for institutions to increase success at securing F31s and modification of policy to promote more equitable distribution of F31s across institutions.
Keywords: biomedical workforce, citation, doctoral student, F31, NIH, productivity, publication
1. INTRODUCTION
Long‐term investment in biomedical research resulted in the rise of US research to a preeminent position in the world and the development of the knowledge base to combat disease and save lives. During the years of a flat NIH budget, concerns about sustaining the biomedical research enterprise re‐emerged amid concerns that the standing of US biomedical research in the world would erode and the development of medicines to manage disease would be impaired. Recommendations to sustain the enterprise focused on sustained and predictable funding for research, reduction of regulatory burden, modifying the biomedical workforce and workforce training. 1 , 2 , 3 , 4 , 5 The focus of this study relates to the biomedical workforce.
The estimated size of the US biomedical workforce between 2010 and 2014 was 289,147 to 305,500 persons. 6 , 7 Fifty percent of these employees worked in government and industry. 7 Sixty percent of the workforce had an MS degree and between 30 and 46% held PhDs. 6 , 7 The estimated personnel supported by NIH extramural programs in FY09 was 121,465 full time effort positions, which provided support to 247,457 individuals. 8
Institutions of higher education are the primary training centers for the biomedical workforce. Consequently, a large percentage of the biomedical workforce at these institutions is comprised of trainees. In 2019, 46% of basic research expenditures occurred at higher education institutions. 9 Thus, the investment in trainees is a critical investment in the future of biomedical research and in current basic research, which is the foundation for tomorrow's cures.
In the biological and biomedical sciences alone, the workforce at higher educational institutions in 2019 was estimated to include 45,466 doctoral students and 19,631 postdoctoral fellows. 10 The balance between students and postdocs varies between institutions. At public institutions, approximately 68% of the workforce is comprised of doctoral students, whereas at private institutions, ~55% are doctoral students. The doctoral student component of the labor force at land grand institutions is similar to that of public institutions. IDeA (Institutional Development Award) institutions (located in states that are historically underfunded by the NIH) have the largest percentage of the workforce comprised of graduate students (~72%). Approximately 10% of the research personnel at each type of institution consists of non‐faculty researchers and the postdoctoral component of the workforce ranges from 18% (IDeA institutions) to 34% (private institutions). As a result of the difference compositions of the workforce, policies affecting trainees will have different impacts at different institutions.
The primary sources of support for doctoral student training are fellowships, traineeships, research assistantships, and teaching assistantships. The NIH and other federal sources provide significant resources for doctoral and postdoc training in the form of fellowships and institutional training grants, which provide traineeships to individuals. In FY17, the NIH investment in training grants was $1.5 billion, which was 4.5% of the NIH budget. 11 The NIH F series of grants provide individual fellowships, the T series of grants provide institutional training grants, and the K series of grants provide support for mentored career development. Evaluation of these grant mechanisms suggest that they are effective and promote research productivity and career advancement to subsequent research funding. Support on a training grant or an individual NIH fellowship increases success at securing a K award. 12 K awardees publish more papers than non‐awardees and are more successful at securing NIH research grants. 11 , 12 , 13 Similar analysis of F32 awardees, who are postdocs supported by individual fellowships from the NIH, shows increased research productivity in number of publications and increased success at securing NIH research grants. 14 , 15
There are a number of concerns related to workforce training. First, the number of PhDs awarded exceeds the number of traditional positions, that is, academic faculty positions. 2 , 16 , 17 , 18 Despite the fact that PhDs are employed in many positions outside of academia, for example, in government or industry, 1 , 2 in the biomedical sciences there is a surplus of PhDs relative to job openings across all sectors. 6 , 19 Second, since many doctorates work outside of academia, they may require additional skillsets, which are not currently part of their training program, for success. 2 , 17 , 18 Third, the length of time spent in training as a doctoral student and in postdoctoral training is a concern since this delays career progression and reduces earnings. 4 , 20 Recommendations were made to reduce the number of trainees, modify curricula to prepare graduates for broader careers, shorten the time to degree, and increase the transparency of outcomes of training programs. There have also been recommendations to shift training support to federal fellowships and training grants. 4 , 20 This is expected to strengthen training since review of the proposed training plan is part of the review of applications. Further increased trainee success is expected, given the productivity and successful career progression of NIH‐supported trainees. While the merits of this recommendation are clear, careful consideration of its full potential impact is required. This study begins this assessment by examining the university biomedical workforce, how trainees (specifically doctoral students) are supported at different institutions and the research productivity and success of students at different institutions.
2. MATERIALS AND METHODS
2.1. Sources of data
NIH award data and publication data associated with NIH awards were captured from NIH Reporter (https://reporter.nih.gov/). Information about trainees and the workforce at academic institutions is from the Graduate Students and Postdoctorates in Science and Engineering Survey (GSS) (https://www.nsf.gov/statistics/srvygradpostdoc/). The data tables for each year from 1985 through 2019 were used for this analysis. 21 Data regarding doctorates awarded are from the Survey of Earned Doctorates (SED) (https://www.nsf.gov/statistics/srvydoctorates/). The data tables for each year from 1994 through 2020 were used for this analysis. 22 GSS and SED searches for data on numbers of doctorates, postdocs, and sources of support were restricted to the field of study of Biological and Biomedical Sciences. Information about doctoral dissertations was retrieved from the ProQuest Dissertations & Theses Global database. PubMed was searched for publication data using the BioEntrez package from Python. 23 Data on research expenditures were from the NCSES National Patterns of R&D Resources 9 (https://www.nsf.gov/statistics/natlpatterns/). Data on career outcomes for doctoral students are from the Coalition for Next Generation Life Science 24 (https://nglscoalition.org/).
2.2. Cohorts
Data were collected on two cohorts of students: (1) F31 awardees from 2001 to 2016 inclusive, and (2) all doctorates completing their dissertation from 2012 through 2016. For the latter cohort, doctorates and their advisors were identified from the ProQuest Dissertations & Theses Global database (www.proquest.com—access provided by West Virginia University Libraries). The database was searched for institutions designated as doctorate‐granting institutions by GSS (235 institutions), and the search limited to subjects listed as Biological and Biomedical Sciences in the GSS. This provided a list of doctorates and advisors for each of 235 institutions. Doctorates with no affiliated advisor listed were excluded from the analysis (694 doctorates).
2.3. Publications and citations
F31 awardee publication data were extracted from NIH Reporter in April and May 2021. The data were curated to ensure all publications include the awardee as an author and to correct misspellings of awardee names in the author list. Curation impacted a few percent of all publications listed. The number of first author publications and total publications for each F31 awardee were calculated using a Python script. Publications by the doctorates identified from the ProQuest database were found by searching PubMed using the Python BioEntrez package. 23 Search criteria for first author publications included the doctorate's name as first author, the advisor's name as an author and the institution name as affiliation. The search for first author publications was performed in May and June 2021. Search criteria for total publications included the doctorate's and advisor's names and the institution name as affiliation. The search for total publications was performed in November and December 2021. In many cases, more than one advisor was listed per doctorate. PubMed searches using each advisor as a co‐author with the doctoral student were performed to identify all publications associated with the doctoral candidate. The year of each publication was also captured. Citations for both cohorts were counted using the BioEntrez package for Python, and the search was performed in November 2021. 23 PMIDs were used to search PubMed for the list of PMCIDs that cite each publication and the number of PMCIDs counted.
2.4. Grant success by F31 awardees
F31 awardees successfully securing additional NIH funding were identified by matching NIH PI IDs between F31 awards and other NIH awards including F32, K99, R15, R21, and R01 awards through fiscal year 2021.
2.5. Institutional review board
The West Virginia University IRB approved the study. IRB approval numbers are WVU Protocol#: 2202521185 and WVU Protocol#: 2203537777.
2.6. Statistics
None of the data exhibit a Gaussian distribution; therefore, nonparametric statistics were used for the analysis. Some of the data presented in the tables cannot be directly compared statistically, since the groups are not independent. Statistical comparisons were made between independent groups, that is, public vs private Institutions, land grant vs non‐land grant institutions, and IDeA vs non‐IDeA institutions. Publication and citation data were compared using the Mann–Whitney test. Rates of securing additional NIH funding data were compared using Fisher's exact test. Details of the statistical comparisons are presented in Table S1.
3. RESULTS
3.1. How are doctoral students supported?
The major mechanisms of financial support of doctoral students are fellowships, traineeships, research assistantships, and teaching assistantships, as defined by the Survey of Graduate Students and Postdoctorates in Science and Engineering (see legend to Figure 1). 10 In 2019, 12.5% of doctoral students in the biomedical and biological sciences were supported on federal fellowships and traineeships. Excluding non‐training grant eligible students, approximately 17.1% of doctoral students were supported on federal fellowships and traineeships. Between 2017 and 2019, federal agencies supported 30.4% of students on fellowships and 61% of students on traineeships. The majority of federal support came from the NIH and NSF. NIH supported 51.1% of federal fellowships during this period and NSF supported 38.7% of federal fellowships. The NIH supported a larger percentage of students on federal traineeships (90.9%) than the NSF (2.5%).
FIGURE 1.
Source of financial support for doctoral students in biological and biomedical sciences. The source of financial support for doctoral students from 2017 to 2019 was retrieved from the GSS. 21 The percentage of all doctoral students supported by different mechanisms at public, private, land grant, and IDeA institutions is shown. (A) Support from all sources. (B) Support from federal sources. (C) Support from the NIH. Fellows = students supported by individual fellowships. That is, a competitive award to the doctoral student, for example, an F31. Trainees = students supported on training grants, i.e. a financial award given to doctoral students selected by the institution, for example, supported on a T32. RAs = research assistantships, that is, students receiving stipends to perform research. TAs = teaching assistantships, that is, students receiving stipends to teach. There is a caveat that different institutions may define “Fellows”, “Trainees” and “Research Assistants” differently. Note that federal sources do not support teaching assistantships.
A larger percentage of students at private institutions are supported on fellowships and traineeships than students at public institutions (Figure 1A). Trainee support was also examined at two other classifications of university, land grant and IDeA institutions. Land grant institutions were originally designated by states for donations of federal land or money to establish the university and frequently include making contributions to benefit the state as part of their mission. The Institutional Development Award (IDeA) program was congressionally mandated in 1992 as a mechanism to provide investment to build research infrastructure in states with historically low levels of funding from the NIH. The percentage of students supported by research assistantships and teaching assistantships at public, land grant, and IDeA institutions is greater than the percentage of students supported by similar mechanisms at private institutions (Figure 1A). Nearly 50% of doctoral students at public, land grant, and IDeA institutions are supported by research assistantships, and approximately 20% are supported by teaching assistantships. The federal government provides critical support for doctoral student training. In 2019, federal funds supported 29% of students supported by fellowships, 60.5% of the students supported by traineeships and 47.4% of research assistantships. 10 Institutions supported 97.3% of teaching assistantships. 10 The percentage of doctoral students receiving federal support at different types of institutions is shown in Figure 1B. The highest percentage of students receiving federal fellowships are at private and land grant Institutions. The percentage of students supported on federal traineeships at private institutions is more than twice as high as at any other type of institution. The percentage of students at IDeA institutions supported by federal fellowships and traineeships lags behind all the other types of institutions. The percentage of students supported by federal research assistantships at different types of institutions is more comparable. The major source of federal funding for biomedical research is the NIH. Support for trainees on NIH fellowships, traineeships, and research assistantships is illustrated in Figure 1C. Since NIH is the major source of funding in the area, the trends parallel the trends seen in total federal funding. Interestingly, the highest percentage of students supported by NIH research assistantships are at private institutions.
3.2. Where do doctoral students train?
In 2019, approximately 65% of biological and biomedical sciences doctoral students trained at public universities and 35% at private institutions (Table 1). 10 The data are consistent between the Graduate Students and Postdoctorates in Science and Engineering Survey (GSS) and the Survey of Earned Doctorates (SED) (Table 1). 10 , 25 Land grant institutions trained between 25% and 30% of doctoral students. Institutions in IDeA states trained approximately 11% of doctoral students in 2019.
TABLE 1.
Number of trainees in 2019 (Biological and biomedical sciences trainees).
| Institution type | # Doctoral students (GSS) | % Doctoral students (GSS) | # Doctorates awarded (SED) | % Doctorates awarded (SED) |
|---|---|---|---|---|
| Public | 29,371 | 64.6% | 5594 | 64.3% |
| Private | 16,095 | 35.4% | 2785 | 32.0% |
| Land grant | 13,461 | 29.6% | 2156 | 24.8% |
| IDeA | 4829 | 10.6% | 988 | 10.9% |
| All | 45,466 | 8702 |
Note: Data on the distribution of doctoral students among different types of institutions (from the GSS) and the number of doctorates awarded by different types of institutions (from the SED).
The SED provides a historical record of the number of doctorates awarded (the GSS only parses graduate students into masters and doctoral students since 2017). The number of biological and biomedical doctorates awarded has plateaued since 2009 and has declined since 2015 (Figure 2A). This trend is encouraging given longstanding concerns about the large numbers of students earning PhDs each year. 2 , 16 , 17 , 18 The percentage of students earning doctoral degrees from public, private, land grant, and IDeA institutions has been relatively constant since 1994 (Figure 2B).
FIGURE 2.
Historical record of doctorates awarded in biological and biomedical sciences and F31s awarded at US Universities. The number of doctorates awarded in biological and biomedical sciences from 1994 to 2020 was retrieved from the SED. 22 The number of doctorates awarded (A) and percentage of total doctorates awarded (B) at private, public, land grant, and IDeA institutions was calculated. The number of F31s for each fiscal year from 1985 to 2021 was tallied using data acquired using NIH Reporter. The number of awards (C) and the percentage of awards (D) at public, private, land grant, and IDeA institutions are plotted.
3.3. Trainee support from the NIH
A history of NIH support for doctoral student fellowships (F31 awards) is illustrated in Figure 2C. For the last twenty years, there was a sustained increase in the number of F31 awards. The percentage of F31 awards made to different types of institutions is illustrated in Figure 2D. The striking observation from this analysis is that approximately 50% of F31s are awarded to students at private institutions and 50% are awarded to students at public institutions. This is in contrast to the observations that 65% of doctoral students train at public institutions and 35% train at private institutions (Figure 2B). Even more striking is the disparity between the doctorates in the biomedical and biological sciences awarded at land grant and IDeA institutions and the F31 awards at those institutions. Averaging the last 5 years of data from Figure 2, ~25.9% of biomedical and biological science doctorates were awarded at land grant institutions and ~ 11.3% at IDeA institutions. In the same time frame, ~13.0% of F31s were awarded to students at land grant institutions and ~ 5.6% were awarded to students at IDeA institutions. Thus, doctoral support at different types of institutions is not comparable to the number of trainees at these institutions. These observations raise questions about support for the training of doctoral students including the factors leading to successful competition for F31 awards, strategies to employ to increase F31 success rates and policies to support the training of the biomedical workforce of the future.
3.4. Correlating F31 funding with R01 funding and program size
The percentage of doctorates awarded, F31s held, and R01s held at different types of institutions in 2019 is compared in Table 2. The percentage of F31s awarded at different types of institutions more closely reflects the percentage of R01s than the percentage of doctorates. Focusing only on institutions holding an F31 in FY19, the number of F31s held at different institutions in FY19 correlated with the number of R01s held at those institutions (Figure 3A). The number of F31s held at different institutions in FY19 also correlated with the number of doctorates awarded at those institutions in 2019 (Figure 3B).
TABLE 2.
Comparison of earned doctorates, F31s awarded and R01s awarded at different types of institutions in FY19.
| Public | Private | Land Grant | IDeA | |
|---|---|---|---|---|
| Percent doctorates 2019 | 64.3% | 32.0% | 24.8% | 11.4% |
| Percent F31s FY19 | 49.9% | 47.4% | 14.1% | 5.5% |
| Percent R01s FY19 | 53.9% | 46.1% | 15.4% | 6.7% |
Note: The percentage of doctorates awarded is based upon data from the SED. The percentage of F31 and R01 awards was extracted from NIH data using NIH Reporter.
FIGURE 3.
Correlations with number of F31 awards. (A) The number of F31s at individual institutions is plotted against the number of R01s held at the institution during FY19. F31 and R01 data were extracted using NIH Reporter. (B) The number of F31s at individual institutions during FY19 is plotted against with number of doctorates awarded at the institution in 2019. F31 data were calculated using data from NIH Reporter. The number of doctorates awarded is from the SED. 22
3.5. Training grant eligibility
The proportion of the doctoral student population that was training grant eligible (TGE) was estimated from the GSS by tallying the total number of US doctoral students and the total number of doctoral students. In 2019, 73.1% of biological and biomedical graduate students were training grant eligible. Private institutions had the highest percentage of TGE doctoral students (74.9%) followed by public institutions (72.2%). Land grant institutions (70.7%) and IDeA institutions (69.6%) had lower percentages of TGE doctoral students. Part of the difference in the number of F31 awards at different institutions can be attributed to the number of students who are training grant eligible.
3.6. Awardee performance
Training outcomes were measured as success in advancing on an independent research trajectory and research productivity. Research success was measured by securing additional funding from NIH such as an F32/K99 and R series grants. Research productivity was measured by the number of publications and citations. The cohort of all F31 awardees from all NIH Institutes from 2001 to 2016 inclusive was selected for this analysis. This cohort consisted of 4579 F31 awardees from public institutions, 3714 from private institutions, 1323 from land grant institutions, and 544 F31 recipients from IDeA institutions. The dates for inclusion in the cohort provided a large number of individual records for analysis and provided time for the cohort to complete training, finish publications, accumulate citations, and secure additional funding. Note that a single awardee can appear in multiple categories, for example, an awardee at an IDeA institution that is a land grant institution and a public university.
Approximately 9 to 10% of this cohort of F31 awardees were also awarded an F32 and 2 to 3% were awarded a K99 (Table 3). Fisher's exact test revealed no difference in the rate of success at securing a more advanced NIH fellowship between students holding F31s at different types of institutions. Approximately 6 to 7% of this cohort of F31 awardees were also awarded an R01 and 4 to 5% were awarded an R21 (Table 3). Fisher's exact test revealed no difference in the rate of success at securing an R01 for F31 awardees at different types of institutions. There was a significant difference in the percentage of F31 awardees at private and public institutions who were also awarded an R21. A small number of awardees secured R15, R35, R41, R42, R43, and R44 grants. The small numbers precluded meaningful analysis. Using securing additional NIH funding as a metric of scientific success, this analysis demonstrates that F31 awardees from different types of institutions are comparably successful. A striking observation was where F31 awardees eventually held their R01s (Table 4). More than half of the F31 awardees at private institutions who secured R01 funding held their R01s at private institutions. Approximately 75% of F31 awardees at public or land grant institutions, who successfully competed for R01 awards, held their R01s at public institutions. Almost 40% of F31 awardees at IDeA institutions held their R01s at IDeA institutions. While there are multiple factors impacting the career trajectory of doctoral students, this observation suggests that the distribution of fellowships across different institutions might contribute to shaping the distribution of successful faculty in the future.
TABLE 3.
Percentage of F31 awardees who secure additional fellowships or R series awards.
| Success rate for additional NIH funding | |||||
|---|---|---|---|---|---|
| Institution type | Number F31 awardees | Secured F32 | Secured K99 | Secured R01 | Secured R21 |
| Public | 4579 | 9.78% | 2.64% | 7.16% | 4.78% |
| Private | 3714 | 9.50% | 3.28% | 6.27% | 3.63% a |
| Land grant | 1323 | 9.90% | 2.27% | 7.63% | 5.14% |
| IDeA | 544 | 8.27% | 2.21% | 6.07% | 4.96% |
Fisher's exact test, Private ≠ Public (p = 0.01).
TABLE 4.
Comparison of types of institutions where F31 awardees hold R01s.
| Public F31 | Private F31 | Land Grant F31 | IDeA F31 | |
|---|---|---|---|---|
| Public R01 | 74.1% | 34.3% | 75.2% | 69.7% |
| Private R01 | 19.8% | 55.4% | 21.8% | 24.2% |
| Land grant R01 | 18.6% | 11.2% | 31.7% | 18.2% |
| IDeA R01 | 13.7% | 5.6% | 16.8% | 39.4% |
The research productivity of F31 awardees at different institutions was also compared. First, the publication record of F31 awardees as listed in NIH REPORTER was compared. The average number of first author publications for the 8293 F31 awardees in the cohort was 1.71 ± 1.79 papers, and the average number of total publications was 2.57 ± 2.83 papers (see Table 5). The average number of first author publications by F31 awardees at different types of institutions ranged from 1.58 ± 1.61 (private institutions) to 1.87 ± 1.87 (land grant Institutions) papers. The average number of first author publications by F31 awardees at public universities was greater than F31 awardees at private institutions. The average number of first author publications of F31 awardees at land grant institutions was greater than F31 awardees at non‐land grant universities. The median number of first author publications at each type of institution was the same (Table 5). The average number of total publications ranged from 2.47 ± 2.6 (private institutions) to 2.72 ± 3.02 (land grant institutions) papers (Table 5). The average number of total publications by F31 awardees at public institutions exceeded the average at private institutions and was statistically different.
TABLE 5.
Comparison of number of first author and total publications of F31 awardees.
| All F31s | Public F31s | Private F31s | Land grant F31s | IDeA F31s | |
|---|---|---|---|---|---|
| First author publications | |||||
| # F31 awardees | 8293 | 4579 | 3714 | 1323 | 544 |
| # Papers (1sts) | 14,142 | 8258 | 5884 | 2474 | 926 |
| % Published 1st | 75.4% | 74.4% | 75.2% | 73.5% | |
| Avg ± SD | 1.71 ± 1.79 | 1.8 ± 1.92 | 1.58 ± 1.61 a | 1.87 ± 1.87 a | 1.7 ± 1.82 |
| 95% CI | 1.75 to 1.86 | 1.53 to 1.64 | 1.77 to 1.97 | 1.55 to 1.86 | |
| Median | 1 | 1 | 1 | 1 | |
| Total publications | |||||
| # F31 awardees | 8293 | 4579 | 3714 | 1323 | 544 |
| # Papers (Total) | 21,318 | 12,147 | 9171 | 3599 | 1424 |
| % Published | 80.0% | 79.9% | 79.7% | 77.9% | |
| Avg ± SD | 2.57 ± 2.83 | 2.65 ± 3 | 2.47 ± 2.6 b | 2.72 ± 3.02 | 2.62 ± 3.34 |
| 95% CI | 2.57 to 2.74 | 2.39 to 2.55 | 2.56 to 2.88 | 2.34 to 2.90 | |
| Median | 2 | 2 | 2 | 2 | |
Mann–Whitney test, Public > Private (p < 0.0001), Land Grant > others (p = 0.002).
Mann–Whitney test, Public > Private (p = 0.026).
The number of citations for first author papers was determined as a measure of the impact of the publications. F31 awardees at private institutions were cited an average of 27.79 times per first author publication (Table 6). This was a higher citation rate than first author publications by F31 awardees from the other types of institutions, which ranged from 19.12 ± 33.49 to 21.1 ± 34.75 citations per paper. The number of first author publications included primary publications and reviews. The F31 awardees at private institutions publish more reviews (9.9%) than F31 awardees at other types of institutions (6.5 to 7.7%). However, this does not account for the difference in number of citations, since F31 awardees from private institutions have more citations for both their first author reviews and their first author primary publications (Table 6). The differences in citations for reviews are not significantly different, while the differences in citations of primary first author papers are significant.
TABLE 6.
Comparison of citations of first author papers of F31 awardees.
| Public | Private | Land Grant | IDeA | |
|---|---|---|---|---|
| # First Authors | 3453 | 2763 | 995 | 400 |
| # First author papers (all) | 8272 | 5896 | 2476 | 933 |
| Number citations | 174,533 | 163,869 | 48,098 | 17,835 |
| Avg. citations ± SD | 21.10 ± 34.75 | 27.79 ± 46.71 a | 19.43 ± 31.86 a | 19.12 ± 33.49 a |
| 95% CI | 20.35 to 21.85 | 26.60 to 28.99 | 18.18 to 20.68 | 16.96 to 21.27 |
| Median | 11 | 14 | 11 | 10 |
| # Reviews | 636 | 584 | 156 | 61 |
| % Reviews | 7.7% | 9.9% | 6.3% | 6.5% |
| Avg Cit./Rev ± SD | 36.36 ± 49.65 | 41.91 ± 60.51 | 34.19 ± 43.18 | 37.64 ± 58.24 |
| 95% CI | 32.50 to 40.23 | 36.99 to 46.83 | 27.36 to 41.02 | 22.72 to 52.56 |
| Median | 21 | 23 | 20 | 17 |
| # First author papers (excl revs) | 7636 | 5312 | 2320 | 872 |
| Avg Cit./1st ± SD | 19.83 ± 32.90 a | 26.24 ± 44.67 | 18.43 ± 30.71 | 17.82 ± 30.67 |
| 95% CI | 19.09 to 20.57 | 25.04 to 27.44 | 17.18 to 19.68 | 15.78 to 19.86 |
| Median | 11 | 14 | 10 | 10 |
Mann–Whitney test, Private > Public (p < 0.0001), Land Grant < others (p < 0.0001), IDeA < others (p < 0.0001).
F31 awardees at different institutions were comparably successful at securing subsequent NIH funding. The research productivity of F31 awardees at different institutions was comparable in terms of publications, although F31 awardees at private institutions were cited more frequently. The equivalent outcomes are incongruent with the disproportionate distribution of F31 awards between private and public institutions. However, it is also possible that the success and productivity of the F31 awardees relate to the fact that they were supported by an NIH fellowship. A comparison of the publication records of postdoctoral fellows who applied for F32 fellowships revealed increased productivity for F32 awardees compared with applicants who were not awarded an F32. 15 It is therefore important to compare the research productivity of a general cohort of doctoral students who graduate from different types of institutions.
3.7. Research Productivity of Doctoral Students
Doctorates and their advisors were identified from the ProQuest Dissertations & Theses Global database. A list of doctorates publishing a dissertation in the biological and biomedical sciences from the 235 institutions designated as doctorate‐granting institutions was compiled. 10 Publications by the doctorates were found by searching PubMed. These searches provided lists of publications for 42,922 doctoral students completing their dissertations from 2012 to 2016 inclusive. This time frame allowed time for completion of publications after completion of training and the accumulation of citations. Further, this cohort contains the peers of a subset of the F31 cohort analyzed above. The searches captured 74,093 first author publications and 182,563 total publications.
Research productivity was measured as the number of first author publications, number of total publications, and citations of first author publications. The PubMed search strategy will identify papers associated with specific doctoral students who trained with specific advisors at specific institutions, but false positives will also be included. These might be due to a shared name or might be authentic publications by the doctorate but from a different phase of their career, for example, from undergraduate or postdoctoral studies. To partially correct for these issues, the publications chosen for analysis were constrained by date of publication and the top 1% of doctorates based upon numbers of publications were excluded as outliers.
The distribution of the differences between the publication year of papers and the year of dissertation is illustrated in Figure 4. The 5th percentile of first author publications and total publications was −3, that is, 3 years prior to the dissertation year. The 95th percentile for first author publications was +4, and the 95th percentile of total publications was +6. To exclude publications that were less likely to be associated with the doctoral project, that is, publications more distant from the dissertation year, the papers selected for analysis were constrained by time. First author publications between the 5th percentile (year −3) and year +2 and total publications between the 5th percentile (year −3) and year +3 were included in the analyses. The upper limit is arbitrary and allows 2 years after the dissertation for publication of first author papers and 3 years after the dissertation for publication of co‐authored papers. These constraints restricted the analysis to 84.1% of the first author publications from the PubMed search (62,323 papers) and 78.5% of the total publications from the PubMed search (143,381 papers). Exclusion of the doctorates greater than the 99th percentile in number of publications constrained the analysis to 42,481 doctorates (99%) with 58,602 first author publications (79.1%) and to 42,359 doctorates (98.7%) with 128,813 total publications (70.6%). Of the first author publications, 53,502 were primary peer‐reviewed publications and 5100 were reviews, and 119,316 of the total publications were primary publications. The analysis of productivity was performed on these publications. Overall, students in the cohort published an average of 1.38 ± 1.35 first author papers. Excluding published reviews, the cohort published an average of 1.26 ± 1.25 first author papers and 3.04 ± 2.91 total papers.
FIGURE 4.
Date of publication related to dissertation year. The year of publication relative to the year of dissertation was calculated by subtracting the dissertation year from the publication year. The number of first author publications and the number of total publications in each year relative to dissertation year are plotted. The 5th and 95th percentiles of first author and total publications are shown. A subset of the entire dataset, restricted based on time of publication relative to the dissertation year, was used for the analysis. Lines delineate the subset of publications used for the analysis of first author papers (“Firsts analysis”) and total papers (“Total analysis”).
The productivity of students at different types of institution was measured (Table 7). A higher percentage of students at private institutions published first author papers (70.5%) than students at public/land grant institutions (66.1%/65.1%) and at IDeA institutions (61.1%). Students at private institutions also published more reviews (12.7%) than students at public/land grant institutions (9.8%/9.5%) and IDeA institutions (7.7%). The average number of first author publications, excluding reviews, was similar between students at private, public, and land grant institutions (1.26 ± 1.27 papers to 1.27 ± 1.21 to papers). While the differences are statistically different, the differences are not meaningful and students at these institutions exhibit similar productivity by this measure. Students at IDeA institutions exhibit a lower level of productivity (1.16 ± 1.21 papers). Given that a higher percentage of students at private institutions publish a first author publication, it was of interest to compare the average number of publications among only the doctorates who have published (and excluding reviews). This analysis changes the outcome. Students at public universities publish more first author papers on average (1.93 ± 1.08 papers) than students at private institutions (1.84 ± 1.04 papers). The average first author publications of students at land grant universities (1.99 ± 1.11 papers) is significantly higher than students at all other universities and the average first author publications of students at IDeA institutions (1.94 ± 1.04 papers) is not significantly different than students at non‐IDeA institutions.
TABLE 7.
First author publications in years −3 to +2 Inclusive (Relative to dissertation year).
| All | Public | Private | Land grant | IDeA | |
|---|---|---|---|---|---|
| All first author publications | |||||
| # Doctorates | 42,481 | 27,907 | 14,574 | 12,760 | 4413 |
| # Papers | 58,602 | 38,043 | 20,559 | 17,512 | 5499 |
| % Published 1st | 66.1% | 70.5% | 65.1% | 61.1% | |
| Avg 1sts ± SD | 1.38 ± 1.35 | 1.36 ± 1.37 | 1.41 ± 1.33 a | 1.37 ± 1.39 a | 1.25 ± 1.35 a |
| 95% CI | 1.35 to 1.38 | 1.39 to 1.43 | 1.35 to 1.40 | 1.21 to 1.29 | |
| Median | 1 | 1 | 1 | 1 | |
| First author publications excluding reviews | |||||
| % Publish Rev. | 9.8% | 12.7% | 9.5% | 7.7% | |
| # Papers | 53,502 | 34,994 | 18,508 | 16,188 | 5114 |
| Avg 1st ± SD | 1.26 ± 1.25 | 1.26 ± 1.27 | 1.27 ± 1.21 b | 1.27 ± 1.3 | 1.16 ± 1.27 b |
| 95% CI | 1.24 to 1.27 | 1.25 to 1.29 | 1.25 to 1.29 | 1.12 to 1.20 | |
| Median | 1 | 1 | 1 | 1 | |
| First author publications excluding reviews and doctorates who have not published | |||||
| # Doctorates | 28,183 | 18,108 | 10,075 | 8159 | 2641 |
| # Papers | 53,502 | 34,994 | 18,508 | 16,188 | 5114 |
| Avg 1sts ± SD | 1.90 ± 1.07 | 1.93 ± 1.08 c | 1.84 ± 1.04 | 1.98 ± 1.10 c | 1.94 ± 1.09 |
| 95% CI | 1.92 to 1.95 | 1.82 to 1.86 | 1.96 to 2.01 | 1.90 to 1.98 | |
| Median | 2 | 2 | 2 | 2 | |
Mann–Whitney test, Private > Public (p < 0.0001), Land Grant < others (p = 0.008), IDeA < others (p < 0.0001).
Mann–Whitney test, Private > Public (p = 0.0005), IDeA < others (p < 0.0001).
Mann–Whitney test, Public > Private (p < 0.0001), Land Grant > others (p < 0.0001).
A larger percentage of doctoral students at private institutions also publish more total papers (81.3%) than students at public/land grant institutions (77.2%/76%) and students at IDeA institutions (73.1%) (Table 8). Excluding reviews, students at private institutions publish more total papers than students at other types of institutions, averaging 2.90 ± 2.68 papers. Students at public and land grant institutions publish comparable numbers of papers (2.77 ± 2.73 papers vs 2.74 ± 2.73 papers), while students at IDeA institutions publish fewer total papers (2.54 ± 2.68 papers). These differences are virtually eliminated if the analysis includes only the doctorates who have published. Only students at IDeA institutions publish fewer papers on average (3.51 ± 2.58 papers) than students at non‐IDeA institutions. Based upon total publications, students at different types of institutions exhibit comparable productivity with deviations within 5% of average total publications.
TABLE 8.
Total publications in years −3 to +3 inclusive (relative to dissertation year).
| All | Public | Private | Land grant | IDeA | |
|---|---|---|---|---|---|
| All total author publications | |||||
| # Doctorates | 42,359 | 27,823 | 14,536 | 12,733 | 4414 |
| # Papers | 128,813 | 82,980 | 45,833 | 37,437 | 12,054 |
| % Published | 77.2% | 81.3% | 76.0% | 73.1% | |
| Avg Pubs ± SD | 3.04 ± 2.91 | 2.98 ± 2.92 | 3.15 ± 2.89 a | 2.94 ± 2.92 a | 2.73 ± 2.89 a |
| 95% CI | 2.95 to 3.02 | 3.11 to 3.20 | 2.89 to 2.99 | 2.65 to 2.82 | |
| Median | 2 | 3 | 2 | 2 | |
| All total author publications excluding reviews | |||||
| % Publish Rev. | 16.5% | 20.0% | 16.1% | 14.5% | |
| # Papers | 119,316 | 77,151 | 42,165 | 34,860 | 11,226 |
| Avg Pubs ± SD | 2.82 ± 2.71 | 2.77 ± 2.73 | 2.90 ± 2.68 a | 2.74 ± 2.73 a | 2.54 ± 2.70 a |
| 95% CI | 2.74 to 2.81 | 2.86 to 2.95 | 2.69 to 2.79 | 2.46 to 2.62 | |
| Median | 2 | 2 | 2 | 2 | |
| First author publications excluding reviews and doctorates who have not published | |||||
| # Doctorates | 33,020 | 21,304 | 11,716 | 9577 | 3197 |
| # Papers | 119, 316 | 77,151 | 42,165 | 34,860 | 11,226 |
| Avg Pubs ± SD | 3.61 ± 2.56 | 3.62 ± 2.59 | 3.60 ± 2.52 | 3.64 ± 2.58 | 3.51 ± 2.58 b |
| 95% CI | 3.59 to 3.66 | 3.55 to 3.65 | 3.59 to 3.69 | 3.42 to 3.60 | |
| Median | 3 | 3 | 3 | 3 | |
Mann–Whitney test, Private > Public (p < 0.0001), Land Grant < others (p < 0.0001), IDeA < others (p < 0.0001).
Mann–Whitney test, IDeA < others (p = 0.0006).
Productivity was also measured by determining the number of citations for each first author publication, which is intended as a measure of impact in the field (Table 9). Given the difference between reviews published by students at different types of institutions, the citations of reviews and citations of publications excluding reviews were compared separately. The average number of citations of publications by students at private institutions exceeded the citations of publications of students at other institutions. Excluding reviews, the average number of citations for doctoral students at private institutions was 26.79 ± 70.29 citations per paper. Students at public and land grant institutions were cited 18.48 ± 104.99 and 18.31 ± 148.38 times per paper. IDeA institution doctoral students had an average of 13.17 ± 18.56 citations per paper. Similarly, the average number of citations per review published by private institution doctoral students (42.52 ± 83.15 citations per review) exceed those by students at public, land grant, and IDeA institutions (31.70 ± 43.89, 29.73 ± 36.81, and 27.25 ± 31.98 citations per review respectively). This analysis suggests that publications by students at private institutions have a larger impact than publications by students at other types of institutions.
TABLE 9.
Citations of first author publications from years −3 to +2 inclusive (relative to dissertation year).
| All | Public | Private | Land grant | IDeA | |
|---|---|---|---|---|---|
| Citations of all first author publications | |||||
| # Doctorates | 28,703 | 18,437 | 10,266 | 8307 | 2695 |
| # First author papers (All) | 58,602 | 38,043 | 20,559 | 17,512 | 5499 |
| Avg citations | 22.64 ± 92.36 | 19.54 ± 101.6 | 28.37 ± 71.87 a | 19.19 ± 143.1 a | 14.17 ± 20.14 a |
| 95% CI | 18.52 to 20.56 | 27.39 to 29.35 | 17.07 to 21.31 | 13.64 to 14.71 | |
| Median | 10 | 14 | 10 | 9 | |
| Citations of first author reviews | |||||
| # Reviews | 5100 | 3049 | 2051 | 1324 | 385 |
| Avg citations | 36.06 ± 62.94 | 31.71 ± 43.91 | 42.54 ± 83.19 b | 29.76 ± 36.83 b | 27.32 ± 32.03 b |
| 95% CI | 30.15 to 33.27 | 38.93 to 46.14 | 27.77 to 31.74 | 24.11 to 30.53 | |
| Median | 20 | 22 | 20 | 17 | |
| Citations of first author papers (Excluding reviews) | |||||
| # First author papers | 53,502 | 34,994 | 18,508 | 16,188 | 5114 |
| Avg citations | 21.36 ± 94.59 | 18.48 ± 105.1 | 26.80 ± 70.33 c | 18.32 ± 148.5 c | 13.19 ± 18.57 c |
| 95% CI | 17.38 to 19.58 | 25.79 to 27.82 | 16.03 to 20.61 | 12.68 to 13.69 | |
| Median | 10 | 14 | 9 | 8 | |
Mann–Whitney test, Private > Public (p < 0.0001), Land Grant < others (p < 0.0001), IDeA < others (p < 0.0001).
Mann–Whitney test, Private > Public (p < 0.0001), Land Grant < others (p = 0.0016), IDeA < others (p = 0.0014).
Mann–Whitney test, Private > Public (p < 0.0001), Land Grant < others (p < 0.0001), IDeA < others (p < 0.0001).
Another factor to consider in the analysis of scholarly activity of doctoral students is the time until the first publication. The year of matriculation into a graduate program for each student is not known, but the year of publication of the dissertation is known. Therefore, time until first publication is measured relative to the year of the dissertation using the total publication data constrained to years −3 to +3 inclusive (including papers and reviews). The first publication of students at private institutions was −1.22 ± 1.44 years relative to the dissertation year (Table 10). Doctoral students at public and land grant institutions published their first paper −1.09 ± 1.51 and − 1.06 ± 1.52 years relative to the year of the dissertation. Doctoral students at IDeA institutions published their first paper −0.98 ± 1.55 years prior to the dissertation year. Students at private institutions published earlier than students at other institutions. While the difference is significant, the difference is unlikely to make a significant real difference as private institution students publish an average of only 1.6 to 2.9 months earlier than students at other institutions.
TABLE 10.
Time to first publication—total papers in years −3 to +3 inclusive (relative to dissertation year).
| All | Public | Private | Land grant | IDeA | |
|---|---|---|---|---|---|
| # Doctorates | 42,359 | 27,823 | 14,536 | 12,733 | 4414 |
| # Published | 33,290 | 21,480 | 11,810 | 9672 | 3226 |
| Avg time 1st Pub (± SD) | −1.14 ± 1.49 | −1.09 ± 1.51 | −1.22 ± 1.44 a | −1.06 ± 1.52 a | −0.98 ± 1.56 a |
| 95% CI | −1.11 to −1.07 | −1.25 to −1.20 | −1.10 to −1.03 | −1.03 to −0.92 |
Mann–Whitney test, Private < Public (p < 0.0001), Land Grant > others (p < 0.0001), IDeA > others (p < 0.0001).
These results support the conclusion that there is little meaningful difference between the average number of first author publications or the average number of total publications by students at different types of institutions (excluding reviews). However, the first author papers published by students at private institutions apparently have a larger impact on their field, based upon the higher number of citations per paper.
3.8. Comparison of the F31 cohort with the doctorate cohort
The research productivity of the F31 Awardees and the cohort of doctoral students cannot be directly compared using the NIH Reporter data and the PubMed data. To make this comparison, F31 awardees were identified in the doctoral cohort by (1) matching awardee name with the name of a doctorate, (2) matching the institution where the F31 was held with the institution where the dissertation was submitted, and (3) matching at least one PMID from NIH Reporter with a PMID for the PubMed search for the doctorates. Matching all these criteria resulted in the identification of 997 doctorates with first author publications who were F31 awardees and 1057 doctorates with any publication who were F31 awardees. The performance of these F31 awardees was compared with the rest of the doctorate cohort (Table 11). Since F31 awardees without publications in NIH Reporter could not be identified among the doctorate cohort using the criteria employed, this comparison excluded all doctorates who had not published. The F31 awardees published more first author papers (2.37 ± 1.27 to 2.03 ± 1.16) and more total papers (4.82 ± 3.04 to 3.84 ± 2.74) than non F31 awardees. Citations of first author reviews by F31 awardees were comparable to the number by their peers. However, first author papers by F31 awardees, excluding reviews, receive more citations than those of their peers (27.20 ± 49.60 to 21.12 ± 95.96). F31 awardees also publish significantly earlier than non F31 awardees. On average, the F31 Awardees publish their first paper 5 months earlier that the non F31 awardees. These findings suggest that F31 awardees are more productive than a large cohort of their peers.
TABLE 11.
Comparison of productivity of F31 awardees with their peers.
| Peers | F31 awardees | |
|---|---|---|
| First author publications | ||
| # Doctorates | 27,706 | 997 |
| # First author papers | 56,244 | 2358 |
| Avg Firsts ± SD | 2.03 ± 1.16 a | 2.37 ± 1.28 |
| 95% CI | 2.02 to 2.04 | 2.29 to 2.44 |
| Median | 2 | 2 |
| Total publications | ||
| # Doctorates | 32,233 | 1057 |
| # Total papers | 123,709 | 5104 |
| Avg Pubs ± SD | 3.84 ± 2.74 a | 4.83 ± 3.04 |
| 95% CI | 3.81 to 3.87 | 4.65 to 5.01 |
| 3 | 4 | |
| Citations of first author reviews | ||
| # Reviews | 4831 | 269 |
| Avg citations ± SD | 36.01 ± 63.85 b | 37.06 ± 43.80 |
| 95% CI | 34.21 to 37.81 | 31.80 to 42.32 |
| 21 | 24 | |
| Citations of first author publications (excluding reviews) | ||
| # First author papers | 51,413 | 2089 |
| Avg citations ± SD | 21.12 ± 95.96 a | 27.20 ± 49.60 |
| 95% CI | 20.29 to 21.95 | 25.07 to 29.33 |
| 22 | 39 | |
| Time to first publication | ||
| Avg ± SD | −1.13 ± 1.49 a | −1.56 ± 1.29 |
| 95% CI | −1.14 to −1.11 | −1.64 to 1.48 |
Mann–Whitney test, Peers < Awardees (p < 0.0001).
Mann–Whitney test, Peers < Awardees (p = 0.15).
4. DISCUSSION
The rationale for shifting support of graduate student training to fellowships and training grants is that peer review will strengthen training programs and provide some level of independence from the trainee's research mentors. 1 , 2 , 5 The comparison of research productivity of F31 awardees versus their peers is consistent with an F31 fellowship having a positive impact on graduate student training. However, it is also possible that factors leading to increased productivity of doctoral students also lead to an increased likelihood of receiving an F31 award. Additional, more rigorous analysis is required to distinguish these possibilities. There is compelling evidence that F32 awards and K awards increase productivity and success of trainees. Studies have analyzed F32 and K award applicants and compared awardees with comparable applicants who were not funded. In each of these studies, awardees exhibited higher research productivity and were more successful in the pursuit of an independent career. 12 , 13 , 14 , 15 Given the success of these programs, expanding support would benefit a larger cohort of trainees and could qualitatively strengthen the biomedical workforce.
A striking observation about F31 funding is the divergence of the number of awardees from the number of doctorates at different types of institutions. Doctorate students from public, land grant, and IDeA institutions are underrepresented among F31 awardees, based upon the number of trainees at those institutions. A recent analysis of the NSF Graduate Research Fellowships Program (GRFP) and institutional factors that correlate with funding success identified affiliation with a public institution as a factor decreasing the likelihood of success. 26 The number of F31 awards at an institution correlates with the number of R01 awards at the institution, suggesting levels of NIH funding are a factor influencing F31 awards among institutions. Consistent with this observation, a recent analysis found that the top 14 institutions in terms of total NIH funding were also the top 14 institutions in amount of support by F series, T series, and K series awards, with a very strong correlation between the top 7 in each category. 27 NIH funding at an institution can affect the success of fellowship applications in several ways. It is important since it provides the resources necessary for trainees to complete their training. Further, sustained R series funding provides resources for recruitment of trainees into the laboratory workforce, building the training record of the PIs. Increases in the NIH budget also correlate at the macro level with increases in graduate enrollment. 28 , 29 The links of funding to training and training records may skew the distribution of F31 awardees to institutions with historically strong records of funding. Thus, very strong candidates in outstanding training environments may be less competitive than candidates in average training environments with very strong funding records.
The number of F31 awards received by an institution correlates with the number of doctorates awarded at the institution. Program size was also associated with funding success in the NSF GFRP. 26 This may reflect a real or perceived better training program at these institutions. It may also reflect more applications from a larger doctoral pool. Data for F31 applications are not available from the NIH precluding further analysis of the applications from different types of institutions. In addition, part of the difference in number of F31 awardees at different institutions may relate to differences in the number of training grant eligible students in attendance.
A confounding factor to this analysis is the intended career paths of doctoral students. A major factor necessitating non‐traditional, non‐academic career paths is the production of PhDs that far exceed the number of academic positions available. 2 , 17 , 18 Consequently, many PhDs enter the workforce in non‐traditional, non‐academic positions. 2 , 30 Large surveys of national cohorts of doctorates indicate that many individuals' primary career interests are in non‐academic and even non‐research careers. 31 , 32 , 33 Pursuing an independent research fellowship may not appear as a benefit to doctorates considering these career paths. The data from these surveys are aggregated, and information regarding career path interest at different institutions and types of institutions is not readily available. Several studies at individual institutions demonstrate similar trends at Emory, Georgia Tech, and UCSF. 34 , 35 To address whether career path interest differs among students at different types of institution will require a more detailed analysis of similar surveys.
Another factor that might affect the number of F31 awards at different types of institutions are the career outcomes expected at different types of institutions. Institutions producing a high percentage of graduates pursuing research or academic careers might attract more F31 awards. Career outcome data from PhD programs are sparse. Multiple workforce training analyses have recommended implementation of mechanisms to increase transparency of training outcomes. 2 , 17 , 18 , 36 , 37 The Coalition for Next Generation Life Science has taken the lead in providing outcomes data on PhD and postdoctoral training at member institutions. 24 Thirty‐four of forty‐six coalition members have posted graduate program outcomes data, and twenty‐one institutions have posted career outcomes. Their use of a common taxonomy for career sectors, career types, and job functions allows useful comparisons, 38 even though all institutions do not post precisely the same information. Comparison of the percentage of students in further training or performing research (career types) upon graduation from public institutions, private institutions, and land grant institutions is comparable (Figure S1). The percentage of graduates from public and private and land grant institutions working in academia (career sector) upon graduation is also comparable (Figure S1). Similarly, the percentage of students working in a research career and in academia 10 years after graduation are comparable between students from these types of institutions (Figure S1). This small data set does not demonstrate differences in career outcomes between different types of institutions, with the caveat that data are available from a small subset of all institutions. It will be interesting to make additional comparisons as more data become available from other institutions.
4.1. Differences in performance at different institutions
This analysis provided insight into the research productivity of doctoral students, which is of general interest in calls for transparency of outcomes of individual training programs. The results demonstrate an average of 1.38 first author publications (1.26 excluding reviews) and 3.04 total publications for a doctoral student. First author publications are cited an average of 22.64 times. The peak time for publications is in the year the dissertation is completed and the second highest number of publications appearing in the year after completion of the dissertation.
Comparison of the research productivity of F31 awardees from different types of institutions reveals little real difference in the numbers of first author publications and total publications, but papers published by F31 awardees at private institutions are cited more often than those of other F31 awardees. Comparison of research productivity of the general population of doctoral students demonstrates that a higher percentage of students at private institutions publish, but little real difference in the numbers of papers published after taking this into account. Again, papers from students at private institutions are cited more often. The increased number of citations suggests that these papers are more impactful on the field. Quality and significance of papers are important factors in citation rates. Increased citations could reflect the higher quality of the student. Interestingly, this may also reflect the composition of the biomedical workforce at different types of institutions. The highest proportion of postdocs are in the workforce at private institutions, and interactions with postdocs as part of a cascading mentorship model of training are reported to increase the development of research skills in doctoral students. 39 Citations are also impacted by extrinsic factors unrelated to the quality and significance of the paper. 40 Higher citations could reflect the quality and prestige of the mentor and the institution. The number of previous publications by the author and the number of previous citations are predictive of the number of citations of future papers. 40 Since doctoral students are just beginning their publication history, extrinsic factors related to citations reflect the record of the mentor, who is usually the senior author of the publication.
4.2. Location of training and independent position
The percentage of trainees securing additional NIH funding was comparable for F31 awardees from different types of institutions. Six to eight percent of F31 awardees in the cohort secured R01 funding. This compares with reports in the literature that about 20% of F32 awardees between 2000 and 2010 also received an R01, ~20% to 40% of K01, K08, and K23 awardees also received an R01 and ~30% to 55% of K99 awardees also secured an R01. 11 , 41 The award site for mentored career development awards (K series) impacts the distribution of trainees in their independent career since 60–80% of K01/K08/K23 awardees who receive an R01 hold their first R01 at the same institution as their K award. 41 While the percentage of F31 awardees receiving R01s was not different between awardees at different institutions, the locations where the F31 awardees held their R01 awards was different. Greater than 55% of F31 awardees at private institutions who received a subsequent R01 held their R01 at a private institution. Seventy‐four percent of F31 awardees at public institutions who received an R01 held their R01 at a public institution. Approximately 39% of F31 awardees at IDeA institutions held subsequent R01s at IDeA institutions. This unexpected observation suggests that the distribution of F31 awards could have an impact on where NIH‐supported trainees establish their independent research careers.
4.3. Limitations
There are a number of limitations associated with this study. Measures of research success were limited to NIH grants. Information on funding from many other federal agencies and foundations is not as robust as information from the NIH. Further, there is no mechanism to link an F31 awardee with another grant from a different agency. The emphasis on research success focused on principal investigators of subsequent grants. This is only one metric to measure research success and individuals achieving success in other ways were not included. The publication records of F31 awardees in NIH Reporter include publications related to the award, but may not include publications prior to the award or publications related to different projects. Thus, their publication record might be incomplete. Identification of doctorates and advisors for the analysis of performance of a large cohort of doctoral students used the ProQuest Dissertations & Theses Global database. While this is a large database it is incomplete and there are institutions that do not deposit all of their dissertations in this database. Some dissertations in the dissertations database have no information on the advisor, which was required for this analysis. These records (694) were excluded. The PubMed search strategy will yield false positives when there are multiple instances of doctorates and advisors with similar names affiliated with the same institution.
4.4. Recommendations
Institutions could take a number of actions to increase the likelihood of success of its doctoral students in winning an F31 award, for example, by developing training programs to better prepare graduate students to compete for F31 awards. One strategy is to incorporate grant writing into the graduate curriculum. 42 , 43 , 44 There is merit in this approach to provide training in a number of competencies required by all graduate students, for example, critical thinking, hypothesis development, experimental design, rigor, and reproducibility, in addition to scientific writing. A second strategy is to develop F31 grant writing workshops/bootcamps/writing groups for students working on a fellowship application. 45 , 46 Different levels of assessment of the success of these programs were performed. Student surveys demonstrate that the students felt that they learned useful information, improved their skills, and gained confidence in grant writing. 42 , 44 , 45 , 46 Evaluation of the quality of proposals demonstrated improvement over the duration of one granting writing course. 44 The success rates of fellowship applications from participants in two of these programs exceeded the national average. 42 , 46 In the assessment of one program, participants in the program were compared with non‐participants from the same institution. Submission rates and success rates for the participants were approximately twice that of non‐participants. 45 Thus, the limited number of published reports on grant writing training for graduate students demonstrates a positive impact on grant writing skills, submission rates, and success rates. Literature assessing the effectiveness of similar programs to assist postdocs and junior faculty members provide additional evidence that these types of interventions are effective. Studies show that program participants feel that their grant writing skills improve following participation in these programs. 47 , 48 , 49 , 50 One study comparing submissions and awards before and after participation in a faculty grant writing program showed an increase in submissions and awards after completion of the program. 49 The National Research Mentoring Network recruits a national cohort of junior faculty to participate in multiple mentoring programs to facilitate grant writing success. The success rate of securing funding by participants in these programs exceeds the national average. 50 , 51 While the number of studies is limited, the evidence supports the effectiveness of grant writing programs making this a viable strategy to increase the competitiveness of doctoral students for F31s.
An early time until first publication distinguishes doctorates at private universities from doctorates at other types of universities and F31 awardees from their peers. Mechanisms created to increase the early productivity of doctoral students, in the form of co‐authored publications could increase their competitiveness for F31s. Recently, a similar strategy was implemented in a postdoctoral program to increase productivity. 52 The program was modified so participants would publish a paper within 1 year of matriculation, using unpublished data from graduate studies or previously unpublished data from the postdoc mentor's research program. The modification succeeded since the research productivity of participants exceeded the productivity of previous cohorts in the program prior to the programmatic change. 52 Time to publication was not measured as part of the study and thus it is unclear if the program change accelerated publication or just increased productivity. One strategy designed to shorten the time to complete a doctoral program includes coordination of undergraduate curricula and research experiences with graduate curricula and dissertation research. 53 This strategy is expected to accelerate the progress of a cohort of doctoral students and might also result in accelerated publications for doctoral students. While we have recently implemented these types of programs at our institution, it is too soon to assess their impact.
Changes in policy could also redress the skewed distribution of awardees among institutions. Mechanisms other than NIH grants to support student projects, including institutional commitments to support students and their mentors in the event of a lapse of funding, should be valued during fellowship review. Evaluation of the training program should emphasize innovation in training at the institution and evidence demonstrating the effectiveness of training innovations during review. This suggestion is similar to the previous recommendation to amend the peer‐review process to value quality of training of all students at an institution. 20 Finally, recognizing that the type of institution where an awardee holds an F31 can influence where the successful trainees hold their R01, managing the distribution of F31 awards could be an effective strategy to develop science infrastructure in underrepresented regions of the country.
AUTHOR CONTRIBUTIONS
The author was the sole contributor to the conceptualization, design, data collection, analysis, and drafting the manuscript.
CONFLICT OF INTEREST STATEMENT
The author has no relevant financial or non‐financial interests to disclose.
Supporting information
Figure S1.
Table S1.
ACKNOWLEDGMENTS
Thanks to Dr. Marieta Gencheva for insightful comments on the manuscript. MS is the director of the Cell & Molecular Biology and Biomedical Engineering Training Program (T32 GM133369). This work was supported by the WVU Department of Biochemistry and Molecular Medicine.
Schaller MD. Research productivity and training support for doctoral students in the biological and biomedical sciences. FASEB BioAdvances. 2023;5:131‐148. doi: 10.1096/fba.2022-00117
DATA AVAILABILITY STATEMENT
Data are publicly available from NIH Reporter (https://reporter.nih.gov/), the Graduate Students and Postdoctorates in Science and Engineering Survey (GSS) (https://www.nsf.gov/statistics/srvygradpostdoc/), the Survey of Earned Doctorates (SED) (https://www.nsf.gov/statistics/srvydoctorates/), the ProQuest Dissertations & Theses Global database (https://about.proquest.com/en/products‐services/pqdtglobal/), PubMed (https://pubmed.ncbi.nlm.nih.gov/), and the Coalition for Next Generation Life Science (https://nglscoalition.org/).
REFERENCES
- 1. Alberts B, Kirschner MW, Tilghman S, Varmus H. Rescuing US biomedical research from its systemic flaws. Proc Natl Acad Sci USA. 2014;111:5773‐5777. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. National Institutes of Health . Biomedical Research Workforce Working Group Report. National Institutes of Health; 2012. [Google Scholar]
- 3. National Academy of Sciences . The Postdoctoral Experience Revisited. National Academies Press; 2014. [PubMed] [Google Scholar]
- 4. American Academy of Arts and Sciences . Restoring the Foundation: the Vital Role of Research in Preserving the American Dream: Report Brief. American Academy of Arts & Sciences; 2014. [Google Scholar]
- 5. National Research Council . Research Universities and the Future of America: Ten Breakthrough Actions Vital to our nation's Prosperity and Security. National Academies Press; 2012. [Google Scholar]
- 6. Mason JL, Johnston E, Berndt S, Segal K, Lei M, Wiest JS. Labor and skills gap analysis of the biomedical research workforce. FASEB J. 2016;30:2673‐2683. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Winkler AE, Levin SG, Allison MT. Investigating the US biomedical workforce: gender, field of training, and retention. Sci Public Policy. 2019;46:913‐926. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Pool LR, Wagner RM, Scott LL, et al. Size and characteristics of the biomedical research workforce associated with US National Institutes of Health extramural grants. FASEB J. 2016;30:1023‐1036. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. National Center for Science and Engineering Statistics (NCES) . National Patterns of R&D Resourcees: 2018–19 Data Update. Alexandria; 2021. [Google Scholar]
- 10. National Center for Science and Engineering Statistics (NCSES) . Survey of Graduate Students and Postdoctorates in Science and Engineering: Fall 2019. Alexandria; 2021. [Google Scholar]
- 11. Conte ML, Omary MB. NIH career development awards: conversion to research grants and regional distribution. J Clin Invest. 2018;128:5187‐5190. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Mason JL, Lei M, Faupel‐Badger JM, et al. Outcome evaluation of the National Cancer Institute career development awards program. J Cancer Educ. 2013;28:9‐17. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Nikaj S, Lund PK. The impact of individual mentored career development (K) awards on the research trajectories of early‐career scientists. Acad Med. 2019;94:708‐714. [DOI] [PubMed] [Google Scholar]
- 14. Heggeness ML, Ginther DK, Larenas MI, Carter‐Johnson FD. The Impact of Postdoctoral Fellowships on a Future Independent Career in Federally Funded Biomedical Research. National Bureau of Economic Research; 2018. [Google Scholar]
- 15. Jacob BA, Lefgren L. The impact of NIH postdoctoral training grants on scientific productivity. Research Policy. 2011;40:864‐874. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Larson RC, Ghaffarzadegan N, Xue Y. Too many PhD graduates or too few academic job openings: the basic reproductive number R0 in academia. Syst Res Behav Sci. 2014;31:745‐750. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. National Academies of Sciences Engineering and Medicine . Graduate STEM Education for the 21st Century. National Academies Press; 2018. [Google Scholar]
- 18. National Academies of Sciences Engineering and Medicine . The Next Generation of Biomedical and Behavioral Sciences Researchers: Breaking through. National Academies Press (US); 2018. [PubMed] [Google Scholar]
- 19. Xue Y, Larson RC. STEM Crisis or STEM Surplus? Yes and Yes. Monthly Labor Review; 2015, 2015. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Pickett CL, Corb BW, Matthews CR, Sundquist WI, Berg JM. Toward a sustainable biomedical research enterprise: finding consensus and implementing recommendations. Proc Natl Acad Sci USA. 2015;112:10832‐10836. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. National Center for Science and Engineering Statistics . (1985. –2019) Survey of Graduate Students and Postdoctorates in Science and Engineering. National Science Foundation. [Google Scholar]
- 22. National Center for Science and Engineering Statistics . (1994. –2020) Doctorate Recipients from US Universities. National Science Foundation. [Google Scholar]
- 23. Cornish TC, Kricka LJ, Park JY. A biopython‐based method for comprehensively searching for eponyms in Pubmed. MethodsX. 2021;8:101264. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Blank R, Daniels RJ, Gilliland G, et al. A new data effort to inform career choices in biomedicine. Science. 2017;358:1388‐1389. [DOI] [PubMed] [Google Scholar]
- 25. National Center for Science and Engineering Statistics (NCSES) . (2020) Doctorate Recipients from US Universities: 2019. National Science Foundation. [Google Scholar]
- 26. Graddy‐Reed A, Lanahan L, Ross NM. Influences of academic institutional factors on R&D funding for graduate students. Sci Public Policy. 2017;44:834‐854. [Google Scholar]
- 27. Conte ML, Schnell S, Ettinger AS, Omary MB. Trends in NIH‐supported career development funding: implications for institutions, trainees, and the future research workforce. JCI Insight. 2020;5(17):e142817. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. Blume‐Kohout ME, Clack JW. Are graduate students rational? Evidence from the market for biomedical scientists. PLoS One. 2013;8:e82759. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Blume‐Kohout ME, Adhikari D. Training the scientific workforce: does funding mechanism matter? Res Policy. 2016;45:1291‐1303. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Sinche M, Layton RL, Brandt PD, et al. An evidence‐based evaluation of transferrable skills and job satisfaction for science PhDs. PLoS One. 2017;12:e0185023. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Roach M, Sauermann H. The declining interest in an academic career. PLoS One. 2017;12:e0184130. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Sauermann H, Roach M. Science PhD career preferences: levels, changes, and advisor encouragement. PloS One. 2012;7:e36307. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Gibbs KD Jr, Griffin KA. What do I want to be with my PhD? The roles of personal values and structural dynamics in shaping the career interests of recent biomedical science PhD graduates. CBE Life Sci Educ. 2013;12:711‐723. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34. St Clair R, Hutto T, MacBeth C, Newstetter W, McCarty NA, Melkers J. The “new normal”: adapting doctoral trainee career preparation for broad career paths in science. PLoS One. 2017;12:e0177035. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Fuhrmann CN, Halme DG, O'Sullivan PS, Lindstaedt B. Improving graduate education to support a branching career pipeline: recommendations based on a survey of doctoral students in the basic biomedical sciences. CBE Life Sci Educ. 2011;10:239‐249. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36. Singh S, Gammie A, Lorsch J. Catalyzing the modernization of graduate education. ASM Microbe. 2016;11:96‐97. [Google Scholar]
- 37. Teitelbaum MS. Structural disequilibria in biomedical research. Science. 2008;321:644‐645. [DOI] [PubMed] [Google Scholar]
- 38. Mathur A, Brandt P, Chalkley R, et al. Evolution of a functional taxonomy of career pathways for biomedical trainees. J Clin Transl Sci. 2018;2:63‐65. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39. Feldon DF, Litson K, Jeong S, et al. Postdocs' lab engagement predicts trajectories of PhD students' skill development. Proc Natl Acad Sci USA. 2019;116:20910‐20916. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40. Onodera N, Yoshikane F. Factors affecting citation rates of research articles. J Assoc Inf Sci Technol. 2015;66:739‐764. [Google Scholar]
- 41. Pickett CL. The increasing importance of fellowships and career development awards in the careers of early‐stage biomedical academic researchers. PloS One. 2019;14:e0223876. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42. Eissenberg T. Teaching successful grant writing to psychology graduate students. Teach Psychol. 2003;30:328‐330. [Google Scholar]
- 43. Kahn RA, Conn GL, Pavlath GK, Corbett AH. Use of a grant writing class in training PhD students. Traffic. 2016;17:803‐814. [DOI] [PubMed] [Google Scholar]
- 44. Leak RK, O'Donnell LA, Surratt CK. Teaching pharmacology graduate students how to write an NIH grant application. Am J Pharm Educ. 2015;79:138. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45. Botham CM, Brawn S, Steele L, Barrón CB, Kleppner SR, Herschlag D. Biosciences proposal Bootcamp: structured peer and faculty feedback improves trainees' proposals and grantsmanship self‐efficacy. PLoS One. 2020;15:e0243973. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46. Mims MC, Hodgson EE, Kuehne LM, Siple MC. The benefits of workshopping graduate fellowships: a how‐to guide for graduate students and early career scientists. Ideas Ecol Evol. 2016;9: 24‐39. [Google Scholar]
- 47. Freel SA, Smith PC, Burns EN, Downer JB, Brown AJ, Dewhirst MW. Multidisciplinary mentoring programs to enhance junior faculty research grant success. Acad Med. 2017;92:1410‐1415. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48. Harwood EM, Jones AR, Erickson D, et al. Early career biomedical grantsmanship self‐efficacy: validation of an abbreviated self‐assessment tool. Ann N Y Acad Sci. 2019;1445:17‐26. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49. Smith JL, Stoop C, Young M, Belou R, Held S. Grant‐writing bootcamp: an intervention to enhance the research capacity of academic women in STEM. Bioscience. 2017;67:638‐645. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50. Weber‐Main AM, Thomas‐Pollei KA, Grabowski J, Steer CJ, Thuras PD, Kushner MG. The proposal preparation program: a group mentoring, faculty development model to facilitate the submission and funding of NIH grant applications. Acad Med. 2021;97:53‐61. [DOI] [PubMed] [Google Scholar]
- 51. Weber‐Main AM, McGee R, Eide Boman K, et al. Grant application outcomes for biomedical researchers who participated in the National Research Mentoring Network's Grant writing coaching programs. PloS One. 2020;15:e0241851. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52. Ross RG, Greco‐Sanders L, Laudenslager M. An institutional postdoctoral research training program: increasing productivity of postdoctoral trainees. Acad Psychiatry. 2016;40:207‐212. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53. Schaller MD, Barbier M. FastTrack, a strategy to shorten time to degree. FASEB BioAdvances. 2021;3:482‐489. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Figure S1.
Table S1.
Data Availability Statement
Data are publicly available from NIH Reporter (https://reporter.nih.gov/), the Graduate Students and Postdoctorates in Science and Engineering Survey (GSS) (https://www.nsf.gov/statistics/srvygradpostdoc/), the Survey of Earned Doctorates (SED) (https://www.nsf.gov/statistics/srvydoctorates/), the ProQuest Dissertations & Theses Global database (https://about.proquest.com/en/products‐services/pqdtglobal/), PubMed (https://pubmed.ncbi.nlm.nih.gov/), and the Coalition for Next Generation Life Science (https://nglscoalition.org/).