Measuring Success in Global Health Training: Data From 14 Years of a Postdoctoral Fellowship in Infectious Diseases and Tropical Medicine

Summary

The Burroughs Wellcome Fund/American Society for Tropical Medicine and Hygiene postdoctoral fellowship program was evaluated by several metrics. Awardees have won grants, authored peer-reviewed publications, and achieved faculty status. The program’s overall return on investment was judged to be 11.9.

Abstract

Background.

In modern academic medicine, especially in the fields of infectious diseases and global health, aspiring physician-scientists often wait years before achieving independence as basic, translational, and clinical investigators. This study employed mixed methods to evaluate the success of the Burroughs Wellcome Fund/American Society for Tropical Medicine and Hygiene (BWF/ASTMH) global health postdoctoral fellowship in promoting scientific independence.

Methods.

We examined quantitative data obtained from the National Institutes of Health (NIH) and qualitative data provided by the ASTMH and program participants to assess BWF/ASTMH trainees’ success in earning NIH grants, publishing manuscripts, and gaining faculty positions. We also calculated the return on investment (ROI) associated with the training program by dividing direct costs of NIH research grants awarded to trainees by the direct costs invested by the BWF/ASTMH fellowship.

Results.

Forty-one trainees received fellowships between 2001 and 2015. Within 3 years of completing their fellowships, 21 of 35 (60%) had received career development awards, and within 5 years, 12 of 26 (46%) had received independent research awards. Overall, 22 of 35 (63%) received 1 or more research awards. BWF/ASTMH recipients with at least 3 years of follow-up data had coauthored a mean of 36 publications (range, 2–151) and 29 of 35 (82%) held academic positions. The return on investment was 11.9 overall and 31.8 for fellowships awarded between 2001 and 2004.

Conclusions.

Between 2001 and 2015, the BWF/ASTMH postdoctoral training program successfully facilitated progress to scientific independence. This program model underscores the importance of custom-designed postdoctoral training as a bridge to NIH awards and professional autonomy.

One hundred years ago, the transition from clinical training to launching a career as an independent physician-researcher was relatively seamless for many bright, academically inclined physicians [1]. In the last 2 decades, this professional arc has lengthened [2] due, in part, to competition for scarce research funds [3]. In the future, the delayed retirement of many members of the current generation of physician-researchers will further exacerbate competition for grants.

To add to the challenge, for young professionals who hope to pursue academic careers in tropical medicine and global health, the greater administrative hurdles and periods of time spent traveling and working abroad can further delay the timely attainment of scientific independence.

Recognizing this trend, in 2000, the Burroughs Wellcome Fund (BWF) and the American Society of Tropical Medicine and Hygiene (ASTMH) created a mentored postdoctoral award designed to support trainees’ salaries and direct research costs. With the stated mission of attracting superior infectious diseases trainees and retaining them within the academic research sphere, the program originally selected and supported 1 fellow per year. In 2006, an additional financial commitment allowed the fellowship to expand to 3 fellows per year, each of whom received 24 months of funding. During the second year of support, the fellows were now expected to receive an academic appointment from their home institution. The revised guidelines further stipulated that fellows undertake research broadly related to global health (eg, projects pertaining to clinical, operational, or diagnostic issues) under the guidance of 2 mentors—1 from their home institution and 1 from their host institution in a low- to middle-income country (LMIC)—and to spend at least 3 months during each year of support in their overseas research site. The overseas site had to be within an LMIC. A final requirement was that awardees invest 75%–80% effort on research during the entire fellowship term.

From 2001 to 2004, the BWF/ASTMH award was $50 000 per year plus research expenses. In 2005, the award for postdoctoral salary rose to $65 000 per year. The selection of postdoctoral trainees followed the same model used to choose Ruth L. Kirschstein National Research Service Award (NRSA) recipients: namely, a written application initially reviewed by 2–3 committee members and later considered by the full selection panel. In addition to weighing innovation, significance, and research strategy of a given proposal, BWF/ASTMH committee members were also charged with assessing (1) whether a specific project was likely to advance an applicant’s personal scientific goals; and (2) whether the applicant’s home institution was strongly committed to the candidate’s future academic career. Although the BWF and ASTMH organizations both support the principle that fellows should be closely supervised by senior scientists, once selected, BWF/ASTMH fellows have more flexibility than their National Institutes of Health (NIH) counterparts and, should their circumstances change, can replace mentors as well as extend, defer, or reallocate their fellowship stipends and funds.

Previous evaluations of postdoctoral training programs have focused on narrower outcomes such as number of research manuscripts published and related short-term outcomes [4–10]. To our knowledge, little research to date [11] has examined career progression as measured by NIH career development awards, independent research awards, or appointment to tenure-track faculty positions. As a result, the current evaluation used mixed methods to examine the value of the BWF/ASTMH training program with respect to career development and scientific independence among young professionals transitioning from fellowship to faculty jobs.

METHODS

Our metrics for evaluating trainees were based on recommendations advanced by the Education Evaluation Working Group of the Clinical and Translational Science Award Consortium [12]. As a result, the evaluation included both quantitative and qualitative data. Quantitative data were derived from ASTMH records and the Research Portfolio Online Reporting Tools (RePORT) system, an online database maintained by the NIH that includes detailed information about all NIH grants, including start dates. iCite, a tool to access information related to research publications (icite.od.nih.gov) was used to identify not just the total number of BWF/ASTMH fellows’ publications [13] but their mean relative citations ratio (RCR), a new measure developed at the NIH which uses citation rates to estimate influence at the article level by comparing citation rates within the same field and timeframe [14]. The RCR is reported as cites per year of each article, normalized to citations per year received by NIH-funded articles in the same field and year.

RePORT was searched on 15 July 2016 to determine if trainees had received a career development award (K series) or an independent research award (R or U series) and the start date, when relevant. iCite was searched on 26 October 2016 to retrieve publication data. Both sets of data were integrated with an ASTMH-maintained Excel database listing trainee names, award year(s), awardees’ institutions when receiving awards, and current positions. As one of the stipulations of the BWF/ASTMH program, the last of these items—current position—is voluntarily reported to the ASTMH on an annual basis for active fellowships, and every 5 years for completed fellowships. Descriptive statistics (frequency, range, standard deviation [SD]) were calculated in Excel.

Qualitative data were collected from ASTMH records. As part of the annual reporting requirements, trainees were asked to describe how the BWF/ASTMH award influenced their career progression. A list of all quotes was reviewed and coded using grounded theory; preliminary codes were reviewed by 3 coauthors.

Finally, we examined the costs and return on investment (ROI) associated with supporting trainees by first retrieving data from their training program co-chairs, then confirming these amounts with the ASTMH manager who administers the BWF/ASTMH program. Return on investment was calculated by dividing total direct costs associated with NIH research grants in which former trainees later served as principal investigators (PIs) or co-PIs by total direct costs invested by the BWF/ASTMH fellowship. We then calculated both the overall ROI and the ROI specific to the subset of fellows with the longest period of follow-up, namely the 2001–2004 cohort.

RESULTS

All told, 41 individuals including 22 men and 19 women, all from the United States, received the BWF/ASTMH postdoctoral training award between 2001 and 2015 (Table 1). A total of 35 of 41 (85%) currently hold faculty positions as academic clinical researchers in the United States. Among those not in academic clinical research positions, 4 are clinicians, 1 is employed by a pharmaceutical company, and 1 works at a nonprofit organization focused on global health.

Table 1.

Brief Summary of Burroughs Wellcome Fund/American Society for Tropical Medicine and Hygiene Fellowship Outputs, 2001–2015 (n = 41)

PI on an NIH Career Development Award, No. (%)	21/35 (60)
PI or co-PI on independent NIH research award, No. (%)	12/26 (46)
Time to faculty position, y, mean ± SD	8.1 ± 3.2
No. of publications, mean (range)	36 (2–151)
ROI overall	11.9

Abbreviations: NIH, National Institutes of Health; PI, principal investigator; ROI, return on investment; SD, standard deviation..

Quantitative Findings

BWF/ASTMH fellows were successful in obtaining career development awards and independent research awards. Excluding 6 fellows who were still in midtraining at the time of this writing, 21 of 35 (60%) received a career development award following a mean of 3.6 years (SD, 2.4 years) from the start of their postdoctoral award. Of the 26 who were ≥5 years postfellowship, 12 (46%) had secured independent research support (U or R series awards) within, on average, 8.1 years of their initial postdoctoral award (SD, 3.2 years).

BWF/ASTMH fellows have produced a substantial number of publications. Among the 35 BWF/ASTMH awardees who were ≥3 years postfellowship, the mean output was 36.4 (range, 2–151; Figure 1). The mean RCR among all trainees was 1.86 (range, 0.62–3.69).

Figure 1.

Total publications (top) and mean relative citation ratios (RCRs; bottom) authored by Burroughs Wellcome Fund/American Society for Tropical Medicine and Hygiene postdoctoral trainees, 2001–2013 (n = 35).

BWF/ASTMH fellows have also demonstrated high rates of success in securing faculty positions at academic medical centers. Among all BWF/ASTMH recipients who completed their fellowships, 28 of 35 (80%) were hired as faculty at the following academic medical centers: Baylor College of Medicine; Boston University School of Medicine; Duke School of Medicine; Johns Hopkins School of Medicine; Harvard Medical School (9 individuals); Stanford School of Medicine; St Louis University; University of California, Los Angeles; University of California, San Diego; Yale University; University of California, San Francisco (2 individuals); University of Maryland (2 individuals); University of North Carolina at Chapel Hill (3 individuals); University of Virginia (2 individuals); University of Washington; and Weill Cornell Medical College.

Between 2001 and 2015, the BWF/ASTMH postdoctoral fellowship distributed $3 933 750 in direct research costs. All BWF/ASTMH postdoctoral fellowship committee members serve on a voluntary basis. The roughly $12 000 received each year for administrative support at ASTMH was routinely donated back to support trainees. To date, former fellows have been named as PIs or co-PIs on NIH grants totaling $49 945 612. As a result, the overall ROI for the BWF/ASTMH fellowship is 11.9 and the ROI for 2001–2004 award recipients, the cohort with the longest follow-up, is 31.8.

Qualitative Findings

BWF/ASTMH trainees identified a range of benefits stemming from this postdoctoral fellowship. The most common benefits included the following: providing leverage for a tenure-track faculty position at the home institution (24 trainees); providing support for preliminary data collection later leading to a career development award or other research grant (20 trainees); facilitating and deepening collaborative relationships between home universities and host institutions in LMICs (6 trainees). One postdoctoral fellowship recipient noted:

I am tremendously grateful for the flexible investment and support of the ASTMH and the Burroughs Wellcome Fund. I view this award (which preceded my K23) as one of the pivotal funding sources that made it possible to launch my research career and now transition to independence.

A partial list of non-NIH grants obtained following the fellowship included the following: Bill & Melinda Gates Foundation Grand Challenges Award (3 awards); Charles A. King Trust Postdoctoral Research Fellowship; Doris Duke Charitable Foundation; Thrasher Researcher Fund Early Career Award (8 awards); Howard Hughes Medical Institute Award; Novartis Foundation; World Health Organization; Fogarty Global Health Fellowship (2 awards); Center for AIDS Research Developmental Award (2 awards); Harvard Institute for Global Health (2 awards); and Canadian Sub-Saharan African HIV/AIDS Network. Other themes included the opportunity to become involved in tropical medicine as a career (3 trainees), the advantage of spending substantial time overseas (2 trainees), and experiences that spurred new research projects (2 trainees). The capacity of the fellowship to spend time overseas was emphasized:

These trips not only allowed me to collect samples, but have been irreplaceable opportunities to build and strengthen collaborations with investigators at South American and US institutions. Several new projects have already grown out of these collaborations that will strengthen my research portfolio.

Additional quotes from fellowship recipients highlight these themes (Supplementary Table 1).

DISCUSSION

In this article, we describe a postdoctoral fellowship program originally established by a private foundation and a professional society with the express purpose of strengthening the pipeline for physicians seeking academic careers in tropical medicine and global health. By providing a bridge between the final year of mentored training and the first faculty year, the fellowship has helped to launch independent research careers. The program has also enabled its trainees to spend extended periods of time at overseas sites in LMICs, thus strengthening relationships and enhancing the foundation for a career in global health. The study expands the limited literature on postdoctoral training by calculating ROI, by measuring faculty positions attained and articles published, and by using new NIH citation metrics.

Our data show that, among the subset of BWF/ASTMH fellows with at least 5 years of postfellowship follow-up, half achieved independent research funding. This finding is consistent with funding outcomes among NRSA fellows [11] and exceeds funding outcomes among graduates of the NIH Clinical Research Training Program [15]. Our trainees’ rate of obtaining independent research awards was also higher than that of the Fogarty International Clinical Research Scholars and Fellows program. Among the cohort enrolled between 2004 and 2011, 32% (30/94) later received at least 1 NIH grant following the fellowship. However, it should also be noted that none of the comparator programs is identical to the BWF/ASTMH fellowship. Our qualitative data suggest that trainees used the postdoctoral fellowship to collect preliminary data to support subsequent NIH career development awards. We also speculate that the relative success of BWF/ASTMH postdoctoral fellows in obtaining independent research funding may be related to the program’s ability to identify exceptional trainees, to ensure high-quality mentorship, and to offer a degree of flexibility not typically found in other awards.

Thirty-six was the mean number of publications among BWF/ASTMH trainees for whom at least 3 years of follow-up was available. This productivity measure exceeds similar metrics for NRSA fellows [11], NIH Clinical Research Trainees [15], Canadian postdoctoral fellows [16], and Fogarty International Clinical Research Scholars and Fellows [6]. The mean RCR value of 1.86 correlates with the upper 85th percentile of all publications and the upper 75th percentile of all NIH-funded publications [13]. Our trainees’ strong record of publication may have been related to their working in LMICs with a greater burden of disease, to the protected research time that their BWF/ASTMH awards afforded them, and/or to the recruitment of individuals with an already demonstrated potential for publication.

Our data reveal a substantial ROI. We calculated an overall ROI of 11.9. This exceeded ROIs reported by postdoctoral training programs in Germany (7.2 for the period 1998–2011) [17] and the United States (5.9 over the period 2000–2010) [18]. Neither of these 2 comparator programs focused on global health or infectious diseases postdoctoral training. Fogarty training programs with a greater focus on infectious diseases did not report ROI within their evaluation [6]. Given that our ROI estimates were based solely on NIH grants in which former fellows served as PIs or co-PIs, the data likely underestimate our fellows’ total research grant output. Infectious diseases fellowship support may be particularly valuable because of lower costs associated with research undertaken in LMICs compared with high-income countries. The ROI provides a compelling justification for continuing the training program.

Finally, qualitative data elicited from BWF/ASTMH fellows highlight several advantages of a fellowship focused on health issues unique to LMICs. Among them are the opportunity to forge deep international, institutional partnerships, to conduct clinical research on diseases and conditions that no longer plague less-endemic, high-income countries, and to develop a career in tropical medicine and global health. Other than the Fogarty International Clinical Research Scholars and Fellows program [6] and the Fogarty International Research Scientist Development Award, few similar grants exist.

Our evaluation of the BWF/ASTMH postdoctoral training program has several limitations. First, there are no comparator programs identical in scope, focus, and geography. On the other hand, by using several standardized evaluation metrics developed by the Education Evaluation Working Group of the Clinical and Translational Science Award Consortium, we believe our findings to be relevant to a range of programs and settings. Second, the BWF/ASTMH fellowship has thus far graduated a relatively small number of fellows (n = 41), which limits its power for comparisons and in-depth analysis. Third, the shorter period of follow-up for recent cohorts of trainees may also obscure trends. Fourth, we did not capture metrics about trainees directly improving the health of individuals and communities. Finally, there may have been selection bias after 2006 when candidates were selected based upon the expectation of receiving a faculty appointment during the second year. However, publication and faculty appointment metrics prior to 2006 were also exceptional.

We conclude that a global health training program with a clear, targeted focus on clinically relevant research can help its participants achieve scientific independence as measured by publications, grants, and faculty positions. Because our program has helped its graduates to bridge to infectious diseases faculty positions, this study’s findings may also hold lessons for reversing the diminishing pool of applicants for all infectious diseases fellowships in the United States [19]. Today, more than ever, sustained programs that ensure career development and scientific independence in infectious diseases are urgently needed.

Supplementary Data

Supplementary materials are available at Clinical Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Notes

Acknowledgments. We thank Jean Kramarik for administrative assistance. We are also grateful for the energy, passion, and productivity of the trainees who have made this program so successful.

Financial support. The postdoctoral fellowship is supported by the Burroughs Wellcome Fund and the American Society for Tropical Medicine and Hygiene.

Potential conflicts of interest. All authors: No reported conflicts of interest. All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.