Eleven-Year Outcomes from an Integrated Residency Program to Train Research Psychiatrists

Abstract

In 2000, to address the critical shortage of psychiatrist researchers, faculty in the Residency Training Program in General Adult Psychiatry at the University of California, San Francisco, School of Medicine developed and implemented a research resident training program (RRTP). In this article, the authors describe the program's development process, including its organizational structure, eligibility criteria for residents, and core program elements, and they report 11 years of outcomes data. Notable RRTP components include: research and career mentorship, individualized training plans, the integration of clinical and research experiences, protected research time, and research funding. From 2000-2011, the RRTP enrolled 48 residents. The authors’ primary outcome of interest in determining the success of the program was whether or not each RRTP resident entered a postdoctoral research fellowship after graduation. The authors found that more than 80% of graduates had matriculated to postdoctoral research fellowships, irrespective of their previous doctoral-level training in the basic or social sciences. The authors conclude that this flexible, individualized, and innovative training program for psychiatry residents was successful in facilitating the entry of participants into primary research careers, reasoning that it may serve as a model for other residency programs with similar goals. More widespread adoption of similar educational models may help to address the critical shortage of psychiatrist researchers.

Many agree that the inadequate numbers of physician scientists are the result of structural failures in the development pipeline,¹ a need that is particularly acute in the field of psychiatry. The Institute of Medicine described this dearth of psychiatrist researchers in a 2003 report.² Nationwide, the number of psychiatrists pursuing postdoctoral research fellowships after residency training declined by nearly 40% between 1992 and 2001,³ which suggests that the current pipeline will not be able to meet future population mental health challenges.

Post-residency programs designed to facilitate research careers have been extensively studied and evaluated in a number of specialties,^4-7 including psychiatry.^8-12 However, delaying such interventions until after residency or clinical fellowship training may be too late to avert the attrition of young investigators from pursuing research careers.^13,14 The gradually shrinking ranks of “late bloomers” (i.e., physicians with little or no prior research experience whose research interests develop after they have begun residency training) also have been described in detail.^15,16 Integrating research training directly into residency programs^17-19 then may be particularly well suited to overcoming these existing barriers.²⁰ However, the obstacles to conducting research during residency training are great,^21-23 and the literature offers little guidance on how to overcome these obstacles.^17,24

In 2000, to address the critical shortage of psychiatrist researchers, faculty in the Residency Training Program in General Adult Psychiatry at the University of California, San Francisco (UCSF) School of Medicine developed and implemented a research resident training program (RRTP). The purpose of this article is to describe the development process and report 11 years of outcomes data from the program.

The UCSF Research Resident Training Program

A National Institutes of Health (NIH) Research Education Grant (R25 MH-060482, Training the Next Generation of Mental Health Researchers) funds the RRTP. Such grants are meant to catalyze the research careers of psychiatry residents with prior experience in research and provide foundational training for psychiatry residents with little or no prior experience in research. The first three funding cycles of the RRTP began on July 1 of 2000, 2005, and 2011. The program's long-term goal is to encourage psychiatry residents to pursue a primary career in research, defined as an academic position with at least 50% percent effort devoted to research.

Below we describe the RRTP's organizational structure, eligibility criteria, and core program elements, which are based on earlier work that identified key factors related to the quality of psychiatric research training.²⁵

Organizational structure

Executive committee

Until 2010, oversight for the RRTP was provided by the sole principal investigator (PI) of the R25 Research Education Grant (V.I.R.). Beginning in the third funding cycle, program oversight was distributed more broadly and is now provided jointly by an executive committee, which includes the contact PI (C.A.M.) and the co-PI (V.I.R.) and a third faculty member who serves as the RRTP liaison to the UCSF Residency Training Program in General Adult Psychiatry (henceforth, we will refer to this program as the residency program). The executive committee meets on a quarterly basis to discuss strategies for recruitment, evaluate research residents’ proposals, assign career and research mentors, and assess the program's progress. The contact PI also assumes overall responsibility for career mentorship matching, day-to-day management of individual residents’ research projects, fiscal and administrative responsibilities for the RRTP as a whole, and maintaining regular communication with the National Institute of Mental Health (NIMH).

Institutional support

The R25 grant funding has been sufficient to provide the equivalent of 1.5 full-time equivalent (FTE) resident stipends (increased to 2.0 FTE in the RRTP's third funding cycle) for protected research time in each year of funding. The RRTP leverages this funding by drawing on institutional support from departmental leadership, which involves both making financial commitments as well as granting the RRTP a substantive voice in the recruitment and selection of promising research-oriented medical students through the National Resident Matching Program (NRMP). As a result of this joint funding scheme, the RRTP has had the capacity to admit all psychiatry residents who express a serious interest in pursuing psychiatric research careers. Without this institutional commitment, the primary limitation to accepting residents into the RRTP would have been financial. Resource constraints would have required the RRTP to create an internally competitive environment in which residents with a serious interest in research but little previous experience would have been at a competitive disadvantage.

Eligibility, recruitment, and selection

All residents are eligible to participate in the RRTP. The majority of participants are identified and recruited at the time of their psychiatry residency application through the NRMP. Prior to their interview day, all UCSF psychiatry residency applicants are asked to indicate and rank their areas of interest within the field of psychiatry. These include, among others, child psychiatry, community psychiatry, geriatric psychiatry, basic science research, clinical/translational research, women's mental health, psychiatric education, and global mental health. The residency program offers several targeted research interview days so candidates can interview with specific members of the research faculty who may ultimately serve as their research mentors. The residency program also reserves up to two of the 16 NRMP openings for candidates who demonstrate exceptional research promise.

In addition to these recruitment strategies, residents who are interested in the RRTP may self-identify at any time during residency. Regardless of when an application is initiated, all RRTP applicants must submit a formal proposal to the executive committee, which includes a research proposal with hypotheses and methods, proposed mentors, and an individualized training plan. If a resident communicates a serious interest in research but submits an inadequately developed proposal, the applicant is not accepted into the program, is redirected to work with his or her research mentors and the executive committee to improve the proposal, and is encouraged to re-apply.

Core program elements

The RRTP differs from the general psychiatry residency track in that it includes individualized research and career mentorship, focused networking opportunities, and funding and protected time for research. RRTP residents are required to meet the Accreditation Council for Graduate Medical Education requirements to graduate from a general psychiatry residency program, including 48 months of training and 12 months of FTE outpatient psychiatry. As with residents in the general residency track, RRTP residents complete six months of FTE outpatient psychiatry during post-graduate year (PGY)-2. During PGY-3, residents in the general track complete 12 months of outpatient psychiatry at 80-100% time, whereas RRTP residents complete 12 months of outpatient psychiatry at 70% time. During PGY-4, residents in the general track complete clinical electives at 90% time, whereas RRTP residents engage in research at up to 90% time (and both general track and RRTP residents have a 10% time longitudinal experience). RRTP residents are not eligible to serve as chief residents because the administrative time demanded of the chief positions is substantial and would negatively affect their clinical and research training experiences.

Additional requirements for completing the RRTP include giving a presentation at least once a year at the RRTP works-in-progress dinner talks, giving a poster presentation at the annual UCSF Department of Psychiatry research retreat during PGY-3 and PGY-4, and giving a departmental grand rounds talk about research findings supported by the RRTP during PGY-4. RRTP residents also are expected to submit manuscripts for publication in peer-reviewed journals, apply for internal or external research and travel funds, and give presentations at external research conferences.

Below, we describe the RRTP's core program elements.

Career mentorship

The selection of appropriate mentors for a given trainee is generally thought to be a critically important decision that may affect the trainee's career choice and productivity.^26,27 Therefore, upon entry, all RRTP residents are assigned a career mentor who has an academic tenure-track position and a funded research program. The specific task of the career mentor is to help the RRTP resident maintain an active interest in research and navigate the developmental tasks in his or her chosen career path, rather than to provide specific research supervision.

Research mentorship

RRTP residents are encouraged to identify a research mentor, with or without the assistance of their career mentor, during PGY-1. While the executive committee must approve all RRTP residents’ choices of research mentors, they pay particular attention to those who are non-departmental faculty. This vetting process is to ensure that research mentors have sufficient research experience and infrastructure to provide appropriate training and mentorship, and also that mentor and mentee expectations are aligned.^28,29 If the executive committee does not approve a research mentor, they assist the resident in finding another mentor who is working in the resident's area of interest and who meets the mentorship requirements. In many cases, they also encourage the trainee to work with the originally proposed mentor as a “secondary” mentor rather than one who assumes primary responsibility for the trainee and his or her training plan.

Core competencies

RRTP residents are expected to fulfill core competencies in three areas: responsible conduct of research, research design and implementation, and scientific writing. RRTP residents typically achieve these competencies through courses offered in the UCSF Training in Clinical Research Program. The RRTP then supplements residents’ exposure to issues related to the responsible conduct of research through a one-day practicum involving lectures, case studies, and role-playing simulations. Residents without a prior research background are required to take an intensive, hands-on course in designing clinical research. The expected outcome of this course is a five-page research protocol, which forms the basis for the resident's formal application to the RRTP. RRTP residents also are strongly encouraged to take the “Building a Career in Clinical Research” course. Topics covered in this course include: choosing a mentor, time management, scientific paper writing, grant writing, and navigating academic promotion.

RRTP residents work at their own pace to meet the program's requirements, but the career mentor and the research mentor, in conjunction with the executive committee, are responsible for determining whether the resident has achieved the core competencies. These determinations are based on both formal and informal evaluations, such as faculty feedback from works-in-progress seminars and feedback forms returned from attendees at grand rounds and the annual department research retreat, and a review of manuscript drafts, if applicable. RRTP residents’ presentations are generally attended by research faculty, including the career mentors, and include an active discussion component with faculty and trainees. Residents who are not meeting the required benchmarks are asked to meet with their career mentors to identify the reasons for these developmental delays, which may include overly ambitious research projects, research mentors who do not have the necessary time or skills to provide appropriate mentorship, or personal factors (e.g., a resident's realization that she or he prefers clinical work to research). In each case, the resident creates a revised plan with the input of his or her career mentor and the executive committee. This revised plan can include a change in or the addition of research mentors who can provide more “hands-on” time with the resident, a reduction in the scale of the research project (e.g., changing the proposal from a primary data collection to a secondary analysis of previously collected data), or, in the case of those residents who learn during the RRTP that research is not a primary interest, a return to the general psychiatry residency track.

Networking opportunities

The RRTP has developed several mechanisms for helping residents connect with and become known by other investigators in their fields of interest, including dinner seminars, field-specific collaborative research groups, and an annual department research retreat. RRTP residents are strongly encouraged to participate in UCSF-wide initiatives, such as the UCSF Clinical and Translational Science Institute (CTSI) Resident Research Program, which provides resources on research career development for all UCSF residents. Beginning during the RRTP's third funding cycle, program faculty planned a biennial California-wide psychiatry research resident retreat, which convenes research residents and their mentors from across California. This research retreat is hosted by the RRTP and is funded by the R25 Research Education Grant.

Protected research time

Protected time for research and other activities is provided to all RRTP residents. During PGY-2, residents are provided with up to one half-day of protected research time per week for six months. Those without prior research experience typically use this time to build a foundation in several core competencies related to the conduct of research. Those with substantial prior research experience who have already finalized the details of their proposed research may begin working in their chosen laboratories or schedule an alternative didactic experience. At the end of PGY-2, the executive committee must approve RRTP residents’ five-page research protocols outlining their proposed projects and the approximate budgetary requirements. During PGY-3, RRTP residents have up to 30% protected research time to work towards achieving their proposed research aims. During PGY-4, residents have 75-90% protected research time and are expected to present their research findings at local and national venues (with travel support provided by the RRTP) and to begin the process of publishing their findings in peer-reviewed journals.

Funding for research

To facilitate RRTP residents’ ability to develop independent research projects within the context of their research mentor's larger research program, the RRTP originally provided pilot funding to all residents. During the third funding cycle, however, the program announcement for the R25 awards (PAR-10-267) contained new language specifically excluding pilot research studies. Prior to this change in language, these funds averaged $15,000 per resident over the training period (range: $5,000 to $20,000) and were allocated based on the detailed budget and budget justification that each RRTP resident submitted at the end of PGY-2. RRTP residents were expected to leverage the pilot data they collected using RRTP funds to compete for external funds from other UCSF-wide initiatives (such as the CTSI Resident Research Program) or national research awards. Now, these funds instead are used for research supplies and ancillary educational expenditures.

Child and adolescent psychiatry training

In response to the dearth of clinician-researchers in the field of child and adolescent psychiatry, ² the RRTP expanded its program during the second funding cycle to accommodate the interests of clinical fellows in the UCSF Child Psychiatry Training Program. PGY-5 and PGY-6 child and adolescent clinical fellows have high clinical demands, but the UCSF Division of Child and Adolescent Psychiatry has granted support for child and adolescent fellows participating in the RRTP to have up to 25% protected time per year to devote to ongoing research activities.

Individualized training plans

During the third funding cycle, training opportunities became more tailored to each resident's particular strengths, background, and career goals. This specific program component has been critically important to accommodating residents with little or no prior experience in research. Soon after beginning the RRTP, residents work with their research mentors to map their previous training experiences and identify specific gaps in their training. The purpose of this exercise is to create an individualized research and clinical training plan that can be modified and expanded over the course of training. For the template used for individualized training plans in the RRTP, see Supplemental Digital Appendix 1. This document guides the career and research mentors in assessing the training that is required for each resident based on his or her prior research experience and future career plans.

Integration of clinical and research experiences

During the third funding cycle, to allow for in-depth research training within the timeframe of a standard residency training program in general adult psychiatry, the RRTP residents’ clinical subspecialty rotations became more closely aligned with their individualized research and clinical training needs. Most of the subspecialty clinics at UCSF not only provide specialized tertiary care for a given disorder or patient population but also support active clinical or translational research protocols. Every effort is made to ensure that RRTP residents are given priority for these rotations.

Assessing the Success of the RRTP

Analysis

We used the RRTP records, updated routinely by the contact PI throughout the course of each resident's training, as our source of data. Our primary outcome of interest was whether or not each RRTP resident entered a postdoctoral research fellowship after graduation. We also extracted data from the publication records of each resident, identifying each journal article using the PubMed database and updating residents’ publication records through July 15, 2012. We identified NIH grants using the NIH Research Portfolio Online Reporting Tools database.

To summarize outcome data, we employed conventional means, medians, and proportions. To make statistical comparisons on covariates of interest between residents with vs. those without prior doctoral-level training in the basic or social sciences, we employed the nonparametric equality-of-medians test for continuous variables and the Pearson χ² test for categorical variables. The UCSF Human Research Protection Program (HRPP) reviewed our study and decided that it did not constitute human subjects research. Therefore, they considered it exempt from requiring approval or review.

Results

From 2000-2011, the RRTP enrolled between one and seven residents per year, for a total of 48 residents (mean of 4 residents per year; median of 3 residents per year). One resident transferred out of the residency program into a family medicine residency, one resident withdrew from the RRTP for personal reasons, and a third withdrew from the RRTP to pursue accelerated training in child and adolescent psychiatry. Of the remaining 45 residents, 24 were men (53.3%) and 21 women (46.7%). Eighteen (40.0%) entered the RRTP with a previously awarded Ph.D. degree.

RRTP residents pursued a broad variety of research topics. Basic science researchers engaged in various laboratory-based investigations, including 5HT2C/5HT2A receptor activity and glutamate receptor signaling. Most residents engaged in clinical/translational research, including the use of intranasal oxytocin to modify social cognition in schizophrenia and the measurement of cortical brain volume in chronic and resolved post-traumatic stress disorder. One notable theme that has emerged in the RRTP, especially in recent years, is the increasing interest among residents in pursuing research careers to address mental health needs in resource-limited settings. Residents have pursued clinical research topics in a diverse range of countries, including Chad, Colombia, Costa Rica, India, South Africa, and Uganda.

Of the 45 eligible residents, 17 (37.8%) published an article in a peer-reviewed journal either during their appointment in the RRTP or within two years of graduating. These 17 residents published 68 articles in total (median of 2 articles; interquartile range [IQR] of 1-4; range of 1-26). Of those, residents were first authors on 37 articles, including several in high-impact journals, such as the Archives of General Psychiatry and Molecular Psychiatry. Seven residents (15.6%) received the NIMH Outstanding Resident Award, one resident (2.2%) was named a Laughlin Fellow by the American College of Psychiatrists, and three residents (6.7%) obtained prestigious postdoctoral research fellowships from the Robert Wood Johnson Foundation. As of December 2012, 38 residents (84.4%) had completed their clinical training; of these, 34 (89.5%) have received board certification to practice psychiatry from the American Board of Psychiatry and Neurology.

Having prior research experience did not appear to exert a substantive influence on the outcomes we measured. Residents with a previously awarded Ph.D. degree published a mean of 2.6 articles (median of 0; IQR of 0-3; range of 0-26) in peer-reviewed journals either during their appointment in the RRTP or within two years of graduating, compared to a mean of 0.8 articles (median of 0; IQR of 0-1; range of 0-10) among those without Ph.D. degrees. A nonparametric equality-of-medians test showed no statistically significant difference between the two groups (χ² = 0.02; P = .90). Of the 38 graduates, 32 (84.2%) entered a postdoctoral research fellowship--12 of 15 graduates who entered the RRTP with a Ph.D., and 20 of 23 graduates who did not (80.0% vs. 87.0%; χ² = 0.33, P = .57).

The differences in residents’ productivity narrowed further after graduation. Graduates with a previously awarded Ph.D. degree published a mean of 2.8 articles (median of 2; IQR of 0-4; range of 0-10) after the post-graduation two-year window, compared to a mean of 2.9 articles (median of 2; IQR of 0-5; range of 0-8) among those without Ph.D. degrees (χ² = 0.00; P = 1.00). The articles published by graduates with Ph.D. degrees were cited 1,997 times over 75 person-years of post-RRTP follow-up (26.6 citations per person-year), whereas the articles published by graduates without Ph.D. degrees were cited 1,444 times over 116 person-years of post-RRTP follow-up (12.4 citations per person-year).

Of the 32 graduates who entered postdoctoral research fellowships, 7 (21.9%) are still engaged in research as postdoctoral fellows, and seven (21.9%) have successfully obtained K- or R01-level research awards. Two (6.3%) have submitted proposals for mentored career development awards now currently pending review, and eight (25.0%) have faculty appointments with an active research component.

Implications of the RRTP

In this article, we described a flexible, individualized, innovative, and successful training program for psychiatry residents, which was designed to facilitate their entry into primary research careers. Core program components include research and career mentorship, individualized training plans, the integration of clinical and research experiences, protected research time, and research funding. More than 80% of program graduates have matriculated to postdoctoral research fellowships, irrespective of their previous doctoral-level training in the basic or social sciences. Many aspects of this program can be replicated in other settings, while other aspects may require specific changes in institutional policies and programming before implementation is possible.

For other psychiatry residency programs seeking to replicate the RRTP, the combination of both research/career mentoring and individualized training plans may particularly benefit the career trajectories of late bloomers^15,16 who enter residency training with little or no prior research experience and whose research interests develop after they have begun residency training. We believe that these residents should receive no less encouragement to pursue primary research careers. The comparable outcomes data between these two groups supports this tenet of the program. Although RRTP late bloomers eventually caught up with their more experienced colleagues, they were initially slower to produce tangible results (e.g., peer-reviewed publications) of their research experience. Individually tailored mentorship, implemented on an ad hoc basis during the first two funding cycles and in a systematic fashion during the third funding cycle, explicitly recognizes that these individuals may have different training and mentorship needs compared to their colleagues with companion doctoral degrees in the basic or social sciences. In this regard, the contact PI (C.A.M.) brought a unique perspective to the program on the challenges faced by such residents seeking to initiate a research career--she herself had little research experience prior to fellowship but participated in an individually tailored mentorship during residency and since has successfully obtained funding from the NIH through the K23 and R01 mechanisms. Future efforts will focus on shifting more of the research training into the early years of the residency and on giving late bloomers more exposure to formal coursework in research methodology prior to their entry into a research group. Alternatively, if current trends continue, consideration should be given to new models of training, such as a combined/integrated Ph.D./psychiatry residency experience that shortens the traditional timeline to independence.

Consistent with our experience, other observers have commented on the need for psychiatry departments to provide incentives, or at least minimize the clear disincentives, for faculty to participate in mentoring.^1,12,30,31 While residents generally seek out RRTP faculty because of their interest in and dedication to mentoring young investigators, the provision of RRTP funding to support residents’ research has helped to lessen the “costs” (from the perspective of a faculty member) of mentoring them. The new funding restrictions on pilot funds have narrowed a novel and useful aspect of the RRTP. Yet the R25 funding still helps to supplement and extend the resources provided by the research mentor. Encouraging the development of strong ties between mentors and mentees may be particularly important during the initial stages of training, where both the clinical workload and attrition risk are high.²¹

Our interpretation of the outcomes data is subject to several limitations. First, some elements of the RRTP, such as the robust institutional support provided by the UCSF Department of Psychiatry, are institution-specific and may not successfully generalize to other settings. However, other elements--such as the mentorship, individualized training plans, and integration of clinical and research training--are potentially generalizable to other programs across the U.S. Second, we are unaware of any standardized benchmark data against which we can assess the comparative effectiveness of the RRTP,^17,24 which further underscores the problematic nationwide shortage of psychiatrist researchers and the need for continued development in this area.² Third, our method of collecting and analyzing outcomes data was unlikely to be 100% sensitive and could have resulted in under-counting of published journal articles or NIH grants. Fourth, right-censoring of the data limit our ability to draw conclusive inferences, as approximately one-fourth of RRTP residents had not yet graduated at the time of publication. Related to this limitation, our study adopted for its primary outcome what is arguably a surrogate--matriculation into a post-residency research fellowship--for a more distal outcome, such as achievement of independent investigator status (i.e., obtaining R01-level funding from the NIH), given that the RRTP has existed only for a relatively short period of time and because such an outcome depends on a host of other post-residency factors, including postdoctoral fellowship mentoring and institutional programmatic support at the critical time of transition.

In summary, the UCSF RRTP may serve as a model for other residency programs seeking to facilitate the entrance of their residents into primary research careers. More widespread adoption of similar educational models may help to address the critical shortage of psychiatrist researchers. Our outcomes data suggest that such models can be successful in achieving this goal and should be vigorously pursued by other residency programs.