Effectiveness of iterative interventions to increase research productivity in one residency program

Abstract

Background

The Accreditation Council for Graduate Medical Education requires residency programs to expose residents to research opportunities.

Objective

The purpose of this study was to assess the impact of a series of iterative interventions to increase scholarly activity in one internal medicine residency.

Methods

Retrospective analysis of the effectiveness of a series of interventions to increase resident and faculty scholarly productivity over a 14-year period was performed using quality improvement methodology. Outcomes measured were accepted regional and national abstracts and PubMed indexed manuscripts of residents and faculty.

Results

Initially, regional meeting abstracts increased and then were supplanted by national meeting abstracts. Sustained gains in manuscript productivity occurred in the eighth year of interventions, increasing from a baseline of 0.01 publications/FTE/year to 1.57 publications/FTE/year in the final year measured. Run chart analysis indicated special cause variation associated with the interventions performed.

Conclusions

Programs attempting to stimulate research production among faculty and residents can choose among many interventions cited in the literature. Since success of any group of interventions is likely additive and may take years to show benefit, measuring outcomes using quality improvement methodology may be an effective way to determine success.

In 1994, the Accreditation Council for Graduate Medical Education began requiring residency training programs to ensure that residents gain experience in research and demonstrate participation in a culture of scholarly inquiry. This requirement evolved into the Practice-Based Learning and Improvement competency in the new outcomes-based educational models in the United States (1). Cited benefits to exposing residents to research experience include increased satisfaction with residency training (2, 3), improved resident analytical skills and lifelong learning habits (4, 5), better patient care (5), increased likelihood of pursuing a career in academics (6–8), increased likelihood of becoming a clinician investigator (9), and as an asset to fellowship candidacy (10). However, significant barriers to resident research have been described, including a lack of resident and faculty time to perform research (11–14), absence of a research curriculum (14–16), availability of funding (14, 17), and availability of mentors (14, 18). Independent academic medical centers note more difficulty exposing residents to research (17, 18), have fewer experienced research faculty (19), and are more likely to be cited for a lack of research by residency review committees (17, 18).

Various multi-faceted interventions have been attempted to improve research productivity, including requiring resident research (13, 17, 20–26), granting protected time (11, 13, 17, 20–23), providing biostatistical and research support personnel (11, 13, 24, 27), appointing a residency research director (RRD) (11, 13, 17, 20, 25, 27–29), assigning mentors (17, 20–25, 27), and offering incentives such as presentation opportunities, awards (13, 24), and funding (17, 20, 27). Financial incentive plans, including performance-based (30–34) as well as salary-at-risk (30, 33, 35) formulations have also been implemented. Most interventions were used in combination, and the effects of any single intervention on specific outcomes across the literature have not been reviewed.

The purpose of this study was to assess the impact of a series of iterative interventions to increase scholarly activity as measured by accepted peer-reviewed abstracts and PubMed indexed manuscripts.

Methods

Setting and participants

This study was performed at a university-affiliated, community-based internal medicine residency program in the northeastern United States over a period of 14 academic years. In academic year 2001–02, the program employed 6 faculty and 27 residents, growing over the subsequent 13 years to 11 faculty and 41 residents by academic year 2013–14. Following a citation by the RRC-IM in 2001 for lack of resident exposure to research, the residency enacted a series of measures to improve resident scholarly activity. The effects of these interventions were studied by retrospective review of peer-reviewed abstracts and PubMed-indexed publications using continuous quality improvement methodology from 2001 to 2015.

Interventions

Descriptions of interventions, reasons for interventions, and the timeline are included in Table 1. The first intervention was identifying a RRD from the full-time faculty. Further interventions were selected based on needs assessments generated from faculty and residents on ACGME surveys, as well post-graduate surveys. Those interventions included formal curricular development, redesign of journal club, and seminars on education topics (vignette writing, evidence-based medicine, and systematic reviews), defining protected time for electives (up to 3 months over 2 years, based on progress from previous work), hiring of a biostatistician, defining research requirements and prioritizing these requirements during mentor meetings, celebrating resident successes with displays of successful work, and with pay-for-performance bonuses for faculty and residents. The scholarly activity component of the faculty incentive plan, in which up to 5% of a faculty member's base salary would be available as a bonus, was based on a point system developed by the internal medicine faculty. The system assigned points for poster presentations at local, regional, or national levels, as well as for publications (based on journal impact factor). The number of points assigned to each type of academic production and the number of points needed to meet varying levels of bonus targets were negotiated each year with the hospital administration. In addition, a pay-for-performance bonus was added to disburse additional funds into the resident's discretionary education fund. Residents had previously received $1,500 to use at their discretion for career-related educational or professional needs (e.g., stethoscopes, board review materials). In academic year 2012–03, in addition to these funds, the residents were awarded an additional taxable $100 bonus per regional or national abstract they authored and $300 bonus per publication they authored. The maximum available bonus in the first year of implementation was $600. The following academic year, this maximum was increased to $1,000.

Table 1.

Timeline of interventions in the research culture development at Reading Health System

Year	Initiative	Description	Rationale
2002–03	Named residency research director	Chose director from faculty without additional protected salary or time; served as mentor and evaluator for projects and elective experience	Coordinate and centralize research
2003–04	Redesigned journal club	Focused on study design and critical appraisal, rather than on study outcomes	Introduce/reinforce skills
	Created research ‘Wall of Fame’	Framed copies of research posters and first pages of publications displayed on wall of departmental conference room	Celebrate successes
2004–05	Mandated resident scholarly activity	Developed ‘point system’ for scholarly activity for all residents and determined minimum point requirement for graduation	Raise expectations
2005–06	Implemented faculty incentive plan that included research production	Scholarly activity bonus initiated for full-time faculty worth approximately 5% of base salary	Counterbalance clinical productivity incentives
	Clinical research noon conference series	Three 1-hour sessions annually covering basics of evidence-based medicine and literature search skills	Reinforce research skills
2006–07	Hired statistician	Full-time biostatistician hired by institution and shared across departments	Added expertise
	Formal research curriculum with associated research elective	Curriculum written by residency research director for resident research elective time	Provide protected time and mentorship
	Created mentoring guidelines that included formal review of resident efforts using structured portfolio	In fall of second year, emphasis of residency mentor discussions was re-focused to resident research efforts recorded in personal development portfolio	Reinforce expectations
2010–11	‘How to write a clinical vignette’ seminar	One-hour seminar with focus on choosing topic and writing with clarity; residents in teams all write abstract on same vignette with top rated abstract awarded rights to submit case	Expand research repertoire
2012–13	Implemented resident incentive plan	Pay-for-performance bonus using residency discretional funds; $100 bonus per regional or national abstract and $300 bonus per publication (maximum: $600)	Re-balance resident priorities
2013–14	Increase in resident incentive plan	Increased maximum resident bonus to $1,000	Reward productive residents
	Resident-initiated ‘How to do a systemic review’ seminar series	Seven 1-hour seminar sessions during which teams developed, researched, and wrote a systematic review and meta-analysis over the course of 14 weeks	Expand research repertoire

Data analysis

The primary outcome measures for scholarly activity output were accepted peer-reviewed abstracts and PubMed-indexed manuscripts. The unit of analysis was scholarly output per full-time equivalent (FTE) per academic year. One FTE was assigned for all residents in the program and staff, but was prorated for part-time staff and staff that left during an academic year. Scholarly activity outcomes were determined by review of resident files, as well as individual searches of Google Scholar, EMBASE, and PubMed for each author by name. All abstracts and publications were reviewed, and duplicates were deleted. Abstracts were characterized as ‘regional’ or ‘national’ based on the meeting description. Scholarly activity was recorded in an Excel spreadsheet (Microsoft Corporation, Redmond, WA) by a trained research associate. Ten percent of entries were double-coded by one investigator to ensure accuracy of the database. Calculations of publications per FTE faculty and resident were performed within Excel. Run charts were then created using the Institute for Healthcare Improvement (IHI) run chart tool (IHI, Cambridge, MA). The center line was created using the mean of the PubMed indexed publications per FTE in the years leading up to the first intervention. In order to compare our results, we reviewed the literature for scholarly activity interventions in the literature, categorizing their specific interventions as well as their publications, measured by reported publications per physician per year averaged over the length of the study. Publication counts were confirmed by direct communication with corresponding authors when necessary. The Reading Health System Institutional Review Board exempted this study as quality improvement.

Results

The program had 5.9 FTE faculty members and 27 residents for a total of 32.9 FTE in the first year of measurement (2001–02) and grew to 10.55 FTE faculty members and 41 residents for a total of 51.55 FTE by the last year of the study (2014–15). There was an initial increase in regional meeting abstracts in the academic year 2003–04, which was surpassed by national meeting abstracts in 2010–11 but declined thereafter. Sustained gains in manuscript productivity occurred in the eighth year of interventions, increasing from a baseline of 0.01 publications/FTE/year to 1.57 publications/FTE/year in the final year measured (Fig. 1). In academic year 2012–13, 27 out of 32 residents qualified for a scholarly activity bonus and received a total of $9,200. The following year, 31 out of 39 residents qualified and received a total of $16,900. In the final year measured, 30 out of 41 residents qualified and received a total of $21,800.

Fig. 1.

Research output of the residency program.

Our run chart of publications per FTE demonstrated three total runs. A run chart with 14 data points should have between 4 and 11 runs, indicating too few runs which we interpreted as an indication of special cause variation in the data set (36). In addition, both a shift (12 points above centerline, starting in 2003–04) and a trend (six consecutively increasing points from 2009 to 2010 and onward) indicated special cause variation in our data (Fig. 2).

Fig. 2.

PubMed indexed publications per academic year per FTE (faculty and resident data).

Discussion

In this study, we found that scholarly activity significantly increased over the past 12 years of our outcomes measures, indicating special cause variation (i.e., statistically unlikely to be the result of random fluctuation or chance). We interpret this finding as indicating a positive association between our interventions and research productivity. Due to the time difference between interventions relative to the time cycle of a typical manuscript from idea inception to publication, we could not determine the individual impact of any single one of our interventions. In addition, the effects of any single intervention would be expected to be additive on prior interventions, making it more difficult to determine the relative effect of any single intervention. Given that the order of interventions was chosen based on local needs as determined by the RRD, the effect of changing that order on scholarly activity cannot be determined. However, given the above limitations, we concluded that the measurement of the impact of scholarly activity programs using quality improvement methods allowed us to definitively determine our overall program's success. Follow-up ACGME surveys of current and graduating residents (in 2013, 2014, and 2015) and at a site visit (in 2010) no longer cited research exposure as a program deficiency.

Similar to other interventions in the literature to increase scholarly activity, our methods involved multimodal programmatic and financial interventions (Table 2). Although other studies confined their efforts to either faculty or resident groups, we chose interventions intended to influence both residents and faculty. Similar to other studies, we retrospectively studied the effects of interventions at a single site studied over a prolonged period of time. Most authors chose outcomes measures that included abstracts and publications or publications only, whereas others measured grant funding received, making direct comparisons between studies difficult. No single intervention appears to be uniformly successful, and no specific pattern of multimodal interventions appears to be more effective than another in our review of the literature, suggesting that the optimal solutions at any one facility may be unique to the barriers at that facility. This suggests that a formal needs assessments and rigorous measurements of outcomes may best guide future individual interventions. Pay-for-performance models have existed in the business literature for approximately 100 years but are more recent additions to the American medical culture (37, 38). These models have increased clinical productivity (defined as volume) and ‘time on task’ (37, 39). However, studies of isolated financial incentives directed towards medical education outcomes, including research, have shown conflicting results (30, 31, 40, 41). How large an incentive is needed relative to the other components of compensation to effectively stimulate research is also currently unknown (31, 41–44).

Table 2.

Interventions performed to increase scholarly activity in the literature

		Time studied (years)	Protected time	Research requirement	Mentors	Curriculum	Research assistant	Research director	Biostatistician	IT support	Research fund available	Opportunities or awards	Funding: performance based	Funding: salary at risk	Specifics of interventions	Reported publication outcomes
Interventions on faculty
Bertram et al. (27)	GIM faculty (n=13–39/year)	16			x		x	x	x		x				Director: 10–20% salary support; co-director: 5–10% support	334 publications/16 years (0.83 pubs/fac/years)
Cramer et al. (35)	Fam med faculty (n=38–49/year)	3												x	RVU-based incentive plan (2% of salary at risk)	Research points increased from 524 to 775 (48% increase)
Filler et al. (40)	Staff Pediatricians (n=32) and administrators (n=5)	3											x		Productivity bonus (approximately 10%) for excellence in research, practice, education, administration	No difference in research scores; publications not reported
Reich et al. (33)	Anesthesia faculty (n=?)	1												x	Productivity-based incentive; 70% of salary at risk	No change in publications
Sakai et al. (30)	Clinical faculty (n=90–145/year)	6												x	Performance-based incentive, 30% of salary at risk	161 publications/8 years (1.13 pubs/fac/years)
Schweitzer et al. (32)	Medical school faculty (n=?)	10			x		x		x	x	x	x	x		Productivity-based incentive tied to tenure	Incr in funding $20–$90 M; publications not reported
Tarquinio et al. (41)	Physicians in 12 clinical divisions	2											x		Financial incentives	Incr growth of research per scientist growth from 9%/year to 23%/year
Interventions on residents in training
Byrnes 2005 et al. (22)	IM residents (n=72/year)	3	x		x	x		x							4 months approved elective time with mentor	Research from 6% to 29%; pubs not reported
Carek et al. (24)	Fam med residents (n=20/year)	10	x	x	x	x			x						Required curriculum for senior residents; protected time	15 publications/1 year (0.05 pubs/res/years)
Chang and Mills (31)	ENT residents (n=10.5/year)	8											x		Productivity-based incentive, for distribution of dept. discretionary funds	41 publications/14 years (0.29 pubs/res/year)
Durning et al. (28)	IM residents (n=30/year)	5		x	x	x		x							Residency research director spent 7 hours/week on projects	17 publications/5 years (0.11 pubs/res/year)
Fancher et al. (25)	IM residents (n=87/year)	4		x	x	x							x		4-week required course; funding to present if accepted	2 publications/1 year (0.02 pubs/res/year)
Fischer and Cation (29)	IM residents (n=24/year)	6		x	x	x		x					x		RRD, elective time, mandatory requirement	No publications
Hepburn et al. (23)	IM residents (n=30/year)	5	x	x	x	x		x		x		x			Mandatory res requirement, 2 months dedicated time	21 publications/5 years (0.14 pubs/res/year)
Holmes et al. (26)	EM residents (n=24/year)	10		x											Required research	36 publications/10 years (0.15 pub/res/year)
Kanna et al. (13)	IM residents (n=84/year)	2	x	x	x	x		x		x		x			2-week required rotation; assigned mentor; awards day	49 publications/2 years (0.29 pubs/res/year)
Roane et al. (20)	Psych residents (n=48/year)	5		x	x	x			x						Required research, assigned mentors	32 publications/5 years (0.13 pubs/res/year)
Rothberg et al. (11)	IM residents (n=54/year)	6	x	x	x	x	x	x	x			x			RRD with 0.25 FTE protected Time; biostats and research assist support	58 publications/7 years (0.15 pubs/res/year)
Vinci et al. (21)	Peds residents (n=126/year)	5	x		x	x		x					x		Elective 3-month rotation, assigned mentors, 25 hours mandatory curriculum	15 publications/5 years (0.02 pubs/res/year)
Interventions in both faculty and residents
Alweis 2015	IM residents, faculty (n=33–51/year)	14	x	x	x	x		x	x		x	x	x		Research electives, incentive plan for faculty and then residents	176 resident publications/14 years (0.44 pubs/res/year); 21 faculty publications/14 years (0.20 pubs/fac/year)

There are several potential limitations to this study. While the number of potential venues for all publications has greatly increased over the time of this study, we limited our outcome measure to only those that were indexed by PubMed to limit the effects that newer open access journals may have had on our results. This may have given us a more conservative estimate of our overall effectiveness than if we had captured all peer-reviewed publications (as all others had done with one exception) (31), but prevented us from potentially overstating the effects of our intervention. The improvement in research productivity as attributed to our interventions is potentially confounded by the growth of the residency faculty and more competitive resident recruitment over the course of the study, although it should be noted that none of the faculty recruited had research backgrounds or protected research time.

Conclusions

Programs attempting to stimulate research production among faculty and residents can choose among many interventions cited in the literature. Since success of any group of interventions may be additive and take years to effect a measurable increase in the outcomes of interest, measuring outcomes using quality improvement methodology may be an effective way to determine success. Whether these efforts lead to future resident research production in fellowship or practice is a matter for further research. The best methodologies to sustain gains in research productivity in the face of rapid turnover of the majority of the participants (i.e., residents) deserve further inquiry.

Conflict of interest and funding

The authors have not received any funding or benefits from industry or elsewhere to conduct this study.