Challenges to publishing pharmacy resident research projects from the perspectives of residency program directors and residents

Abstract

Objective

To identify barriers to completing and publishing pharmacy residency research projects from the perspective of program directors and former residents.

Methods

This was a cross-sectional survey of pharmacy residency program directors and former post-graduate year one and two residents. Directors of pharmacy residency programs whose residents present their projects at the Western States Conference (n=216) were invited to complete an online survey and asked to forward the survey to former residents of their program in 2009, 2010, or 2011. The survey focused on four broad areas: 1) demographic characteristics of the residency programs, directors, and residents; 2) perceived value of the research project; 3) perceived barriers with various stages of research; and 4) self-identified barriers to successful research project completion and publication.

Results

A total of 32 program directors and 98 residents completed the survey. The minority of programs offered formal residency research training. Both groups reported value in the research project as part of residency training. Significantly more directors reported obtaining institutional review board approval and working through the publication process as barriers to the research project (46.7% vs. 22.6% and 73.3% vs. 43.0%, respectively p<0.05) while residents were more likely to report collecting and analyzing the data as barriers (34.4% vs. 13.3% and 39.8% vs. 20.0%, respectively, p<0.05). Both groups self-identified time constraints and limitations in study design or quality of the study as barriers. However, while program directors also indicated lack of resident motivation (65.5%), residents reported lack of mentorship or program structural issues (43.3%).

Conclusions

Overall, while both groups found value in the residency research projects, there were barriers identified by both groups. The results of this study may provide areas of opportunity for improving the quality and publication rates of resident research projects.

Introduction

The pharmacy profession is moving towards producing highly skilled clinicians with advanced clinical training.1,2 Pharmacy residency programs play an important role in meeting this objective with the number of programs increasing in response to higher demand for more skilled clinicians. To receive accreditation by the American Society of Health-System Pharmacists (ASHP), pharmacy residency programs must provide opportunity for residents to “conduct a practice-related project using effective project management skills”.3,4 Projects can be original research, medication use evaluations (MUE), quality improvement projects, or problem solving exercises.3,4 Residents typically present their work at regional residency conferences and are often encouraged, although not typically required, to publish their findings in peer-reviewed journals.

Recent studies demonstrate low publication rates of pharmacy resident projects.5,6,7 Only 16% of resident projects presented at the Southeastern Residency Conference (SERC) were published.5 Similarly, publication rates of projects presented at the Western States Conference (WSC) have been estimated to be between 4.3% and 6.3%.6,7

Pharmacy directors acknowledge that executing a successful research project is one of the most challenging aspects of residency training.8 However, the specific aspects of the research and publication process that are most challenging are not well defined. Identifying these aspects may prove useful for residency programs desiring to improve their research training and project publication rates. The purpose of this study was to identify barriers to pharmacy residency research project completion and publication from the perspective of the program directors and former residents. We hypothesized that perceived barriers to publishing resident research projects from the perspective of the residency director will differ from those of the resident.

Methods

Study Design and Participants

This was a cross-sectional survey of pharmacy residency program directors and former residents. Postgraduate year 1 (PGY1) and postgraduate year 2 (PGY2) residents who would have presented projects at WSC in 2009, 2010, or 2011 were eligible to complete the survey. If a resident had completed both a PGY1 and PGY2, then they were asked to answer the survey based on their PGY2 experience. This study was submitted to the institution review board and determined to be exempt from review.

Survey Development and Content

Two surveys specific to the perspectives of program directors and former residents, but which contained similar items for comparison purposes were developed. The surveys focused on four broad areas: 1) demographic characteristics of the residency programs, program directors, and former residents, 2) level of difficulty associated with various stages of research (rated on a Likert-scale: 1=hardly challenging, 5=very challenging), 3) perceived value of the research project(rated on a Likert-scale: 1=strongly disagree, 5=strongly agree), and 4) self-identified barriers to successful research project completion (open-ended response).

Survey Administration

The ASHP residency program directory was used to identify residency directors of programs who would likely attend the WSC. The directors were sent an email containing a short explanation of the study and a link to the online survey (SurveyMonkey, Palo Alto, CA). Three additional emails were sent at seven to ten day intervals to non-responders to encourage completion of the survey. Program directors were asked to forward the email to their former residents who had completed a residency between 2009 and 2011. This timeframe was chosen as it allowed residents at least 12 months to submit their project for publication while limiting the potential for recall bias.

Statistical Analysis

Percentages were used to summarize categorical data. Likert-scale responses were dichotomized by grouping answers 1 through 3 and 4 and 5. Differences in categorical variables between program directors and former residents were compared using Pearson’s chi-squared test of association or Fisher’s exact test for small numbers. Responses to open-ended questions were categorized into major themes independently by two study investigators. Discrepancies in categorizing responses were resolved by consensus and involvement of a third investigator. Data analysis was performed using Microsoft Excel (Microsoft, Redmond, WA) or GraphPad (GraphPad Software, La Jolla, CA). A p-value of <0.05 was considered statistically significant.

Results

There were 32 pharmacy directors who completed the survey. The number of former residents who received the survey is unknown, however 97 completed the survey. Demographic and program characteristics are detailed in Table 1. The majority of program directors represented institutions offering PGY1 training (87.5%) and used a variety of projects types to fulfill the research requirement including: MUEs (65.6%), original research (81.3%), and quality improvement projects (78.1%). The minority of programs provided formal research training (41.9%) and most required projects to be in a format suitable for publication at the end of the residency (81.3%). In general, program directors were experienced (62.5% had been the program director for ≥5 years) and reported feeling comfortable or very comfortable (50.0%) with the research process.

Table 1.

Residency program director, residency program, and resident characteristics

Characteristic, % (n)	Residency Program Director and Program (n=32)	Resident (n=97)
Years as a residency director
< 5 years	37.5 (12)	N/A
5 to 9 years	37.5 (12)	N/A
> 10 years	25.0 (8)	N/A
Types of residencies offered*
General- PGY1	87.5 (28)	N/A
Specialty - PGY2	40.6 (13)	N/A
Community- PGY1	21.9 (7)	N/A
Managed Care- PGY1	9.4 (3)	N/A
Administration- PGY1/PGY2	9.4 (3)	N/A
Years institution has offered residency
< 2 years	3.1 (1)	N/A
3 to 5 years	12.5 (4)	N/A
6 to 10 years	21.9 (7)	N/A
> 10 years	62.5 (20)	N/A
Number of residents per year
<2	43.8 (14)	N/A
3 to 5	21.9 (7)	N/A
6 to 15	31.3 (10)	N/A
> 15	3.1 (1)	N/A
Residency completed
PGY1	N/A	67.3 (66)
PGY2	N/A	31.6 (31)
Number of publications
None	15.6 (5)	64.3 (63)
1 to 2	37.5 (12)	31.6 (31)
3 to 4	21.9 (7)	3.1 (3)
>5	25.0 (8)	1 (1)
Comfort with the research process
Very comfortable	21.9 (7)	6.1 (6)
Comfortable	28.1 (9)	35.7 (35)
Moderately comfortable	25.0 (8)	37.8 (37)
Somewhat comfortable	25.0 (8)	17.3 (17)
Not comfortable	0.0 (0)	2.0 (2)
Status of residency project
Writing in progress	N/A	20.4 (20)
Submitted, not accepted, will not resubmit	N/A	1.0 (1)
Submitted, not accepted, plan to resubmit	N/A	5.1 (5)
Submitted, currently under revision	N/A	10.2 (10)
Manuscript in press or published	N/A	15.3 (15)
No plans to publish	N/A	46.9 (46)
Project types used to fulfill requirement*
Original research	81.3 (26)	81.6 (80)
Quality improvement	78.1 (25)	42.9 (42)
Drug or medication use evaluation	65.6 (21)	51.0 (50)
Problem-solving exercise	6.3 (2)	21.4 (21)
Other	3.8 (3)	2.0 (2)
Program month that residency projects start
1st month	28.1 (9)	31.6 (31)
2nd or 3rd month	59.4 (19)	57.1 (56)
4th or 5th month	12.5 (4)	7.1 (7)
> 6 months	0.0 (0)	3.1 (3)
Formal research training offered	41.9 (13)	44.9 (44)
Projects required to be in a format suitable for publication	81.3 (26)	83.7 (82)
* Not mutually exclusive PGY1= postgraduate year 1; PGY2= postgraduate year 2

The majority of residents had completed only a PGY1 residency (67.3%). Consistent with the director responses, the minority of residents reported receiving formal research training (44.9%) during their residency and that their project was required to be in a format suitable for publication by the end of the residency (83.7%). Almost half of resident respondents (46.9%) were not planning to publish their projects while 15.3% reported their project was either published or in press.

Overall, both the program directors and residents felt the projects were beneficial to the institution (73.3% vs. 64.5%, respectively, p=0.50) and provided an opportunity to develop adequate management and administrative skills (60.0% vs. 59.1%, respectively, p=0.93) (Table 2). Approximately half of program director and resident respondents felt resident projects adequately prepared them for conducting future research (46.7% vs. 50.5%, respectively, p=0.87). Only about a third of program director and resident respondents thought that conducting a resident project expanded career opportunities (33.3% vs. 35.5%, respectively, p=0.97).

Table 2.

Perceived value of pharmacy resident research projects

Survey Item, median (%, n)	Residency Program Director (n=30)	Resident (n=93)	p-value
The results of my research or practice-related project were valuable to my institution	4.5 (73.3%, 22)	4.0 (64.5%, 60)	0.50
The research or practice-related project adequately prepared me to conduct research in the future	3.0 (46.7%, 14)	3.5 (50.5%, 47)	0.87
The research or practice-related project provided me adequate management/administrative skills	4.0 (60.0%, 18)	4.0 (59.1%, 55)	0.93
The research or practice-related project provided expanded career opportunities	3.0 (33.3%, 10)	3.0 (35.5%, 33)	0.97
Rated using a 5-point Likert scale, where 1=strongly disagree and 5=strong agree. Responses dichotomized and presented by proportion who reported strongly agree or agree as well as median values.

The most commonly reported barriers to completing resident projects reported by directors and residents were developing a realistic timeline to complete the project in one year (40% vs. 45.2%, respectively, p=0.67) and working through the publication process (73.3% vs. 43.0%, respectively, p<0.01) (Table 3). However, 46.7% of program directors identified obtaining IRB or department approval as a significant barrier compared to only 22.6% of residents (p=0.02). There was also a significant difference between groups in the perceived barrier posed by both collecting (13.3% vs. 34.4%, p=0.02) and analyzing data (20% vs. 39.8%, p=0.04) with residents viewing these aspects as more challenging than the directors.

Table 3.

Comparison of perceived barriers to completing residency research projects*

Research Area, median, (%, n)	Residency Program Director (n=30)	Resident (n=93)	p-value
Identify a mentor to guide the resident through research process	2.0 (16.7%, 5)	2.0 (10.8%, 10)	0.52
Identify an original research question	3.0 (33.3%, 10)	3.0 (31.2%, 29)	0.82
Formulate an appropriate study design and research protocol	3.0 (26.7%, 8)	3.0 (25.8%, 24)	0.93
Develop a realistic timeline to complete the project in one year	3.0 (40.0%, 12)	3.0 (45.2%, 42)	0.67
Obtain IRB and/or department approval	3.0 (46.7%, 14)	2.0 (22.6%, 21)	0.02
Collect data	2.0 (13.3%, 4)	3.0 (34.4%, 32)	0.02
Analyze data	2.0 (20.0%, 6)	3.0 (39.8%, 37)	0.04
Working through the publication process	5.0 (73.3%, 22)	4.0 (43.0%, 40)	<0.01
* Not mutually exclusive IRB, institutional review board Rated using a 5-point Likert scale, where 1=hardly challenging and 5=very challenging. Responses dichotomized and presented as proportion who reported very challenging or challenging.

There were 29 program directors and 90 residents who identified at least one barrier in the open response portion of the survey (Table 4). Major themes that emerged included: time constraints, lack of resident motivation, IRB approval or compliance with other rules or regulations, limitations in study design or quality of the study, effective mentorship or program structural issues, resident knowledge gaps, and continuity issues following residency completion. Both program directors and residents reported that two of the top three barriers to publication were time constraints (58.6% vs. 62.2%, respectively) and limitations with the study design or quality of the study (59.6% vs. 56.6%, respectively). However, as the third barrier, program directors cited the theme of resident motivation (65.5%) while residents cited the theme of effective mentorship or issues related to the program structure (43.3%).

Table 4.

Self-identified barriers to completing and publishing residency projects*

Major Theme, % (n)	Residency Program Director (n=29)	Resident (n=90)
Time	58.6 (17)	62.2 (56)
Resident motivation	65.5 (19)	27.8 (25)
Limitations in study design or quality of the study	58.6 (17)	56.7 (51)
Mentorship or program structural issues	24.1 (7)	43.3 (39)
Institutional Review Board and other rules and regulations	17.2 (5)	8.9 (8)
Resident knowledge gap	17.2 (5)	15.6 (25)
Continuity post-residency	10.3 (3)	3.3 (3)
* Not mutually exclusive

Discussion

While some literature currently exists assessing the perceived value of residency research projects9, limited information exists identifying barriers to conducting and publishing these projects. The barriers identified in our work are similar to those identified with medical resident research training and practicing pharmacists.10,11 We hypothesized that perceived challenges may be different between residency program directors and former residents and indeed, we learned that while there were many similarities between the groups, there were important differences in key areas.

Both program directors and residents valued the pharmacy residency projects, felt they were beneficial to their institution, and helped residents develop project management and administrative skills. These findings support those of another study aimed at documenting the perceived value of resident projects.9 It is our belief that an important component of the ASHP resident project requirement is to increase the resident’s competence and comfort with the research process so that they will participate in practice-based research in some capacity throughout their careers. While our study did not evaluate research competence, only about half of directors and residents reported feeling comfortable with the research process. Similarly, about half of respondents agreed that the residency project adequately prepared them to conduct future research, also supporting similar finding from an earlier study.12,13

Not surprisingly, common barriers identified by both program directors and residents involved generating quality projects that could be successfully completed within the time constraints of a one year residency. For a variety of reasons, this likely translates to low publication rates. Issues surrounding time constraints emerged as a recurring theme at all points of the research process including obtaining IRB approval, data collection and analysis, and manuscript preparation. Residents also noted that competing residency responsibilities further diminished their ability to work on research throughout the year. Time constraints may also result in many programs opting to engage residents in institution-specific small scale retrospective studies, such as MUEs or quality assurance projects, which often have limited external validity and are more difficult to publish. Most respondents indicated that projects were initiated between the second and third residency month. Recognizing that the first month of residency is extremely busy for both the program and resident, beginning projects as soon as possible during this time may help alleviate time constraints.

As discussed previously, while program directors and residents agreed on the first two self-identified barriers to project publication, they differed on the third. Whereas program directors frequently cited a theme relating to resident motivation, residents identified a theme relating to effective mentorship and program support. While reasons underlying this disconnect are not fully addressed by this study, we believe that our study results provide some insight and potential areas of opportunity. First, more than half of both program director and resident respondents reported a lack of a formal research training process and individuals from both groups identified resident knowledge gaps as a top barrier to publication. Studies have shown that objectively assessed research-related knowledge of residents is low after residency completion.13,14 As a result, approaching the research requirement from the same structured process as many of the other residency components may help increase resident confidence and competence in the research process. Recognizing that this may not be feasible at all institutions, pairing residents with mentors experienced in the research process and motivated to produce high quality research may help achieve similar endpoints as it has with practitioners.11 Additionally, complementing this mentorship with individuals with expertise in targeted areas, such as biostatistics or administrative data queries, may further facilitate movement through the various stages of the research process.

Second, while the vast majority of respondents stated a manuscript in a format suitable for publication was required for residency completion, very few residents published their work.5,6,7 It has been previously remarked that residency research projects often fall short of the high standards set by peer-reviewed journals simply because projects are not given priority or designed for publication from the start.15 If publishing resident projects is a goal of residency programs, then programs might consider allowing multiple residents to collaborate on a single project16, offering projects that span multiple years, capitalizing on completed research with follow-up studies, or conducting a feasibility analysis prior to project initiation.

Both groups identified difficulties ensuring project continuity and publication following residency completion as another barrier. Once again, assigning an experienced researcher from the institution to each resident project may help in developing a post residency communication plan to ensure that projects do not languish. Additionally, setting clear expectations at the start of the residency about submitting and publishing the study results may help alleviate this barrier. Finally, a newly created journal dedicated to publishing pharmacy resident (past and present) projects offers another option for submission of these projects which may improve publication rates.17

Our study has limitations. We had a low response rate from program directors despite repeated reminders. While we still feel that the responses we received provided valuable insight in barriers to residency research training there is the possibility that these results are not reflective of the entire population of residents and program directors. Second, the survey responses were anonymous and thus we do not know specifically which programs responded which could have affected the results, as residency programs with a larger infrastructure may be more likely to have current contact information for past residents. This bias may be supported by the fact that 15.5% of residents stated that their residency project was either published or in press. This percentage is higher than that previously reported for WSC research projects of less than 10%6,7, although consistent with a study of the SERC.5 Our inquiry about whether programs had a “formal research process” was not specific thus may have been interpreted differently among respondents.

Conclusions

In conclusion, it appears both residency program directors and residents agree that time constraints and low study quality are major factors limiting publication of resident research projects. However, program directors also felt that residents lacked the motivation to work through the publication process while residents felt poor mentorship and a lack of program support were larger factors. If programs wish to improve the publication rate of residency research projects, results of this study may provide some opportunities for improvement including formal research training that includes information on biostatistics, a plan for continued communication between resident and mentor upon residency completion, and an increase in the time allocated for research. Additionally, if developing pharmacy researchers is a goal of our profession, then perhaps strengthening the language in the accreditation standards to specifically say “research” may push programs to develop stronger original research projects.

Funding Statement

This study was funded internally by the Kaiser Permanente Colorado pharmacy department.