Pharmacy resident publication success: factors of success based on abstracts from a regional meeting

Abstract

Objective

To determine the publication rate among pharmacy resident research projects in a region of the United States and to compare characteristics of published and unpublished projects.

Methods

Research project abstracts from the Great Lakes Pharmacy Residency Conference in 2003, 2005, and 2007 were reviewed. Two independent investigators collected all study data. Data on residency year, state, institution, study design, and whether results were reported were extracted from available abstracts. Publication rate was determined systematically using a search algorithm within the following databases: Scopus, International Pharmaceutical Abstracts (IPA), and MEDLINE (PubMed). Kappa-statistic was used to determine inter-rater variability. Descriptive statistics were used to analyze nominal and continuous data. Univariate and multivariate regression analyses were used to determine characteristics of publication success. Sensitivity analysis was performed on projects that were successfully published.

Results

Information was extracted from 655 abstracts in which 76 abstracts were published (11.4%). Publication rate trended down over the three years analyzed (2003=12.9%, 2005=12.2%, 2007=9.9%; p=0.57). Study design (interventional, observational, cross-sectional, or service development, p=0.115), direction of inquiry (prospective or retrospective; p=0.146), intervention of interest (drug, human, or other; p=0.096), results in abstract (p=0.096), and institution type (university-affiliated, veterans affairs, community-hospital, or retail; p=0.001) were entered into the multivariate model. Cross-sectional design (odds ratio (OR) 3.6), human (OR 1.9) and other (OR 2.1) interventions, as well as university-affiliated residency (OR 2.6) remained significant for publication success. The mean time to publication from abstract to presentation was 24.5 months, and 83% of projects were published within pharmacy journals.

Conclusion

Publication rate of pharmacy resident research projects presented at the Great Lakes Pharmacy Residency Conference is low, but it is consistent with other regions of the United States. Study design and study outcomes may influence chance of project publication as well as institution-type, which may have unique research resources, training, and mentorship.

Introduction

A component of clinical pharmacy is the generation of new knowledge.¹ Consistent means of training pharmacists with the research and practice skills required to advance knowledge through translational research have been identified as a need.^2,3 There are multiple opportunities to prepare pharmacists to conduct research. These include Doctor of Pharmacy (PharmD) degree programs, biomedical research degree programs, mentored training experiences offered by national pharmacy organizations, and postgraduate residency or fellowship training. However, experiences focused on intensive research training are insufficient to meet this demand.³

Postgraduate residency training programs offer pharmacists additional opportunities to develop research skills while advancing clinically. Residency is the final training experience many pharmacists have prior to a clinical pharmacist position. Completion of a quality improvement or research project and presentation of a final project report in a manuscript style suitable for publication is required of American Society of Health-System Pharmacists (ASHP) accredited post graduate year 1 (PGY1) programs and post graduate year 2 (PGY2) programs.^4,5 Interest and availability of PGY1 and PGY2 residency positions are continuing to grow⁶ due to increased expectations of residency training for all pharmacists participating in direct patient-care.^7–9 The number of pharmacists serving as primary authors of biomedical research is increasing along with the volume of pharmacists completing residency training;¹⁰ however, it cannot be assumed that the observed increase in pharmacist publications along with residency training is correlated, as a small subset of academic pharmacists may be responsible for the majority of these publications.¹¹ Furthermore, the publication rates of pharmacy resident research projects after presentation at regional or national conferences are <15%, with some studies suggesting a decrease in the publication rate.^12–18

Completion of resident research is considered highly challenging but beneficial, to the resident, program, and institution.^19,20 Bookstaver et al found scholarly activity related to resident research is important to residents but barriers exist to publication of their research.²¹ Inadequate time is commonly cited while lack of mentorship, low research quality, inability to complete the project or final manuscript, and lack of knowledge or difficulty working through the publication process have all been identified as barriers to publishing resident research.^13,21–23 Residency program directors, preceptors, and pharmacy residents would benefit from identifying the characteristics of a residency research project that make it more likely to be published. The objective of this study is to evaluate the characteristics of published pharmacy practice residency projects presented at the Great Lakes Pharmacy Residency Conference from 2003 to 2007.

Methods

This is a cross-sectional analysis of pharmacy resident abstracts presented at the Great Lakes Pharmacy Resident Conference (GLPRC). This study was deemed exempt by the St. Louis College of Pharmacy Institutional Review Board, as all data were accessed on the conference website (http://www.glprc.com/).

We aimed to assess the publication rate of pharmacy resident research projects based on abstracts presented in 2003, 2005, and 2007. These years were chosen as 2003 was the earliest conference in which abstracts were available online. We selected 2007 as our end date in order to evaluate publications for up to five years post-presentation and to assess publication success.¹² We selected only 2003, 2005, and 2007 to avoid residents being counted twice (if participating in two years of residency training).

Primary Outcome

Our primary outcome was the publication rate of the pharmacy resident research abstracts. Publication success was used as a measurable outcome to assess research training and experience achieved during residency. A pre-defined systematic search strategy was used to determine publication success. The search strategy included the search engines in the following order: 1) Scopus®, 2) International Pharmacists Abstracts, and 3) MEDLINE (PubMed) using the resources available through the St. Louis College of Pharmacy Library. The search strategy within the search engine continued until the search terms returned <25 results which were then manually reviewed. Each search within the individual search engines included, in sequential steps: 1) author’s last name, 2) author’s last name and first initial, 3) author’s last name, first initial, and three to five key words from the project’s title.¹⁵ If there were no hits on the first author, the same strategy was performed for each available author (to account for a change in last name). When the list of authors was exhausted, the search strategy was repeated in the subsequent search engines. During any point of the search strategy in which a publication was found, it was considered publication positive and the search was completed. If no publication was found by the end of the search strategy using all three search engines, the resident research was considered publication negative. A research project was considered published if found regardless of publication type (e.g., original research, brief report, meta-analysis).

Covariates and Secondary Outcomes

We collected the abstract year, study design, study type, direction of inquiry, intervention of interest, results in the GLPRC abstract, state residency conducted in, and institution type from the published abstracts similar to previously published studies.¹⁵ Study type was categorized into the following groups and collapsed where indicated for regression analysis: randomized-controlled, non-controlled interventional, non-randomized interventional, bench (further collapsed into interventional), cohort, case-control, (further collapsed into directional/observation), cross-sectional, medication utilization evaluation (MUE), survey, systematic review (further collapsed into cross-sectional), or service development project. We defined study design as either interventional or observational. We defined direction of inquiry as prospective or retrospective and defined intervention of interest as drug (e.g., medication utilization review), human (e.g., educational intervention to a patient or provider), or other (non-drug/non-human; e.g., evaluation of adherence to guidelines). Positive results in the GLPRC abstract were defined as preliminary or final results. An institution type was defined as university-affiliated (school of pharmacy or academic institution), veteran affairs (VA)-affiliated, retail/community, non-university/non-VA affiliated hospital, and other (e.g., industry).¹⁵ Moreover, we collected time to publication (months), journal type (pharmacy journal or non-pharmacy journal), number of total authors, and resident authorship from abstracts that were publication positive. For our secondary outcomes, we assessed for publication trends, publication success based on parameters of the study (as listed in the abstract), types of journals the abstracts were published in, and the time to publication.

Variable Confirmation

Two independent investigators collected variables and used the search strategy to identify publication positive abstracts. If the investigators were in agreement, then the result was carried forward for analysis. If there was not agreement on the publication result, a consensus was formed between the two investigators, and the result was carried forward for analysis.

Statistical Analysis

Descriptive statistics were used to analyze nominal and continuous data. Kappa-statistic was utilized to calculate inter-rater reliability for each variable prior to consensus. Chi-square (or Fisher’s Exact) tests were used to compare characteristics in the published versus non-published abstracts. Means and standard deviations were used to assess time to publication as data were normally distributed. Sensitivity analyses were performed within university-affiliated programs and on publication positive resident research projects.

Multivariate logistic regression was used to determine characteristics that predict publication success. Variables were tested for multi-collinearity, interactions, and outliers. Variables with p<0.2 on univariate analysis were forced into the model. Residency state and abstract year were not entered into the model regardless of significance (determined a priori), as the GLPRC did not include all programs/residents within each state and year. Statistical analyses were performed using IBM-SPSS version 22.0 (IBM Corp., Armonk, NY).

Results

A total of 655 abstracts were evaluated for the GLPRC in years 2003 (n= 171), 2005 (n=221), and 2007 (n=273). Level of agreement for dependent variables was fair to moderate while the independent variable of publication positive was classified as substantial agreement (Appendix 1).²⁴ No interactions, outliers, or multicollinearity existed in these data.

Using our search strategy, 11.4% (n=76) met the criteria for publication. Demographic information regarding year, state, study design, study type, direction of interest, intervention of interest, results, and institution type are described in Table 1. The overall model significantly predicted publication success (Nagelkerke R-square = 0.093; p<0.001) (Table 2). Significant predictors for publication success included the following: 1) cross-sectional studies were more likely to be published compared to service development projects; 2) research with human or other outcomes were more likely to be published compared to drug outcomes; and 3) projects performed in a university-affiliated residency were more likely to be published compared to non-university/non-VA affiliated hospitals (Table 2).

Table 1.

Demographic Information for Resident Research Projects at the Great Lakes Pharmacy Resident Conference: 2003, 2005, 2007

Outcomes	n/total (%)	P-Value
Year		P=0.570
2003	22/171 (12.9)
2005	27/221 (12.2)
2007	27/273 (9.9)
State*		P=0.186
Illinois	18/143 (12.6)
Indiana	4/81 (4.7)
Kentucky	1/22 (4.3)
Michigan	17/121 (14.0)
Ohio	24/170 (14.1)
Wisconsin	12/123 (9.8)
Study Design		P=0.001
Randomized Controlled	1/20 (5.0)
Non-Controlled Interventional	6/43 (14.0)
Non-Randomized Interventional	4/21 (19.0)
Bench	2/8 (25.0)
Cohort	28/255 (11.0)
Case-Control	2/13 (15.4)
Cross-Sectional	12/106 (11.3)
Medication Utilization Evaluation	5/87 (5.7)
Survey	13/39 (33.3)
Systematic Review	0/1 (0)
Service Development	3/72 (4.2)
Study Type		P=0.380
Interventional	13/92 (14.1)
Observational	63/573 (11.0)
Direction of Interest		P=0.130
Prospective	16/101 (15.8)
Retrospective	60/564 (10.6)
Intervention of Interests		P=0.100
Drugs	26/304 (8.6)
Human	28/206 (13.6)
Other	22/155 (14.2)
Results in GLRPC Abstract		P=0.082
Yes	18/111 (16.2)
No	58/554 (10.5)
Institution Type		P=0.002
University-Affiliated	49/287 (17.1)
Veterans Affairs Affiliated	6/101 (5.9)
Community/Retail	3/24 (12.5)
Hospital (non-University/no-VA)	18/248 (7.3)
Other	0/5 (0)

^*May not represent all residency programs/residents within the state

GLRPC = Great Lakes Regional Pharmacy Conference

Table 2.

Logistic Regression Analysis for Resident Research Projects at the Great Lakes Pharmacy Resident Conference: 2003, 2005, 2007

Outcomes	Univariate Analysis OR (95% CI)	P Value	Multivariate Analysis OR (95% CI)
Study Design (Collapsed)†		P=0.115
Interventional	3.78 (1.04–13.84)		1.73 (0.26–11.44)
Directional/Observational	2.99 (0.86–9.79)		3.27 (0.93–11.51)
Cross-sectional	3.40 (1.01–11.49)		3.57 (1.03–12.39)
Service Development	[Reference]		[Reference]
Study Type		P=0.393
Interventional	[Reference]		----
Observational	0.75 (0.40–1.43)
Direction of Interest		P=0.146
Prospective	[Reference]		[Reference]
Retrospective	0.63 (0.35–1.15)		2.49 (0.63–9.85)
Intervention of Interest		P=0.096
Drugs	[Reference]		[Reference]
Human	1.68 (0.96–2.96)		1.91 (1.05–3.47)
Other	1.77 (0.97–3.24)		2.11 (1.11–4.03)
Results in GLRPC Abstract		P=0.096
Yes	1.66 (0.93–2.94)		0.70 (0.38–1.28)
No	[Reference]		[Reference]
Institution Type*		P=0.001
University-Affiliated	2.63 (1.49–4.65)		2.61 (1.45–4.67)
Veterans Affairs Affiliated	0.81 (0.31–2.10)		0.87 (0.33–2.29)
Community/Retail	1.83 (0.50–6.71)		1.42 (0.36–5.57)
Hospital (non-University/no-VA)	[Reference]		[Reference]

CI = confidence interval; OR = odds ratio

^*Other (n=5) removed from Analysis

^†Interventional=randomized controlled, non-controlled interventional, non-randomized intervention, bench; Directional/Observational=cohort, case-control; Cross-sectional=cross-sectional, medication utilization review, survey; systematic review

GLRPC = Great Lakes Regional Pharmacy Conference

VA = Veterans Administration

To further explore university-affiliated resident research project publication success, a sensitivity analysis was performed. Of the 287 abstracts affiliated with these programs, 17.1% (n=49) met the criteria for publication. Additionally, we performed a univariate analysis for study design (collapsed), study type, direction of interest, intervention of interest, and results, all of which culminated in p>0.2 and were, therefore, not further explored (data not shown).

Finally, the characteristics of publication positive residency research projects were assessed. The mean time from abstract presentation at GLPRC to manuscript publication was 24.5 months (Table 3). A majority of publications were in pharmacy journals and included four or more authors with the resident most likely to be first author.

Table 3.

Characteristics of Published Resident Research Projects

Outcomes	N=76
Time to Publish (months): Mean (SD)	24.5 (14.3)
Journal: n(%)
Pharmacy Journal	63 (82.9)
Non-Pharmacy Journal	13 (17.1)
Number of Authors
One or Two	10 (13.2)
Three	17 (22.4)
Four	28 (36.8)
Five or More	21 (27.6)
Resident Author Position
First Author	66 (86.8)
Second Author or Later	10 (13.2)

Discussion

In our study, the resident research publication rate of 11.4% was low but similar to the results of previously published studies examining pharmacy resident publication rates in different regions of the U.S. and Canada.^12–17 McKelvey and colleagues found the publication rate of resident abstracts after presenting at the Southeastern Residency Conference in 2001 to be 12.5% using MEDLINE and IPA.¹² This study also observed a decreasing publication rate from 1981 to 2001, similar to our findings. A survey of 278 residency program directors (RPD)s across the U.S. found a resident research publication rate of 13.2% over a three year period from 1996–1999.¹⁹ Olson and colleagues found a publication rate of 6.3% using MEDLINE and EMBASE after presenting at the Western States Conference for Pharmacy Residents in 1995, 2000, and 2005; the publication rate increased from 4.2% in 1995 to 8.2% in 2005.¹³ O’Dell and colleagues found a lower publication rate of 4.3% at the same conference in 2008 using only MEDLINE.¹⁴ Canadian resident publication success has also been investigated. Hung and colleagues used MEDLINE, The Canadian Journal of Hospital Pharmacy, and Google to find a publication rate of 20.6% over a ten year period from 1999 to 2009.¹⁵

Pharmacy resident publication rates are lower compared to research by pharmacists (11–33%)^16,17 and medical professions (44.5%) after presentation at national conferences.²⁵ Direct comparison of these publication rates to the present study is cautioned, however, as research quality and intent to publish is assumed to be higher at national conference than at a regional resident conference. To the authors’ knowledge, this is the first study examining the factors associated with success of research project publication within five years of pharmacy residency completion using multivariate analysis.

For many pharmacists, the training received during a pharmacy residency program is the final step taken prior to entering positions where scholarly activity is a job expectation. However, there are many reasons why residencies may not adequately prepare pharmacists for performing independent research and achieving scholarly success. A survey of pharmacy residents conducted by Ellis and colleagues found that residents’ self-assessed research skills and knowledge improved during residency, but this was unchanged when assessed objectively using a multiple choice pre-post residency test.²⁶ Research and writing skills are not adequately reinforced as part of the goals and objectives of a residency training experience.^27,28 The pharmacy resident research experience is highly dependent on the study type and location of the residency. In our study, resident research from university-affiliated residencies was more likely to have publication success. The success of university-affiliated residency programs may be the result of having features that promote innovative and feasible research. University-affiliated programs may have access to larger and more diverse patient populations, information technology services and database access, or experienced research mentors who share the expectation of scholarly activity.

Time is a large barrier to resident publication success. In a survey of pharmacy residents and RPDs participating in the Western States Conference, the most common obstacle to completing resident projects ranked by both RPDs and residents was developing a realistic timeline to complete research in one year (40% vs. 45.2%, respectively) and working through the publication process (73.3% vs. 46.7%, respectively).²³ Bookstaver et al found similar results in their survey of PGY1 residents.²¹ The limited time may prevent some residents from completing and submitting manuscripts for publication, as noted by Hasegawa.²² Further, designing a high quality study to answer meaningful questions is difficult to complete in one year. Low research quality, poor investigation design, or projects with limited external meaning limit residents to pursue publication and be successful completing the publication process.^16,18,22 Low publication rates may also reflect submission of manuscripts to journals that are not interested in the scope and focus of common resident research. AJHP now offers “AJHP Residents Edition,” a quarterly online supplement aimed to publish high-quality resident research and offers a promising opportunity for residents to experience the publication process.²⁹ Time and resources should be considered when deciding on investigation design and scope of a residency project with the aim of producing a manuscript of publication quality.

We found that certain resident research project designs were relatively more likely to be published. Descriptive studies of service development and randomized controlled trials were successfully published ≤ 5% of the time. These studies are often left to the mercy of factors outside of the resident’s control such as enrollment and implementation. Service development may also have limited generalizability. Cross-sectional studies (including surveys) and retrospective cohort studies, which may be more feasible to complete within one year, were the most commonly conducted types of studies and were associated with a higher likelihood of publication (≥ 10% successful). Programs, project preceptors, and residents serve to benefit from a feasibility analysis of a research idea prior to embarking on a research project. The resources and time needed to adequately answer the research question should be fully considered before committing to a project, as there may be limited time during the residency to backtrack or start over and still complete a high quality study.

Limitations

First, not all residency programs within the Great Lakes region present at the GLPRC, decreasing our studies ability to capture the true resident publication rate in this region. Because of this, publication rates by state were not included in the univariate and multivariate analysis. Second, we were unable to differentiate publication success between PGY1 and PGY2 residents, as this was not clearly described on the abstracts. Previous studies suggest that publication success was significantly higher in PGY-2 residents (30.8%) compared to PGY1 residents (10.5%; p=0.0185).¹⁴ Presentation at specialty meetings may also indicate higher research quality and therefore higher chance of success; however, our design was unable to capture this information. Third, we were unable to determine if certain residents participated in higher levels of training (i.e., master’s level research courses) either previously or concomitantly with their residency program which may have influenced the success of their research project. Fourth, misclassification bias may have been present as it was difficult to infer variables (e.g., study design, study type) based on the abstract alone; however, two separate investigators evaluated each variable which resulted in a fair level of agreement. Fifth, over-fitting of the model may have occurred with the vast number of initial variables that were assessed; to overcome this problem, several related variables were collapsed. Sixth, residents may receive research training or gain experiences completing their projects that are not captured based upon manuscript publication. However, publication success is a measurable and consistent outcome across residency programs. Lastly, our study uses historical data. While this is the only study in this population, publication rates may be different after 2007.

There are other studies that may be conducted to further expand on this body of literature. Publication success during a residency program may predict future publications after completing the residency. Additionally, residents with master’s level training in research may have higher publication success rates, which may suggest allowing residents to pursue further didactic training if desired. Moreover, further studies focusing on resident research through a research advisory panel may help to improve study design and understand feasibility, therefore improving the quality of research given the limitation of time.³⁰

Conclusion

Overall, the publication rate of pharmacy resident research projects was 11.4% as determined by our search strategy. The publication rate for 2003, 2005, and 2007 at the Great Lakes Pharmacy Residency Conference was low but consistent with other regions of the United States. Several variables were identified that may influence the chance of project publication.