A Systematic Review of Extramural Presentations and Publications from Pharmacy Student Research Programs

Abstract

Objective. To conduct a systematic review of reports of pharmacy student research programs that describes the programs and resulting publications or presentations.

Methods. To be eligible for the review, reports had to be in English and indicate that students were required to collect, analyze data, and report or present findings. The outcome variables were extramural posters/presentations and publications.

Results. Database searches resulted in identification of 13 reports for 12 programs. Two-thirds were reports of projects required for a course or for graduation, and the remaining third were elective (participation was optional). Extramural posters resulted from 75% of the programs and publications from 67%.

Conclusion. Although reporting on the outcomes of student research programs is limited, three-quarters of the programs indicated that extramural presentations, publications, or both resulted from student research. Additional research is needed to identify relevant outcomes of student research programs in pharmacy.

INTRODUCTION

Evidence-based medicine, defined as the integration of research evidence with clinical expertise and patient values¹ is based on the premise that a body of research relevant to patient care is available and that practitioners can locate, interpret, and apply the research to clinical practice. For research to be applicable to pharmacy practice, practitioners and scientists need to conduct relevant research and disseminate the findings, which implies that practitioners and scientists have the skills to create and use research findings. The Task Force on Research in the Professional Curriculum of the American College of Clinical Pharmacy (ACCP) identified eight key curricular competencies related to research skills for students in professional pharmacy programs. The competencies include identifying relevant problems, generating a hypothesis, designing a study, analyzing data using appropriate statistical tests, interpreting and applying the findings to practice, effectively communicating research and clinical findings to both professionals and patients, and applying regulatory and ethical principles to the conduct and use of research findings.² The American Society of Health-System Pharmacists (ASHP) includes research skills in the objectives for a postgraduate year one (PGY1) residency. Residents need to design and execute a study and report the results in the context of pharmacy practice issues.³ The need for students to learn skills in research and interpreting and using research is also recognized by pharmacy groups in other countries. The European expert committee on pharmaceutical education recommended in 1994 that all pharmacy curricula include a “significant” final year project. In 1996, the requirement for a student research project was made mandatory by the Royal Pharmaceutical Society of Great Britain.⁴

A key issue associated with research is whether or not the results are communicated to other pharmacy practitioners and health care professionals. Communication of research and clinical findings is specifically mentioned by ACCP and ASHP. The former specifically listed communication of research and clinical findings to pharmacists, other health care professionals, and basic scientists as one of the essential curricular competencies related to research content for doctor of pharmacy (PharmD) programs.² The latter lists effectively using communication skills to report the results and recommendations from an investigation as an instructional objective for PGY1 residency training.³ Communication with individuals, groups, and organizations also is listed as a competency in the Center for the Advancement of Pharmacy Education (CAPE) outcomes for professional pharmacy education programs.⁵ With respect to medical student research, Beirer observed that publication of student research in peer reviewed journals seemed to serve as the “gold standard” for success of student research programs.⁶

Studies on publication in pharmacy journals indicate that experience with research in professional programs is associated with subsequent publication and that student research projects can result in publications. In a study of factors associated with the publication of scholarly articles by pharmacists, Morris et al found that training in research methods and scientific writing as well as introduction to the publication process during training was associated with publication.⁷ Forty-two percent of respondents indicated that the research reported was completed during residency training, and 73% reported that they wrote their first publication while enrolled in some type of training program.⁷ Nykamp et al surveyed student authors of manuscripts published in pharmacy journals from 2004 through 2008.⁸ Forty-one percent of the respondents indicated that the manuscript was associated with a project that was part of either a didactic or experiential training requirement. Experience during an advanced pharmacy practice experience (APPE) accounted for 33% of the publications, about 11% were from a graduation requirement, and 5% were from a required course.⁸ The purpose of this study was to conduct a systematic review of reports of pharmacy student research programs and describe the characteristics of the programs and the extent to which student research projects resulted in extramural presentations and publications.

METHODS

Based on the definition used by Murphy et al,⁹ a report had to indicate that students were engaged in the entire research process from developing a protocol and/or submitting an institutional review board (IRB) application, collecting and analyzing data, and preparing a written report of findings or presenting/defending their findings at the college level or beyond to be included in the systematic review. The report needed to involve a specific student research program at a single college or school of pharmacy. Reports of student surveys involving multiple programs or of student research identified from publications in professional journals were excluded. Research projects that involved students in someone else’s research such as collecting data or that involved only writing a protocol were excluded. The reports needed to be published in English but there was no restriction on the country in which the student research was conducted. The primary outcomes of interest were extramural student presentations (including both poster and podium presentations) and extramural publications.

The search included eight bibliographic databases and six websites. The databases included PubMed/Medline, Embase, ERIC, Academic Search Complete, International Pharmaceutical Abstracts, Web of Science, Education Full Text, and Google Scholar. The databases were searched from inception to September 2014 with updates from PubMed until June 2015. Websites searched included the American Association of Colleges of Pharmacy, Association of American Medical Colleges, Accreditation Council for Pharmacy Education, ACCP, ASHP, and Liaison Committee on Medical Education. Search terms included controlled vocabulary for database and incorporated key words. Example search terms included pharmacy student, medical student, professional pharmacy, pharmacy graduates, PharmD student, publications, posters, research project, research experience, research activity, and presentations. The search strategy was developed in PubMed/Medline and used as a template for other bibliographic databases and focused on retrieving articles with the terms in the title and abstract. Key terms were used also to search the websites; however, there were limitations to the number of terms allowed. (See appendix A for the search strategy.) Also, citations in retrieved reports were searched for additional studies and known references were included in addition to those captured by the search.

Abstracts of the studies identified in the search were screened using a standardized tool that included five items on the study topic (eg, whether the topic related to student research and whether the student research involved collecting and analyzing data) and reasons for exclusion (eg, editorial, letter, or commentary, not reported in English). Two investigators screened the abstracts independently then resolved differences by consensus. The full text of any report identified as a potential candidate was retrieved and the process repeated using the full text. Only reports involving pharmacy student research were retrieved.

Data extraction was conducted using a standardized tool recording the citation, the school of pharmacy involved, whether the project was required, part of a research track, or an elective course, whether students worked individually or in groups, type of support provided to students for projects, duration of the project process, which process components (eg, written protocol, collection of data) were required, and the terminal requirement for the project (eg, poster, written report). Information available on extramural presentations or publications resulting from student projects also was recorded. Data were extracted independently by each investigator then discrepancies were resolved by consensus.

A meta-analysis¹⁰ and review⁶ of medical student presentations and publications from student research projects did not report assessing bias in study reports; hence, there are no published tools for assessing bias in this area. Using guidance from the Cochrane Collaboration,¹¹ we identified four factors that could have introduced bias into the review. Because the primary interest was in summarizing rates of publication and presentation, the focus was on factors that affect the numerator or denominator required to calculate rates. Four types of bias seemed possible: selection, methods, reporting, and follow-up. Selection bias was defined as bias resulting from inability to accurately categorize a study as meeting inclusion criteria because the description of the student research program is inadequate. Methods bias was defined as bias resulting from the procedures used to collect data on publications and presentations as an earlier study indicated that data on student presentations and publications were difficult to collect.¹² Reporting bias was defined as the bias resulting from reporting per student rates vs per project rates. Follow-up bias was defined as the bias resulting from a short data collection period given the length of time required for publication.

A flow chart was created to summarize the search procedure. Data on the characteristics of student research programs were summarized by constructing a table to display data on the location of the program, number of students involved, duration, number of students per project, support for projects, types of projects allowed, process components included, and the terminal requirement. Data on extramural publications and presentations were analyzed by creating a table summarizing the data obtained from each study report then calculating the percent of programs that reported publications and presentations. Bias assessment data are presented in Table 3 displaying the level of risk for bias for selection bias, methods, reporting, and timeframe.

Table 3.

Risk of Bias Assessment^a

RESULTS

The selection of studies for the systematic review is shown in Figure 1. The total number of records screened was 5238, including both the reports related to pharmacy and to medicine. Only reports related to pharmacy student research were included in this systematic review. Most full text articles involving pharmacy students were excluded because the student program involved only writing a protocol.^13-17 were quality improvement projects,^18,19 were community or clinic-based programs in which it was not clear what students did,^20-22 or students were only involved in data collection.²³ After screening, 13 reports describing student research programs were included in the systematic review. Two reports involved the same research program and were reported as one student research program.^24,25

Figure 1.

Flow Diagram of Study Selection. Records consisted of the citation and usually an abstract; Abstract only indicates that only the abstract was published, typically the citation was for a conference abstract and no report was identified. No report of single research program indicates that the citation referred to a survey or other description of multiple programs; QI=quality improvement.

The characteristics of student research programs included in the systematic review are shown in Table 1. There were three reports^26-28 (25%) of elective student research programs, one report of a student research track²⁹ (8%), and eight reports^24,30-36 (67%) of required student research programs. The elective student research programs involved relatively few students (the number ranged from 7 to 25) and the types of research projects were limited to primarily bench science and clinical studies. The duration of the elective research programs varied, from 10 weeks during the summer to four semesters. The one research track offered was more similar to the elective student research programs than to the required programs; only two types of research were offered, bench science and practice issues, and only 20 students participated at a time.²⁹

Table 1.

Characteristics of Studies Included in the Systematic Review

In the eight required research programs,^24,30-36 reported class sizes ranged from 50 to 210. Only two programs^33,34 required students to work individually; other programs allowed students to work in groups of one to seven students. The required student research programs typically offered a wide variety of topics for student research in addition to bench science and clinical topics, including public health, education, and business topics. One report involved research in public health as part of a course in public health that represented a unique approach for including research in the professional pharmacy curriculum.³¹ Seven of the programs (58%)^24,32-36 indicated that the duration of the research experience was at least a year; one of these³⁰ was described as a 2-year experience.

Only two of the elective research programs^26,27 reported that monetary support was provided to students, and these programs provided stipends. Student support for required projects included coursework, workshops, and guidance documents. Overall, the support seemed relatively limited given the multiple steps from developing a protocol to writing a report and/or presentation of findings required to complete a project.

Limited data were provided on the exact research process components required (Table 1). None of the reports specifically listed all the relevant components identified by ACCP as research skills that students should have.² Three quarters (75%) of the student research programs^24,26,30-36 required a written report as the terminal outcome, and two-thirds (67%) required a poster presentation.^{24,26-28,32,34-36} Half (50%)^{24,26,32,34-36} required both a poster presentation and a written report. Several reports indicated that students needed to write a publishable quality report but only one elective research program²⁶ required that students submit for publication. Of the programs requiring students to present posters, only one program required that the poster be presented in an extramural venue.²⁸

The extramural presentations and publications resulting from pharmacy student research projects are shown in Table 2. Ten of the 13 reports indicated that extramural presentations or posters had resulted from student research projects. The number of presentations resulting from elective research projects ranged from two to six and from seven to 68 for required projects. The number of publications reported ranged from one to six for elective projects and from 2 to 32 for required projects. In general the data were not adequate to calculate overall presentation or publication rates. The output from one elective student research program appeared to be one publication and six presentations from two projects.²⁸ The one required student research program reporting adequate data to calculate rates indicated that extramural presentation and publication rates in the first two years of the program were 28% and 5%, respectively, but were 24% and 29% in the following three years.^24,25

Table 2.

Extramural Presentations and Publications Resulting from Student Research Projects

The results of the risk of bias assessment are shown in Table 3. There was likely some selection bias as seven (54%) of the included studies were rated as unclear, indicating that there could have been misclassification of studies because the description of the process components was inadequate. The risk for bias related to methods was rated as high for six (60%) of the studies reporting data on presentations or publications. Risk of bias related to reporting the numbers of students and number of projects was generally unclear or high (62%) while risk of bias related to timeframe was generally low.

DISCUSSION

The most important finding of this systematic review of reports of research programs for pharmacy students is that student research projects resulted in extramural publications and presentations or both in 75% of the programs. Required research programs tended to offer a wider variety of projects to students than did elective programs. Elective programs were restricted to clinical or bench science topics while some required programs offered additional topics related to public health, other practice issues, education, business, and the use of computer information systems. It was also clear from this systematic review that substantially more students obtain experience in research through required student research programs than through elective programs. An additional important finding was that only 7% (10) of the 132 pharmacy programs in the United States had reported research on student research programs, hence little is known about pharmacy student research.

A unique and interesting model of providing students with research experience entailed conducting research in a public health course or community experience. Fuentes et al described a public health course in which students conducted research as part of a course.³¹ Students developed and administered a questionnaire, analyzed the data, and reported it to the community. Two other reports^21,22 described public health experiences in which students participated in research; however, the description of exactly what was involved in the research experience was limited so the reports were not included in the systematic review. Regardless, public health courses or experiences seem to offer an excellent opportunity for students to learn research skills.

The rates of extramural presentations and publications resulting from pharmacy student research projects are difficult to compare to other reports in the literature because of inadequate data. Multiple presentations and one publication resulted from two elective research projects,²⁸ while presentation and publication rates for one required student research program were between 24% and 29%.^24,25 In a review of the research outputs of medical students, Bierer et al found publication rates to vary between 8% and 85% and the rate for extramural presentations to vary between 10% and 41%.⁶ A detailed study of the output from student research at the Mayo Clinic found that 41% of student research projects resulted in publications and 41% resulted in presentations, with 66% of the projects resulting in one or the other or both.³⁷ In a meta-analysis of 17 studies from around the world of medical student research, the overall average for publication was 30% (CI = 0.19-0.44).¹⁰ In pharmacy, McKelvey et al examined research resulting from residency projects and found that the overall average for three years was about 16%.³⁸ In their study of publications, Morris et al implied that a rate of 20% for residency projects was low.⁷ Cooley et al found that the presentation/publication rate for students, including quality assurance and research projects, increased from 64% to 81% of students following an intervention.¹² Riveros et al noted that about half of studies registered in ClinicalTrials.gov were not published,³⁹ and Weber et al found that 55% of papers presented at a medical conference were not published five years later.⁴⁰ However, an appropriate presentation and publication rate has not been established, and, indeed, there is not agreement that extramural presentations and publications are appropriate outcomes to consider for student research. Mottram et al specifically stated that student research should not be undertaken to obtain presentations or publications.³³ Bierer et al noted that there were other outcomes such as effects on the student and subsequent practitioner behaviors, patient outcomes, and effects on organizational structure or institutional culture that also were important.⁶

As the assessment for risk of bias indicated, there were problems with conducting a review of published reports of student research programs. More than half the included studies (54%) were rated as unclear for selection bias indicating that the components of the student research program were not clearly described. Consequently, it was difficult to determine if a specific report was eligible for the systematic review and to interpret the findings reported related to publications and presentations. However, regardless of whether the focus is publications and presentations or some other aspect of the program, an adequate description of the student research program is needed. A standard framework for describing student research programs would assure that relevant components of the research process were described and enable users to compare programs. The framework provided in the ACCP document on the research skills that students should have on completion of their training² could serve this role. We could identify which of the eight identified research components were included in the research experience. Because the use of guidelines is thought to improve not only the quality of research reports but also improve the quality of research,⁴¹ guidelines specifically related to student research programs should be helpful.

The other area with the highest risk for bias was the methods used to collect data on publications and presentations. That the focus of most of the studies was on an aspect of student research programs, for example, faculty and student attitudes, likely accounts for the failure to describe methods for collecting data related to extramural publications and presentations. A concern with the methods used to collect data on extramural publications and presentations is that a single data collection method may not be adequate. Cooley et al demonstrated that a student survey identified only 64% of the publications and presentations that were identified using multiple other methods (eg, review of student CVs, review of posters presented at conferences).¹²

The third area with risk for bias was reporting. To calculate publication and presentation rates, an accurate denominator is required that clearly denotes the number of projects that serve as the basis for calculation. Hence, it needs to be clear whether students worked individually or in groups and the total number of projects involved. Using the number of students involved will indicate the proportion of students who presented or published but does not indicate the proportion of projects resulting in presentations or publications if students work in groups of varying sizes. Therefore, to compare rates across programs, the total number of projects is needed.

Given that most students lack hands-on experience with conducting research,⁶ and have little knowledge of how to write a proposal,⁴² curricular support for students during the research process seems especially important. This observation is supported by a survey of pharmacy residents who identified lack of support as one of the barriers to conducting research.⁴³ A report of a pharmacy student research program that implemented a longitudinal pathway with specific objectives related to developing research skills and providing mentoring, demonstrates one method of supporting students. The pathway begins in the first professional year by reviewing a published research report and identifying an idea for a research project. A proposal is written in the second year, an IRB application and data collection and analysis occur in the third year, and presentation at seminar and as a poster in the fourth year. Grading and feedback is provided by both the advisor and the pathway committee and includes peer advising by upper-level students.⁴⁴

Based on information included in the reports, most student research programs appeared to provide some support for the research process but there was no information on the amount and kinds of support provided for developing posters and presentations beyond that provided by project advisors. Given that the terminal course requirement for most programs was a manuscript suitable for publication and/or a presentation, it appears that extramural publication and presentation typically occurred outside a formal course, leaving publication and presentation up to the students and their faculty advisor. The expectation that individual faculty members would provide all the support for students seems to put a heavy workload on the project advisor and does not appear to be an efficient method of mentoring students. One could expect that most students need a considerable amount of basic information (eg, how to structure a research report or what to put on a poster), that could be provided more efficiently through coursework or workshops). Cooley at al found that a 3-part intervention, including a workshop on writing abstracts and making posters, increased the total proportion of students presenting or publishing from 64% to 81%.¹² However, there was no indication in the reviewed literature what level of support is needed for students conducting research and preparing extramural presentations or writing manuscripts for publication.

This systematic review identified only 10 reports of student research programs in the United States (7% of all pharmacy programs) and those reports were limited in the information provided. Hence, this systematic review showed that little is known about student research programs and that further research on student research programs is needed to improve all aspects of programs including how to report on student research, identify outcomes, and define the level of research training required to advance the profession.

Research on instructional strategies for teaching research skills would be helpful. For example, comparing lecture vs workshop formats for providing support to students on writing or on statistical and research design topics would provide useful information for instructors. Student research programs require substantial faculty time and expertise. Research could help clarify whether it is more efficient to have faculty advisors serve as both content experts and mentors on how to structure the proposal, write the sections, analyze the data, write the report, or to have the project advisor serve as the content expert and a separate faculty member provide the basics on how to put the proposal together or analyze the data, and write the report. Research also could help identify the critical components of student research programs so that guidelines or recommended practices be established to assist schools that want to develop student research programs, for example, whether individual or group projects or a combination of both seems to work best. Research also is needed on the outcomes of student research programs. As discussed by Bierer, student research programs can influence student behaviors, patient outcomes, organizational structures or culture, as well as more immediate outcome of student satisfaction.⁶ Not included in the Bierer list was the effect of student research on faculty members. Does student research increase or decrease output by faculty members? Can student research enhance faculty research skills? Research on student research programs could increase understanding of these and other issues.

The ultimate goal of student research programs is to assure a workforce with research training that can use and create the evidence needed for patient care by pharmacists.⁴⁵ However, further research is needed to determine what level of expertise in research is required to advance the profession and whether training should be provided as required coursework or as elective experiences. Fuji and Galt found that schools requiring research also required more rigorous coursework related to research than did schools with elective programs.⁴⁶ Elective programs offered training primarily through elective experiential courses. They also found that about 26% of schools required some research experiences (about 19 out of 79 responding programs),⁴⁶ while Murphy et al identified only 12 schools (15%) that required students to complete a research project from developing a proposal to reporting findings.⁹ Further research is needed to determine whether a research project should be required for all pharmacy students or if offering elective research projects will adequately train the pharmacy workforce.

This systematic review is limited in that it included only reports of student research programs published in English, and it seems that few student research programs publish about their programs. Hence, it is difficult to reach any conclusions about the overall state of pharmacy student research. The systematic review process also assumes that the investigators can reliably interpret the information provided in the reports and that the information in the report accurately represents the student research program. In addition, this review did not include courses or programs that were specifically aimed at quality assurance. Also, if students presented or published outside the formal curriculum, their presentations and publications would not have been included in this review.

CONCLUSION

In the studies of pharmacy student research programs included in this systematic review, required student research programs involved substantially more students and offered a greater variety of topics for research than did elective programs. Most of the student research programs (75%) reported that extramural presentations and/or publications resulted from student research projects. This systematic review also revealed that little is known about student research programs in the United States. This review could serve as a starting place for discussion by pharmacy educators and the profession about: what type of student research training is needed (ie, requirements for research by all US pharmacy students); what types of support from academic and professional organizations are required (eg, special interest groups or taskforces on student research); what types of data on student research should be collected (eg, collecting data on student research on the graduating student survey); and the responsibility of professionals to not just maintain their competence but also improve practice through research.