An Advanced Pharmacy Practice Experience in Application of Evidence-Based Policy

Abstract

Objective. To determine the impact of an advanced pharmacy practice experience (APPE) to develop skills needed to apply an evidence-based approach to population-level practice decisions.

Design. A 4-week evidence-based medicine APPE was implemented that included active-learning techniques and online learning modules, participation in state drug-policy committee meetings, and completion of an evidence-based medicine review for a specific drug agent or class.

Assessment. Students’ mean score on application of principles related to biostatistics and information mastery on posttests increased 15.8% from pretest to posttest. Students’ mean score on a 22-question information mastery quiz was 90.8%. Mean scores for course evaluation components ranged from 4.8 to 5.0 on a 5-point Likert scale. All respondents indicated they would recommend the APPE to other students.

Conclusions. An APPE that incorporated content from active drug-policy committees increased students’ evidence-based medicine skills and enhanced their understanding of, appreciation for, and confidence in evidence-based practice.

INTRODUCTION

Applying evidence-based medicine principles to patient care decisions ensures the conscientious, explicit, and judicious use of best evidence in practice.¹ Evidence-based medicine is the principle of incorporating information gained from the best available research evidence with clinical expertise and applying it to patient and policy decisions.^2,3 Healthcare providers face the daily challenge of providing the best possible care while controlling cost in today’s environment of direct-to-consumer advertising and high-impact marketing. Therefore, pharmacy students must develop evaluative skills that differentiate therapeutic options with high-yield outcomes measures from those with inferred or surrogate endpoints.

This course addresses curricular outcomes described in the Accreditation Council for Pharmacy Education (ACPE) Accreditation Standards and Guidelines for the Professional Program in Pharmacy under Standard 12, Professional Competencies and Outcome Expectations. ACPE calls for graduates to be competent to provide evidence-based patient-centered and population-based pharmacy care. Evidence-based practice and decision-making are specifically outlined within the clinical sciences content in the pharmacy practice and pharmacist-provided care, and literature evaluation and research design sections for curricular content.⁴ Taking an evidence-based approach to practicing pharmaceutical care is one way to ensure that patients and healthcare providers are satisfied that efforts target a therapeutic outcome that is sensitive to the needs and desires of the patient.

In addition to the widely practiced 5-step approach for pharmacists to provide drug information to patients and other professionals,⁵ information mastery skills aid the health care provider in developing a systematic and rapid approach to identifying the most useful information available for clinical decision-making. An elective course in information mastery increased knowledge and confidence regarding evidence-based medicine. Additionally, students were able to produce higher-quality search strategies and clinical answers than students who did not complete the course.⁶ An evidence-based medicine elective course that incorporated case studies, problem-based learning, and journal club active-learning activities improved perceived skills and confidence in evaluating the medical literature. More than 86% of preceptors surveyed agreed that students who completed the elective course had stronger evidence-based medicine skills than nonparticipants.⁷ An active-learning approach to assessment of landmark trials and application of findings to patient care decisions increased students’ comfort with the technical aspects of using evidence-based medicine skills to evaluate literature in a third-year elective course.⁸

The evidence-based approach to patient care is becoming more widely recognized, and the decision-making process regarding therapeutic choices can be approached similarly with a population perspective such as formulary management of prescription drugs. A few courses described in the literature have targeted students’ development of pharmacoeconomic evaluation and information mastery skills to integrate into an evidence-based approach to patient or population-based care. One such course used a business plan model to evaluate the implementation of clinical pharmacy services. One-third of the course focused on pharmacoeconomics and outcomes research with topics including cost-minimization, cost-benefit, cost-effectiveness, and cost-utility analysis.⁹ Another pharmacoeconomics course used assignments to develop and assess drug monographs in a formulary debate approach as a tool to incorporate evidence-based medicine and pharmacoeconomic principles.¹⁰ Others have highlighted instruction on the utility of new technology that aids in the rapid retrieval and evaluation of medical information.¹¹

In a population-based approach to apply evidence-based medicine, the available evidence does not change, but individual patient characteristics and preferences are replaced by a societal need to provide the most effective care for the entire covered population while controlling expenditures. Applying evidence-based medicine techniques in the development of preferred drug lists or strategic tier placement within a formulary structure may improve the chance that the medications with best therapeutic evidence will be the drugs chosen for placement. This avenue of pharmacy care is providing exciting opportunities for graduates.

With limited focus in the curriculum to foster an evidence-based medicine approach to managed care formulary development, the University of Arkansas for Medical Sciences (UAMS) College of Pharmacy performed a needs assessment to determine students’ preparedness to make evidence-based decisions. After a 2-hour review of evidence-based medicine principles, a posttest was administered to 78 third-year pharmacy students to assess students’ ability to critically appraise the medical literature, including interpreting confidence intervals, statistical significance, and level of evidence. The mean score for the post-assessment was 71.3% with 48 (61.5%) students scoring 70% or better. Based on these results, an APPE was developed to focus on development of skills needed to adopt an evidence-based approach to practice decisions, with emphasis placed on the contributions of a pharmacist toward patient care decisions from a population perspective.

DESIGN

An Evidence-based Medicine APPE designed to teach students the skills needed to retrieve, evaluate, and apply the available evidence in practice was developed and delivered as a 4-week, indirect patient-care experience. The course included a review of statistical assessment and interpretation and organizing a presentation for population decision making. Specific goals for the APPE included: (1) demonstration of an evidence-based approach in the development of clinical judgment, (2) exposure to issues related to managed care, cost containment policies, and reimbursement, (3) development of skills that allow rapid and accurate assessment of evidence, and (4) use of appropriate and effective communication skills. The specific learning objectives for the APPE are outlined in Table 1.

Table 1.

Learning Objectives for an Evidence-based Medicine Advanced Pharmacy Practice Experience

All APPE students had successfully completed a 3-hour drug literature evaluation/biostatistics course in the spring semester of their second year that covered drug information database resources, provided an overview of published clinical trials and review articles, and addressed the principles of statistical evaluation. Students had completed a 3-hour pharmacy administration course in the fall semester of their third year that covered management concepts for managed care organizations and third-party administration. They also completed evidence-based medicine assignments that consisted of 1 drug literature evaluation exercise using a critical appraisal worksheet and 2 clinical questions requiring search strategy documentation and final clinical recommendation. The topics and contact time for this curricular content are outlined in Table 2.

Table 2.

Core Curricular Content Related to Evidence-based Medicine Principles Covered in an Advanced Pharmacy Practice Experience

The evidence-based medicine APPE was designed using an ability-based outcome philosophy. Students participated in real-time, real-world scenarios that required integration of the knowledge, skills, and attitudes necessary to provide evidence-based recommendations for the management of covered populations. Assignments and final projects were identified from the ongoing business of the Arkansas Medicaid Drug Review Committee and the Drug Utilization and Evaluation Clinical Committee of the Arkansas State Employees Employee Benefits Division as the 2 preceptors for this APPE served on these state committees. In addition to the 2 pharmacists, other medical professionals, including physicians and nurses, as well as administrative personnel from the agency and members-at-large from the lay community, served on the committees. Student participation on the committee projects promoted development of a multidisciplinary approach to population-centered decisions. Committee activities on which student assignments and projects were based included recommendation for drug placement within formulary structures for pharmacy benefit plans, development of prior authorization criteria and pathways for specialty drugs, and development of a new drug database to process compilation of evidence and recommendations for agents newly entering the market. Some of these activities required students to develop questions from clinical case scenarios to demonstrate the variety of evaluation criteria that should be addressed in anticipation of the application of a decision-making algorithm to the various clinical encounters that might arise for a population. These questions and search criteria were reviewed with the preceptor. Students then used search engines to identify and retrieve information useful to answering the question. Students were required to have a computer and were provided access to the Internet and all University of Arkansas for Medical Sciences (UAMS) library content, including search engines and databases.

APPE activities included completion of online learning modules and assessments from the Information Mastery curriculum,¹² participation in formal meetings of the 2 state drug-policy committees, and completion of a final project consisting of an evidence-based medicine review of a specific drug agent or drug class to be addressed by one of the aforementioned committees. A scheduled timeline for completion of the required activities was outlined during APPE orientation. The project was assigned on the first day of the APPE, and students were given 1 week to complete a search strategy and evidence selection process. Students completed an online information mastery module on the fifth day of the APPE. Periodic discussions with the preceptor during the project development were designed to review the student’s literature retrieval and assessment progress and afford the student the opportunity to model the presentation of his/her recommendations prior to the final presentation on the last day of the APPE. At these sessions, the students’ question model (ie, the PICO model consisting of the Patient problem or population, Intervention, Comparison, and Outcomes), databases used, literature search strategies, and focus on appropriate patient care outcomes were reviewed. After ensuring that the evidence-based process was progressing appropriately, the preceptor asked the student to compile the available evidence into a recommendation, incorporating efficacy, safety, tolerability, and pricing considerations. One such project charged the student with evaluating the evidence to support the use of a new specialty product, etanercept, within the pharmacy benefit plan. The student developed prior authorization criteria that were presented to and adopted by the state committee. Content for the evidence-based medicine APPE was mapped to UAMS College of Pharmacy Competency Statements (Table 3).

Table 3.

University of Arkansas for Medical Sciences College of Pharmacy Competency Outcome Content for the Evidence-Based Medicine Advanced Pharmacy Practice Experience

The information mastery content of the APPE was based on an evidence-based medicine course created by the Office of Educational Research and Development at Michigan State University. The course consists of 5 modules including an introduction to evidence-based medicine and individualized modules for evaluation of articles about diagnosis, therapy, prognosis, and meta-analysis. Goals of the online course covered evidence-based medicine terms and definitions such as POEMs (patient oriented evidence that matters), DOEs (disease oriented evidence), and information mastery, and outlined the rationale for applying an evidence-based approach to literature review and clinical decisions.¹² The entire course required 10 to 14 hours to complete. Each module has individualized learning objectives that include recognizing drawbacks in commonly used approaches to clinical decision-making; determining study validity; understanding terms such as “intention to treat,” “relative and absolute risk reduction,” and “number needed to treat/harm”; understanding study designs and their utility, including the design of meta-analysis studies; and interpretation of effects models for this design. Supporting articles, practice problems, and self-quizzes are included in each of the learning modules.

EVALUATION AND ASSESSMENT

Ten students completed the evidence-based medicine APPE over a 3-year period. Grades for the APPE were calculated as follows: information mastery quizzes, 20%; search strategy, 20%; assessment skills, 20%; final project presentation, 20%; and professionalism, 20%. The mean score on the information mastery quiz was 90.8%. This quiz was completed on the fifth day of the APPE. A rubric was used to assess performance items including documentation of appropriate search strategies, literature assessment and interpretation, the final project presentation, and professionalism (Table 4).

Table 4.

Evidence-Based Medicine Performance Assessment of Advanced Pharmacy Practice Experience Students, N = 8

During the first 2 years of the APPE, grades were calculated for 8 students using the evidence-based medicine APPE rubric. However, in the final offering of the APPE, the college implemented a new assessment rubric for use in all APPEs. The grades for the 2 students who completed the evidence-based medicine APPE in the third year were calculated using the new assessment rubric; therefore, their scores were not included in the mean grades reported.

During the first year of the course offering, 5 students completed a 10- to 12-question pretest/posttest that consisted of multiple-choice and short answer or calculation questions. The pretest was administered on the first day of the APPE to evaluate students’ baseline knowledge and the posttest was administered on the final day to evaluate students’ acquisition and retention of key concepts related to understanding and application of evidence-based clinical decision-making principles. The concepts evaluated included application of principles of biostatistics (ie, power, p values, intention-to-treat, confidence intervals) and information mastery (ie, level of evidence, resource evaluation, study design, PICO). The mean score on the pretest was 68.5%, compared to a mean score of 84.3% on the posttest, an increase of 15.8%. Pretest scores were not factored into the students’ grades.

The Office of Experiential Education distributed a student course evaluation each month upon completion of APPEs. These evaluations used a 5-item Likert scale (5 = excellent to 1= poor) to rate APPE preceptors and courses and included an open response section with questions on the value of the APPE to professional development and understanding of the patient-oriented role of the licensed pharmacist. De-identified results were returned to preceptors at the conclusion of each academic year. Eight of the 10 students who completed the evidence-based medicine APPE submitted a course evaluation (80% response rate). The average rating was ≥ 4.8 on a 5-point scale (Table 5). All students completing the course evaluation said that they would recommend the APPE to another student.

Table 5.

Evidence-based Medicine Advanced Pharmacy Practice Experience Course Evaluation Components, N = 8^a

DISCUSSION

The principles of evidence-based decision-making and clinical practice are essential components in the health sciences. The integration of these concepts within required and elective course work in the pharmacy curriculum is supported by accreditation and professional standards.^4,13-15 To our knowledge this is the first report of an evidence-based medicine APPE focused on a population-based approach to application of evidence-based medicine principles. At the time of this APPE offering, courses within our curriculum (drug information, management, and therapeutics) included content for development of evidence-based practice skills. That content was designed to impart basic literature evaluation skills and analytical concepts related to decision-making for individual patients or related to practice management concepts for managed care organizations rather than to providing patient-centered care from a population perspective. Despite the inclusion of this content within required course work, a needs assessment demonstrated a deficiency in the concepts required for students to interpret and apply medical literature effectively from a population-based perspective. Comments on the course evaluation confirmed that students did not consider the skills learned and practiced in this APPE to be repetitive of previous course work and recommended these types of activities be extended to all students.

Basing the required components for this APPE within preceptors’ real-time, real-world committee responsibilities created an active-learning environment that facilitated association of conceptual application of concepts to patient-centered pharmacy care decisions. Student evaluations showed that the relevance of this APPE was apparent to students. Students’ comments reflected appreciation that the APPE fostered professional development by increasing understanding of pharmacy practice roles and responsibilities in managed care and by developing skills and confidence for life-long learning related to evidence-based approaches to therapeutic decision-making.

Despite students’ sentiment that the skills developed in this APPE should be required for all students, the opportunity for a specialized practice experience of this type for all students was not practicable. This APPE was delivered by 2 college faculty members with active, full-time clinical practice sites. The service responsibilities of these faculty members to state and university committees are established components of faculty time and effort, so integration of APPE learning opportunities focused on these components of their practice was readily accomplished. However, the categorization of the APPE as an indirect patient care experience limited the number of APPE offerings available to interested students each academic year because the requirement for direct patient care APPE opportunities provided by these faculty members was a lateral consideration. In each instance the indirect patient care evidence-based medicine APPE was offered, a direct patient care Acute Care APPE was forfeited.

The small sample size is a limitation of this study. Also, some nonresponse bias occurred as not all students who completed the APPE returned the course evaluation. In addition, because the course evaluations are returned to preceptors in aggregate at the end of the academic year, the time elapsed between the conclusion of the APPE and completion of the evaluations is indefinite, and it is unknown whether students’ attitudes and confidence were sustained over time.

Based on the results of the college’s needs assessment, feedback from the evidence-based medicine APPE, and findings from the ongoing curricular mapping and assessment of accreditation standards and UAMS competency outcomes, several modifications to the required curriculum have been implemented subsequent to this APPE offering. Current content relating to evidence-based medicine includes 3 required courses. US Health Care System for Pharmacists, a 2-hour course in the fall semester of the first year, is designed to equip pharmacy students with a knowledge base in the areas of pharmacy marketing and health economics that supports good management decisions when providing pharmaceutical products and services for patients and other consumers. Drug Information, a 2-hour course in the spring semester of the first year, introduces students to drug and medical literature with an emphasis on location, evaluation, and communication of medical and drug information, and incorporates use of available library and Internet resources. Evidence-Based Medicine, Biostatistics, & Pharmacoeconomics, a 3-hour course in the fall semester of the third year, is designed to teach students to formulate a focused clinical question and to improve students’ literature critical-appraisal skills, ability to determine the appropriateness of various biostatistical analyses, and application of pharmacoeconomic techniques to inform drug product selection through integration of statistical analysis and study design concepts into practical application. Additionally, the APPE structure has been reorganized to incorporate more specific categorization than direct and indirect patient care experiences. Students now have 3 specialty elective APPE offerings with a managed care focus from which to choose.

CONCLUSION

An APPE that incorporated assignments and projects based in active, real, and relevant state committee business provided an excellent opportunity to enhance students’ understanding and appreciation of the application of evidence-based medicine principles to population-based care. Students’ course evaluations for this APPE served as supporting information in the college’s decision to restructure the delivery of evidence-based medicine principles throughout the core curriculum.