A Monograph Assignment as an Integrative Application of Evidence-Based Medicine and Pharmacoeconomic Principles

Abstract

Objective

To describe the development and assessment of monographs as an assignment to incorporate evidence-based medicine (EBM) and pharmacoeconomic principles into a third-year pharmacoeconomic course.

Design

Eight newly FDA-approved drugs were assigned to 16 teams of students, where each drug was assigned to 2 teams. Teams had to research their drug, write a professional monograph, deliver an oral presentation, and answer questions posed by faculty judges. One team was asked to present evidence for inclusion of the drug into a formulary, while another team presented evidence against inclusion.

Assessment

The teams' average score on the written report was 99.1%; on the oral presentation, 92.5%, and on the online quiz given at the end of the presentations, 77%.

Conclusions

Monographs are a successful method of incorporating and integrating learning across different concepts, as well as increasing relevance of pharmacoeconomics in the PharmD curriculum.

INTRODUCTION

Pharmacoeconomics has been taught within a mandatory, lecture-based Pharmacoeconomics and Health Outcomes course at the end of the third year of the doctor of pharmacy (PharmD) program at Western University of Health Sciences for the past 12 years. The 4-credit modular month-long (18 days, 6 hours per day) course covers a review of research methods and statistics and pharmacoepidemiology; an introduction to health outcomes and pharmacoeconomics terminology; varied pharmacoeconomics techniques, including cost-minimization, cost benefit, cost-effectiveness, cost utility analyses, decision modeling, and sensitivity analyses; the 10-14 step approach to literature evaluation of pharmacoeconomic articles; and a module on patient adherence and patient-reported outcomes, including quality of life and patient satisfaction.^1-3

Students traditionally have regarded the pharmacoeconomics course at our institution with anxiety and uncertainty because of its close association with research methods and statistics — an area that strikes fear in students because they feel it has limited connection to clinical practice. Further, although students are educated on the relevance of pharmacoeconomics to formulary decision making by pharmacy and therapeutics (P&T) committees, it is unclear what the processes and applications of pharmacoeconomics are, especially in regard to community pharmacy practice, which attracts more than half of PharmD graduates.⁴ This uncertainty is reinforced by the lack of pharmacoeconomics material on the North American Pharmacist Licensure Examination (NAPLEX) – a factor that changed in 2010.⁵ In addition, students often consider pharmacoeconomics to be an uninteresting topic. Pittenger and colleagues reported that 45% of students surveyed were uninterested in a career in managed care and that students' misunderstanding of the field may develop into negative opinions.⁶

Prior to 2008, active learning in the pharmacoeconomics course consisted of discussion of cases and evaluation of published articles in each area of pharmacoeconomics (cost-minimization, cost-benefit, cost-effectiveness, cost utility analyses, and health-related quality of life). Although they found the course challenging, were engaged in the active-learning exercises, and gave the course faculty good reviews, students often commented that the material was esoteric and they were unable to see its application to pharmacy practice. Those pharmacy faculty members across the country who coordinate and/or teach this course express concern about these same challenges and an interest in methods to increase the relevance of and student interest in the course. In institutions where this course is an elective, the challenges are only compounded by lack of enrollment.

In 2005, the college of pharmacy began incorporating EBM throughout the PharmD curriculum (first through fourth year), through teaching, practicing, and evaluating EBM concepts. EBM is defined as the integration of the best research evidence with clinical expertise and the patient's unique values and circumstances.^7,8 The long-established McMaster EBM model has been adopted at our institution and the process is summarized in the following 5 steps⁸:

• (1) Ask: develop a focused and answerable clinical question using PICO (patient, intervention, comparison, and outcome).

• (2) Acquire: effectively and comprehensively obtain information that answers a clinical question.

• (3) Appraise: critically appraise the evidence for validity, importance, generalizability and applicability.

• (4) Apply: apply the information to a patient case/population of interest.

• (5) Assess: self-assess your ability to practice the above 4 steps and refine your skills and abilities.

Incorporation of EBM across the curriculum requires development of a breadth of activities, from individual patient case scenarios to population-based settings.

Feedback from student pharmacists and preceptors from advanced pharmacy practice experiences (APPEs) about inadequate skills and confidence in monograph preparation prompted the college's curriculum committee to recommend inclusion of a monograph assignment in the didactic curriculum. The required pharmacoeconomics course was considered the optimal venue for introduction and practice of monographs, since the EBM and therapeutic courses taken prior to the pharmacoeconomics course prepare students for this activity.

Inclusion of the monograph assignment in the course presented a challenge for the course coordinator (AVL) in terms of incorporating clinical topic areas into the course, because like many pharmacy administration faculty members, the coordinator was not a practicing pharmacist. However, by collaborating with a clinical faculty member who led teaching of EBM in the curriculum (CJ), the monograph assignment was introduced into the course as an innovative application of both EBM and pharmacoeconomics. Our objective was to develop a monograph assignment and assess student performance across the 2 years following its incorporation.

DESIGN

The purpose of the monograph assignment was to locate, evaluate, and present evidence to support or deny a medication's formulary status for a fictional healthcare plan using evidence-based medicine and pharmacoeconomic methods. The monograph team assignment was introduced to the class prior to the pharmacoeconomics course to give students time to conduct the first EBM task, ie, ask a focused clinical question and acquire the evidence. The preassigned student teams (teams worked together throughout each year of the PharmD program) were asked to assume the role of formulary manager of a hypothetical health plan with 10 million covered lives who must convince the decision maker to either include or exclude a drug into the formulary. Teams had to complete both a written monograph and an oral presentation of the monograph.

The monograph assignment focused on drugs that have recently received US Food and Drug Administration (FDA) approval because: (1) monographs in actual P&T committees are usually prepared to guide decision-making on inclusion of newer drugs into the formulary; (2) our assignment was designed to simulate real-world practice, (3) since less clinical and economic outcome data are available for newer drugs, choosing only recently FDA-approved drugs ensured an equitable volume of available data for each drug, and (4) lack of widely available data forced teams to utilize data efficiently to calculate, estimate, and critically evaluate the evidence and to make decisions under conditions of uncertainty.

Eight drugs approved by the FDA within the previous 5 years were assigned to 16 teams of approximately 8-9 students each, so that each of the 8 drugs was evaluated by 2 teams. One of the 2 teams was required to present evidence for inclusion in the formulary (“for”), while the other team had to present evidence against inclusion in the formulary (“against”). Forcing teams to take a position on formulary inclusion/rejection would illustrate the various ways data could be applied/interpreted and encourage creative thinking for application to actual issues, such as strategy and business models. The pharmacoeconomics course followed completion of a course in pharmacy practice management; thus, this assignment helped to integrate some of the learning from the previous course about budget impact, contracting, and relationships between pharmacy benefit management companies and drug manufacturers.

Written Report

The data required in the written reports ranged from simple, easily available information, such as FDA-approved indications of the drug, the comparator drug of choice, and the gold standard and adverse effect profiles, to information on the pharmacology and mechanism of action of the drug, to more complicated evaluations of evidence of clinical efficacy/effectiveness of the drug from published data. The clinical evidence table required completion of the patient population, intervention, comparisons, and outcomes (PICO) for the study; study duration; and evaluation of the validity of the study design according to the previously taught EBM framework (including relative and absolute risk reduction and number needed to treat or harm, as relevant to the study). Pharmacoeconomic analyses were sometimes available for new FDA-approved drugs; however, a majority of analyses had to be calculated based on published efficacy data and estimated drug acquisition cost data to formulate a budget impact model. In addition, students were required to conduct sensitivity analyses to examine the impact of variability in their cost and efficacy data estimates. Written reports also required identification of the additional information that would be needed to make a better judgment, and questions that needed to be answered before approving the drug. Finally, the approval status recommendation (“for” or “against”) was required, with justification for the particular health plan, taking into consideration patient care, costs, ethical considerations and lack of certainty.

Oral Report

Teams presented their monographs to their classmates and faculty judges over 2 consecutive days at the end of the course. To ensure equity of team members' efforts, half of each team gave the presentation using PowerPoint, then the other half answered questions from the judges. Presentations were limited to 12 minutes; with an additional 3 minutes allowed for a question & answer session with a 4-member panel of faculty judges. To encourage creativity, students were not provided any guidance on how to deliver their presentations; however, the required content had to be presented within the allotted time.

In the second year of the monograph assignment, an online quiz was added to ensure class attendance by non-presenting students and active listening. The quizzes counted 3% of the course grade, and were completed at the end of each day following the presentations. These quizzes were developed by the course facilitator in discussion with the judges to ensure validity of content. Overall, the monograph assignment counted 20% of the course grade (17% was the team grade and 3% was an individual grade).

EVALUATION AND ASSESSMENT

The grading rubric for the monograph assignment used a modified version of the Academy of Managed Care Pharmacy (AMCP) format for formulary submissions version 2.1.⁹ The grading rubric had 16 questions, including a clinical evidence table and economic analysis (Table 1 and Table 2). Students were provided a sample dossier using the AMCP format as a reference, although they were not expected to replicate their report at the same level as the sample. Teams were asked to respond to the rubric questions in a written report due the last week of the course, 4 days prior to the oral presentations. Written reports were graded on a 100-point scale and counted 10% of the overall course grade.

Table 1.

Example of a Clinical Evidence Table for One Study in the Monograph Assignment

Abbreviations: R = randomized, PC = placebo-controlled, P = prospective, DB = double-blinded. Alz = Alzheimer's disease; MMSE=Mini Mental State Examination.

Table 2.

Pharmacoeconomic Section of Monograph Assignment

Teams were provided with the grading rubric that would be used for the oral presentation and a bank of questions the judges might ask. Presentations to the class were judged on content, delivery, and students' responses to questions. Two of the judges were pharmacy practice faculty members who focused on clinical and therapeutic questions, 1 was a pharmacy administration faculty member who focused on pharmacoeconomic questions, and 1 was a practicing pharmacoeconomic clinical specialist who asked reality-based questions. The 4 judges' scores for each team's oral presentation were averaged, and the resulting score counted for 7% of each student's course grade. At the end of the presentations each day, the judges were asked to provide overall feedback to the class and include any tips and pointers to improve the presentations or responses to questions.

The 16 teams in each year (280 students over 2 years) scored an average of 9.9/10 points (Table 3). Their reports were precise, accurate, and comprehensive, yet succinct in content presentation. Formative feedback was provided on the reports for improvement, as needed.

Table 3.

Results of Student Performance in Monograph Assignment Across 2 Years

^a Scores were out of 10 points; there were no quizzes in Year 1

^b Scores were out of 7 points

Oral presentation scores averaged 95% in year 1 and 90% (6.3/7 points) in year 2, with the range of scores varying widely for both years. Quiz scores averaged 77% with a range of 56%-94%. These scores were lower than expected; however, this was the first year that the students had experience with the quiz.

Students were asked to provide comments on the monograph assignment as part of their evaluation of the course. Students appeared to value the assignment and its application. Suggestions for improvement focused on logistical issues such as spacing out and timing of oral presentations, the type of questions asked by judges, and material on the quizzes rather than on completion of the assignment or the time given for it. Students also provided informal feedback indicating that they could finally see the value of pharmacoeconomics. Comments from students on APPEs noted the relevance of monographs in practice. Student feedbacks on quizzes were mixed; they recognized the need for the quizzes but believed that the questions needed improvement.

DISCUSSION

The monograph assignment focused on improving student application of EBM to evaluate the evidence in order to make a decision to include or exclude a drug to the formulary of a hypothetical health plan. It also was intended that students learn to estimate or evaluate pharmacoeconomic data on the drug and its influence on the budget if added to the formulary.

Students performed well overall on the written reports and oral presentations; however, responses to questions following oral presentations indicated the need for improvement in terms of critical and practical thinking. Students appeared to know the drug information thoroughly; but, their ability to apply this knowledge to making decisions for a patient population was weak and unsophisticated/unprofessional. Students were reluctant or uncertain on how to deny the medication under consideration or to suggest alternate possibilities, such as conditional approval. Some improvement may be achieved by adding more practical exercises to the course, but further improvement may be seen only when students gain experience in real-life settings.

Student performance on quizzes was lower than expected; student feedback indicated that they found the quantity of content overwhelming and the questions too specific. Since this was the first year that quizzes were administered following the presentations, students may not have known what to expect and may have been unprepared. Further improvement on the quality of quiz questions should be made; however, the questions must balance testing of application and synthesis that is expected of critical thinking. This is especially true in a field where clinical decisions depend on assessment of the available evidence.

From a curricular perspective, this assignment provided multiple benefits. Students were able to apply EBM and pharmacoeconomic principles to decision making for a population rather than for a single patient. Previous applications of EBM in therapeutic courses were predominantly in comparative evaluation and recommendation of drugs for an individual patient presenting with symptoms.

Students gained experience in synthesizing scarce evidence to formulate an argument, providing a skill set that is useful in diverse practice settings. Students learned the value of appraising evidence when there were no data from head-to-head trials. They also used available resources in calculating budget impact and pharmacoeconomic ratios when there were no pharmacoeconomic studies available for the drug. Further, students learned the importance and applicability of sensitivity analyses when varied information was available for both cost and efficacy.

Students learned that data are objective but decisions have to be balanced using clinical and economic perspectives. Although decisions should be made impartially and objectively , this was not easy and required integrating best evidence with clinical expertise and the unique values and circumstances of a patient population. Some teams were unable to provide a good defense for denying a drug to the formulary. This could be attributed to the patient advocacy component of clinicians who assume that medication should be provided to patients rather than restricted. This philosophy neglects critical components of healthcare that involve economic, social, and humanistic consequences and focus on the patient rather than the population.

The students' presentation of evidence to the faculty judges and classmates simulated P&T committee meetings held in real-world pharmacy practice, and illustrated the importance of stating information succinctly. The assignment highlighted the importance of pharmacoeconomics and EBM to decision-making. An additional benefit was that this assignment helped interested students prepare for the P&T competition that is conducted by AMCP each year.

The authors are considering the following changes to the assignment based on experience and feedback:

• Reviewing and modifying the rubric as needed according to the recent AMCP format version 3.0.¹⁰

• Adding the calculation for Cohen's d effect size to the expected results section for use with continuous outcomes. This topic was added in the P2 year of the EBM curriculum to allow students to calculate a summary measure of effect for continuous data study endpoints, which are common with newly marketed drugs.

• Enhancing evaluation of the quality of studies to be more explicit and possibly graded with levels.

• Ensuring searches for and evaluation of relevant systematic reviews and meta-analysis rather than a sole focus on individual clinical trials.

• Examining student perception of the value of the assignment by a direct survey.

One of the challenges we faced was to establish equity in the amount of work each student contributed to the assignment. We had half of the students from each team present while the other half answered questions. However, some students may not find this equitable due to the amount of stress associated with not knowing the answer to a question. Further research should investigate possible methods for developing equity in terms of workload among team members. Online quizzes allowed us to assess individual understanding of the drugs presented; but did not satisfactorily assess students' synthesis and application of clinical evidence.

CONCLUSIONS

Monographs are an innovative and useful method of integrating learning concepts in the PharmD curriculum. Incorporating monographs increased PharmD students' understanding of the relevance of a pharmacoeconomics course.