A Pharmacoeconomics and Formulary Management Collaborative Project to Teach Decision Analysis Principles

Abstract

Objective. To implement and assess the effectiveness of a 2-course collaborative decision analysis project intended to help students understand the relevance of pharmacoeconomics to clinical pharmacy practice and provide them an opportunity to apply skills taught in pharmacoeconomics to a “real world” problem.

Design. Students were assigned a pair of drugs, 1 commonly used as standard therapy and 1 newly approved, and conducted a decision analysis. The results were then used in a mock pharmacy and therapeutics (P&T) committee meeting.

Assessment. Ninety-eight of 106 (92%) students completed a 4-question survey instrument. Ninety-six percent of students agreed or somewhat agreed that the decision analysis project met the learning objectives. Students felt the shared assignment influenced their choice of formulary drug, augmented understanding of factors influencing decisions, broadened their thinking about drug costs, and was a good approximation of a “real world” application.

Conclusion. An innovative joint-course assignment proved to be a successful technique for teaching decision analysis.

INTRODUCTION

Effective formulary management is necessary because of finite health care resources.¹ Measuring the value of competing treatment strategies or medications is a complex process. Decision analysis is a method for modeling this complexity and quantifying outcomes so that evidence-based, rational decisions can be made.² This methodology is used widely in public health as decision analysis allows comparison of costs and expected gains of 2 competing health interventions.³ When the health interventions are medications, it is easy to identify the cost-effective choice by determining which drug is more effective and less costly than its competitor. It is much more challenging, however, to identify cost-effective therapy when a medication is more effective but also more costly. This is typically the case with branded medications that enter the market. With decision analysis, data from multiple sources can be pooled and synthesized to determine an answer to whether a newly approved medication is cost-effective compared to its competitor.⁵ Researchers quantify probabilities of specific events and outcomes, ie, adverse events, clinical cure, as well as costs associated with these. Doing so allows a mathematical model to be created representing the overall value associated with each option, which can be used to determine the best of course action.⁵ With the current emphasis on rational, evidence-based decision-making in health care and the existence of limited resources, cost-effectiveness analyses based upon decision analyses are becoming more common.

Understanding methods for determining the value of medication therapies is vital to pharmacists as they increasingly assume leadership roles in formulary management and resource allocation.⁴ Therefore, these concepts should be included in the pharmacy curriculum. To address this gap in the curriculum, a decision was made to add a collaborative decision analysis project to an existing course at the University of Cincinnati James L. Winkle College of Pharmacy. The decision analysis project was implemented to expose students to the process of evaluating the total value of a drug rather than just direct medication costs, and to teach them how this process impacts decision making in formulary management. These topics are taught over 3 courses. Foundational concepts on the role of pharmacy and therapeutics (P&T) committees and the role of pharmacists in development of formularies are taught in the second year in Health Systems Pharmacy Practice (HSPP). Application of these concepts occurs in Pharmacy Practice Skills Development, which is a series of performance-based courses. A Formulary Management Module was added to this course to augment information provided in HSPP. Lastly, more advanced topics, such as decision analysis, are taught in the Pharmacoeconomics course.

In 2008, a collaborative decision-analysis project was implemented in the Pharmacoeconomics course and Formulary Management Module of the Pharmacy Practice Skills Development course. The purpose of the decision analysis project was to help students understand the relevance of pharmacoeconomics to clinical pharmacy practice, and provide an opportunity for students to apply skills taught in the Pharmacoeconomics course to a “real world” problem. This joint-course collaborative project was intended to help students achieve Accreditation Council for Pharmacy Education (ACPE) standards (Table 1),⁶ as well as specific terminal outcomes of the Winkle College of Pharmacy PharmD program (Table 2).

Table 1.

2007 Accreditation Council for Pharmacy Education Standards Applicable to a Pharmacoeconomics and Formulary Management Collaborative Project to Teach Decision Analysis Principles

Table 2.

Terminal Outcomes for the James L. Winkle College of Pharmacy PharmD Program

DESIGN

In 2002, a practical application module related to the formulary approval process was developed to provide an opportunity for students to apply what they had learned about formulary management in the HSPP course. The Formulary Management Module was designed for each small group (approximately 6 students per group) to evaluate and compare a new drug that had been requested to be added to the formulary with a drug currently on the formulary. A different drug pair was given to each group. Examples of drug pairs are liraglutide compared to exenatide, and duloxetine compared to venlafaxine. Each group was further divided into 2 subgroups. Each subgroup created a drug monograph: 1 for the new drug and 1 for its competitor. Students completed this part of the module prior to their assigned laboratory day. Additionally, students were assigned a P&T committee role and role-played a P&T committee meeting during the class session (Table 3). The Formulary Management Module preceptors commented there was a need to add a cost measurement component to the module in order to mimic analysis processes used in real-life formulary management. Thus, a joint course decision-analysis project was implemented in the Pharmacoeconomics course and Formulary Management Module of the Pharmacy Practice Skills Development course. The purpose of the decision analysis project was to help students understand the relevance of pharmacoeconomics to clinical pharmacy practice, and provide an opportunity for students to apply skills taught in Pharmacoeconomics to a “real world” problem.

Table 3.

Characters Played by Pharmacy Students in a Formulary Management Role-Play Assignment

The decision analysis project was designed with collaborative and project-based learning andragogical approaches. Students were taught the foundational principles of decision analysis in Pharmacoeconomics. In the class following this lecture, students worked in their skills development-assigned groups on a team-based learning assignment. For this assignment, a decision-analysis model comparing 2 antibiotics,² was created in Microsoft Excel. Although commercial software was available to conduct a decision analysis (2012 TreeAge Software, Inc), it would have required the college to purchase the software and the students to learn to use it. Therefore, faculty members opted to use Microsoft Excel as the necessary calculations could be done by programming the decision analysis formulas for the various pharmacoeconomic outcomes (eg, incremental cost-effectiveness ratio) into an Excel spreadsheet.

In the team-based learning exercise using this example, students were asked a series of questions that led them through the base case and sensitivity analyses. Because students worked together in groups, they facilitated each other’s learning. In-depth discussions resulted from the team-based learning approach which deepened understanding as students expressed opinions and provided a rationale for their answers. The small group workshop was meant to familiarize students with the Excel spreadsheets and a decision analysis model which they would then use to complete their own small-group decision analysis projects.

The students worked in groups outside of the Pharmacoeconomics class to complete their decision analysis projects on the assigned drug pairs. They then integrated their findings from the decision analysis project into their formulary monograph in the Formulary Management Module. Based on all of the information gathered about the drug pair, each group made a decision on whether to add the requested drug to the formulary.

The entire process (the lectures in multiple courses, the small group workshop, and the decision analysis project) guided students through the cognitive domain of Bloom’s Taxonomy of Learning.⁷ Students gained knowledge and demonstrated comprehension by examination of formulary management concepts in the Health Systems Pharmacy Practice course and decision analysis concepts in the Pharmacoeconomics course. They then applied those pharmacoeconomic principles in a structured manner to the question “Which drug is the most cost-effective option?” They synthesized data from multiple sources and used decision analysis techniques to investigate the options. Finally, they evaluated the results of the decision-analysis model, concluded which was the appropriate formulary choice, and attempted to justify that decision in a role-play of a P&T committee meeting. The learning objectives for the joint course assignment can be found in Table 4.

Table 4.

Learning Objectives for the Formulary Management Module and the Decision Analysis Project

In the Formulary Management Module, the students were responsible for being able to discuss: the role of a P&T committee and the formulary management process of hospitals and pharmacy benefits managers, the 2 articles about the drugs and their independently chosen article about their assigned drug, and the decision analysis project and how research such as this facilitates formulary decisions. Their assignments were the decision analysis project, drug monographs of both medications, a Powerpoint presentation incorporating the drug monograph information to be given to the P&T committee, and participation in the P&T role play.

In the Formulary Management Module, the preceptors were responsible for facilitating a discussion on P&T committees and formulary management, a discussion on the decision analysis project, and the P&T role-play. Additionally, preceptors were responsible for grading the Formulary Management Module.

In the decision analysis project, the students were responsible for participating in the team-based learning assignment using the decision analysis model from their textbook and Excel spreadsheets, scheduling time outside of class to meet with their group members to determine which medication was cost-effective relative to the other by applying the costs and probabilities to the decision-analysis Excel spreadsheets, and conducting sensitivity analyses. They also were responsible for creating a written report that outlined methods for obtaining input data, justified their decision for the most cost-effective drug, and explained their interpretation of the incremental cost-effectiveness ratio and the incremental net benefit (Appendix 1). Additionally, they submitted a single Excel file with the base cases analysis of the 2 assigned drugs on the first tab and subsequent multiple tabs containing each sensitivity analysis.

In the decision analysis project, the Pharmacoeconomics course director was responsible for updating drug pairs to reflect current formulary discussions in clinical practice, teaching foundational principles of pharmacoeconomics, and facilitating the team-based learning discussion on the textbook example of a decision analysis and sensitivity analyses of 2 antibiotics. She was also responsible for overseeing each group’s decision analysis project and providing guidance when asked on specific issues with specific drugs and for grading decision analysis projects according to the project rubric.

To facilitate this shared assignment, the course coordinator for Pharmacoeconomics was appointed the coordinator for the Formulary Management Module. With a class size of approximately 100 students and 6-7 students in each group, this module had to be taught 15 times using 15 different drug pairs. The Formulary Management Module was taught by a combination of full-time and adjunct faculty members. Adjunct faculty members were typically clinical pharmacists at local institutions whose job responsibilities included drug policy development or pharmacy benefits management. Based on their current job responsibilities and career paths, these individuals were uniquely qualified to teach this module. Instructors were assigned groups based on which drug pairs they were most familiar with already. Additionally, first-time instructors were offered an opportunity to shadow another group preceptor prior to facilitating their first Formulary Management Module so they could become familiar with the flow of the module and how the students needed to perform.

Notably, a significant amount of faculty time was involved in identifying 15 different drug pairs and ensuring that enough literature existed on the drugs for students to fulfill the requirements of the module. Drug pairs were updated by evaluating medications newly approved by the Food and Drug Administration (FDA) and competitor drugs already on the market. This pre-work was accomplished well in advance of the module and took approximately 3 to 4 days.

ASSESSMENT AND EVALUATION

Project Grades

Using IRB approval for course quality improvement initiatives, the project was evaluated to ensure it was meeting our curricular goals. The Formulary Management Module was a competency-based module; thus, students were required to achieve all of the objectives listed to receive a pass grade for the module. The objectives were assessed by preceptors for the Formulary Management Module.

Assessment of the decision analysis project has evolved since its implementation in 2008 and in response to student comments. Initially, the project was worth 10% of the Pharmacoeconomics course grade. Because of the amount of time they spent gathering information to conduct the decision analysis, students felt the project should be worth more of the overall Pharmacoeconomics course grade. Consequently, beginning in 2009, the project comprised 20% of the overall grade. Additionally, in 2011, a grading rubric was designed to ensure consistent evaluation from group to group and to educate students on the expectations for their final product. Using the grading rubric in 2011, the average score for the class on the decision analysis project was 95% ± 4% (range 86-100). In 2010, before the rubric was created, the class average was 98% ± 5% (range 93-100). Though the grades decreased with use of the rubric, the instructors believe the rubric allowed more meaningful and objective assessment of performance.

Student Survey

In addition to student performance on the assignment, assessment of how well the shared assignment facilitated learning occurred via several mechanisms: a student survey, comments on the course evaluation, and preceptor evaluation. A survey of students’ perceptions was administered to determine how well the joint assignment met learning goals by providing an approximation of a real-world application. The survey instrument was pilot tested on 4 fourth-year PharmD students who indicated it was straightforward, easy to understand, and appropriately assessed student perceptions. The survey instrument was then administered to all students after completion of the final examination in the Pharmacoeconomics course. Ninety-eight of 106 (92%) students completed the survey instrument.

The majority (92%) of respondents felt the decision analysis project was useful in making a formulary choice during their skills development Formulary Management Module. Ninety-three percent also felt that completing the decision analysis exercise augmented understanding of factors influencing formulary management decisions. Though this exercise was not a real-world application of the skills learned, 94% of students felt that it was a good approximation of how decision analysis would be used in practice. Finally, 95% of students felt that the decision analysis project broadened their thinking about what constitutes “drug cost.”

Student Course Evaluations

A course evaluation also captured subjective evaluative information about the decision analysis project. Forty-eight of 106 (45%) students responded to the course evaluation survey instrument and 28 commented on the question, “What was an effective teaching method the faculty member used?” Twelve of the 28 responses were positive comments about the decision analysis project. Students commented that the project helped with comprehension of pharmacoeconomics and prepared them for decisions they would have to deal with in their professional lives. Overall, they felt the joint-course assignment prepared them to participate on P&T committees. Three comments suggested improvements for the project. The main concerns were ensuring there was adequate time for all groups to complete the project and addressing challenges that students had working in groups in which other members did not fulfill their responsibilities. Overall, student responses were positive, and from their perspective, using the work from the decision analysis project in the Formulary Management Module accomplished the learning objectives.

Preceptor Evaluation

Preceptor assessment was also solicited. Adjunct faculty members who taught the Formulary Management Module were asked to provide their perception of the decision analysis project as part of a quality improvement process. They commented that the decision analysis project: intensified the level of student engagement in the formulary management process, deepened understanding of how decision analysis principles are likely to be encountered in pharmacy practice, and demonstrated the need to incorporate safety and efficacy into cost decisions. Preceptors who taught in the module prior to initiation of the shared assignment felt that, through completion of this project, students gained respect for drug policy and pharmacoeconomic principles.

DISCUSSION

This project provides a unique approach to meeting specific student-learning outcomes. ACPE Standards 2007 Guideline 15.1 states a college’s evaluation of student learning should “use teaching and learning techniques that promote: knowledge base development; integration, application, and assessment of principles; critical thinking and problem solving and professionalism.” This integrated joint-course assignment provides an opportunity for students to further their knowledge based in pharmacoeconomics by using evidence to develop a monograph and evaluate the cost effectiveness of 2 drugs. Students integrate the information to make a determination about formulary approval and demonstrate characteristics of a professional when role-playing the P&T Committee. The purposes of this shared assignment were to: (1) help students understand the relevance of pharmacoeconomics to clinical pharmacy practice and (2) provide an opportunity for students to apply skills taught in Pharmacoeconomics to a “real world” problem.

Typically, pharmacy courses and their associated assignments are developed and taught independently without any collaboration among course instructors. Although many colleges, including ours, have concepts integrated into a course (eg, pharmacotherapeutics with pharmacology, medicinal chemistry, etc), this joint-course assignment is innovative because it requires application of foundational principles learned in 2 courses (HSPP and Pharmacoeconomics) to an authentic performance assessment (eg, formulary process approval) in another course. In pharmacy curricula, health systems concepts and practice management issues are often perceived by students as “less valuable” than other curricular areas (eg, therapeutics). However, this crossover assignment stimulated student interest and demonstrated applicability of information to real-life scenarios.

Not uncommon with initial implementation of a new assignment, the first-year revealed/exposed some areas for improvement. Initially, students received a minimal amount of guidance on the decision analysis project. After receiving student feedback, changes were made to improve the process. Class time was used to review the directions for the project and more explicit criteria were given. Since these changes have been implemented, the process has flowed more smoothly.

In the first year of the project, faculty members in the Pharmacoeconomics course explained to students how to use the Excel spreadsheets. The faculty members quickly recognized that the lecture format was ineffective in teaching the skills necessary to navigate the spreadsheet, and this ultimately resulted in students not being able to adequately interpret the results of the base case and sensitivity models of the paired drugs. In 2010, instead of lecturing on the Excel spreadsheets, student groups were given a single spreadsheet of a one-way sensitivity analysis. The instructor reviewed the base case spreadsheet of a one-way sensitivity analysis and then each group reported on their Excel spreadsheet of a sensitivity analysis constructed from the original base case. This technique was more effective than lecturing because it involved active learning. Each group was asked the same question: “What changed and how did it affect the model?” However, the session quickly became tedious and students lost interest. In 2011, a team-based learning assignment was developed. Working in their Pharmacy Practice Skills Development course groups, students were given a packet that contained the base case scenario, all of the sensitivity analyses in the Excel workbook, and directed questions for each scenario. This was a high-energy activity, during which students were engaged in relevant discussions with one another that resulted in deeper learning. The students gave excellent feedback on this group activity in course evaluations so the workshop will be continued.

As most students had never attended a P&T committee meeting, they noted in course evaluations that they were unsure of how to perform the role play. As a quality improvement measure, the Formulary Management Module in the Pharmacy Practice Skills Development course was videotaped during the P&T committee role plays of the groups from the second year. The 2 most accurate representations of a P&T committee meeting were uploaded to the Blackboard course management system so that students in subsequent years could review them for guidance on how their role-play should be conducted. As all preceptors for the module are involved in P&T committees, they facilitate the role play. Prior to the role-play, the preceptors generally remind students about the purpose of P&T committees and then discuss their individual roles in the role-play and what the concerns and perceptions of someone in that role might be. This process works well as the students generally engage in lively discussions and enjoy the role-play activity.

Selecting appropriate drug pairs for the decision analysis project presents more challenges than simply identifying appropriate literature. While 2 drugs may have the same mechanism of action, their outcomes may have been measured differently in clinical trials and this is difficult to deal with in a decision analysis. For example, the outcome for one insomnia medication might have been “number of minutes till sleep” and the other “number of nighttime awakenings.” Another challenge is selecting medications whose outcomes measures are easily converted to probabilities of success, for example “walking time to first episode of angina.” Because the drug pairs treat a wide variety of disease states, there are differences in the length of time that patients could be treated. This project is simplified in that all groups evaluate parameters no longer than 1 year so that the time burden for project preparation is similar for all groups. Nonetheless, it is recognized that a shortened timeframe may not accurately reflect the cost-effectiveness of therapies used to treat chronic conditions, and this observation is highlighted during the lecture on Markov modeling.

As the project evolved, faculty members recognized there was a need to develop an assessment rubric. The purpose of the rubric is twofold. First, it allows for objective assessment of the project. Second, it helps students understand what is expected of them. After using the rubric once, faculty members determined that it needs to be improved. The weighting system will be changed so that students are not penalized harshly for minor imperfections and more details will be included under the steps to guide students on expectations.

Group management skills are also an important part of this project and something with which students often struggle. To encourage equal participation from all students in the group on the project, a peer grading system is being developed based on the same system that students use in other team-based learning activities within the college.

SUMMARY

Formulary management strategies are an important part of pharmacy practice today. Therefore, as outlined by ACPE standards, pharmacy students must be able to apply pharmacoeconomic principles and to participate in, or at least understand, the formulary management process to ensure their patients are receiving the most appropriate medications, taking into account safety, efficacy, and cost. Using a joint-course decision analysis assignment, students put themselves in the roles of P&T committee members and used complex analysis processes, as would be used in real-life, to determine the most cost-effective medication between 2 choices. Active learning through the team-based assignment and role-playing, as well as collaborative and problem-based learning from the project, created a rich educational environment that augmented students’ understanding of pharmacoeconomic principles in clinical practice.

Appendix 1. Decision Analysis Report and Excel Spreadsheet of Paliperidone versus Risperidone

Formulary Management- Decision Analysis

The direct drug cost for the usual adult dose for the first FDA indication listed for paliperidone and risperidone was schizophrenia¹.

Usual adult dose

Paliperidone(Invega): 6mg/day

Risperidone(Risperdal): 4mg/day

Drug cost for one inpatient treatment was approximately 21 days and the estimated readmission was 3 times a year.²

Paliperidone: For 100 tablets of 6mg it cost $1790.00, so for 1 tablet it cost about $17.90 and for a length of treatment of 21 days and 3 readmissions it will be $1127 for one year.¹

Risperidone: For 60 tablets of 4mg it cost $400, so 1 tablet cost $6.66 and for 21 day length of treatment and 3 times readmission it will be $420 for one year. We assumed they were at maintenance dosing and thus once daily dosing is sufficient.¹

***Although, the data above is the actual price of the drugs.¹ The wholesale charge of risperidone is $14.46 for 60 tablets, thus 1 tablet costs $0.241 and for a 21 day length hospital stay and 3 readmissions the total yearly cost is $15.20.³

The wholesale charge of paliperidone is $470 for 30 tablets thus $15.66 for one tablet and $987 for one year.³ The base case is based off this information.

Estimates of clinical success and clinical failure.

Paliperidone success rate was about 53%, success was defined as a ≥30% reduction in the PANSS score.⁴ Risperidone’s success was 60% for a ≥20% reduction in the PANSS score.⁵ These numbers were converted to a related proportion to make comparable. Thus paliperidone’s success was 62% and risperidone success is 60% based on data given in clinical trials.

Estimated cost to treat EPS: Give Benztropine 2mg IM x7 days inpatient = $1.80 and propranolol 30mg/day for 7 days inpatient= $2.94.¹ The adverse drug reaction would on average increase the hospital stay by 7 days and thus the direct cost per day by $500, so $3,500 for one week. ² So total $3,505 for adverse events per hospital stay.

The probability of having an adverse event of EPS for paliperidone is 10% for 6mg tablets versus 12% for risperidone 4mg tablets. ^6,7

Interpretation of Decision Analysis and Sensitivity Analysis

The decision analysis indicates that risperidone is less expensive than paliperidone in treating schizophrenia. The probability of clinical success with paliperidone is 2% higher than risperidone (62% vs 60%) and the probability of an ADE is 2% lower with paliperidone (10% vs 12%).

The incremental cost for each additional clinical success with paliperidone would be $45,085 per extra success. Paliperidone is both more effective and more expensive, thus the value of an extra success must be judged relative to the value of clinical success for schizophrenia. Although, the incremental cost effectiveness ratio is close to the accepted maximum value, of $50,000, per additional success. The incremental net benefit for paliperidone and risperidone over the range of $1000-$2000 was -$882 and -$862. Since the incremental net benefit was negative that means paliperidone’s success is not worth the extra cost compared to risperidone.

The sensitivity analysis suggests that paliperidone is more expensive than risperidone under all conditions including: increase and decrease of 25% for each drug’s cost, clinical success, probability of an ADE, and the cost of an ADE. The analysis is insensitive for these conditions and gives the base case confidence in its original findings. The incremental net benefit remains negative for all sensitivity analysis concluding the extra success of paliperidone is not worth the cost. The incremental cost effectiveness ratio is sensitive (it becomes negative) when the success of paliperidone is decreased by 25% and the success of risperidone is increased by 25%, thus risperidone is dominating paliperidone under these conditions.