Improving Pharmacy Calculations Using an Instructional Design Model

Abstract

Objective. To describe an evidence-based instructional design to improve performance and foster retention of pharmacy students’ calculation skills longitudinally across the curriculum.

Methods. Gagne’s nine events of instructional design were employed in a longitudinal pharmacy calculations curriculum. Mean pharmacy calculation examination scores from four courses spanning the didactic curriculum for four different academic years (before and after the redesign) were compared.

Results. Students demonstrated more stable outcomes with consistently higher means after the redesign, which may indicate improved retention. Additionally, the post-redesign classes have experienced fewer failures (score of <80%).

Conclusion. Using an instructional design model to optimize immediate instructional outcomes is an effective method of enhancing retention of calculation skills over time.

INTRODUCTION

Calculations play an integral role within the pharmacy profession and are an essential component of professional degree foundational knowledge [Accreditation Council for Pharmacy Education (ACPE) standards 2016].¹ Developing instruction that results in high levels of calculations proficiency, retention of skills over time, and transfer of skills to authentic settings is important to optimize pharmacist training and patient care. Numerous studies of instructional strategies for calculations are reported in the literature, indicating a need for and interest in effective instructional interventions.^2-10 Although many instructional strategies are effective in promoting immediate recall of learning, studies seldom address retention of learning over time. Gagne’s Conditions of Learning Model aligns external instructional activities with the learner’s internal cognitive processes to maximize learning outcomes, retention, and transfer.¹¹

Brown and colleagues surveyed pharmacy programs in the United States and, of the 72 program respondents, found that 88% teach pharmacy calculations in the first professional (P1) year and half of those teach pharmacy calculations as a standalone course.² Almost all (94%) of the respondents used classroom live lectures to teach pharmacy calculations with an average of 24.8 (+/- 18.1) didactic hours. Only 16% of respondents reported formally testing for retention annually.² The traditional approach of a standalone, single semester, live didactic approach in the P1 year, may not be the best method for ensuring retention. The Center for Advancement of Pharmacy Education (CAPE) 2013 educational outcomes recommended that programs should use “integrated assessments to ensure that students are retaining, integrating, and applying the knowledge, skills, and attitudes.”¹² In addition, ACPE standards 2016 emphasize the importance of assessment to improve the quality of pharmacy education. Standard 10 discusses the curriculum and the "reinforcement of content and the demonstration of competency."¹

With these goals in mind, Western University of Heath Sciences (WesternU) developed a pharmacy calculations curriculum that addresses issues of reinforcement, integration, and retention. The purpose of this paper is to describe an evidence-based instructional design adapted by the program to foster retention of calculations skills longitudinally across the curriculum and to evaluate the effectiveness of the instructional method at improving performance in pharmacy calculations. The study investigators hypothesized that with the implementation of the instructional design, students will demonstrate retention of learning longitudinally and failure rates would decrease over time.

METHODS

At WesternU, pharmacy calculations was previously taught for 10-15 classroom hours by a faculty member in the pharmaceutical sciences department during the second semester of the P1 year. The pharmacy calculations material was integrated into a pharmaceutics course. Passing scores of 80% were required on a series of pharmacy calculations assessments administered during that course. However, the curriculum committee received feedback from faculty members and experiential preceptors that many students did not demonstrate competence in performing pharmacy calculations during subsequent courses.

To remedy this, faculty members used an effective instructional design. Gagne’s Conditions of Learning Model has been identified as one of the most useful instructional design models.¹³ In this model, Gagne proposes nine instructional events that correspond with internal cognitive processes of the learner (Table 1). This systematic approach to instruction promotes meaningful learning, retention, and transfer of learning to appropriate contexts.¹¹ Use of instructional design in medical and nursing education has been described in the literature.¹⁴ Miner and colleagues followed Gagne’s nine instructional events in developing instruction for every lesson in a pre-licensure medical-surgical nursing course.¹⁵ Study of student performance and satisfaction over three semesters indicated improved grades and course evaluations. Students’ ratings of instructor preparedness, instructor mastery, instructor enthusiasm, and instructor effectiveness improved significantly.

Table 1.

How to Apply Gagne’s Nine Events of Instruction for Intellectual Skills¹⁵

In 2010, WesternU changed the pharmacy calculations curriculum to enhance retention by changing the standalone pharmacy calculation course to a longitudinal curriculum. Three review didactic lectures and retention exams were added throughout the second and third professional years (P2 and P3) of the curriculum. Even with these changes, approximately 10% to 20% of the students consistently scored below passing (80% correct). Therefore, in the academic year of 2014-2015, the instruction in the spring P1 year course was revised to a more structured and systematic approach modeled after Gagne’s nine events in instructional design.

An overview of the newly designed longitudinal calculations curriculum at WesternU using Howard Ansel and Shelly Stockton’s Pharmaceutical Calculations textbook can be found in Figure 1 and Table 2.¹⁶ Pharmacy calculations is not currently taught as a standalone course, but accounts for 10% of the grade in each course (Spring P1, Fall P2, Spring P2, and Fall P3). In the P1 year, concepts and material from the textbook were taught didactically. In the P2 and P3 years, a hierarchical approach was developed, revisiting concepts taught in the P1 year using increasingly complex problems. The focus was on the recall and application of prior learning to more difficult problems to aid in retention of new concepts and to increase transfer to appropriate contexts. Following is a description of each event in Gagne’s instructional model with examples from the calculations curriculum.

Figure 1.

WesternU Longitudinal Pharmacy Calculations Curriculum.

Table 2.

Book Chapters from Pharmaceutical Calculations Covered in the Pharmacy Curriculum¹⁶

Event 1: Gain attention of the students. Gaining the attention of learners can be accomplished in many ways, such as with a stimulus change, like a loud noise, or a video or by proposing a thought provoking question. Communicating a startling fact will also work. For example, during the syllabus review students are informed that the passing threshold for the course is 80%. This startling fact is followed by a thought provoking discussion of why 80% may be too lenient. In addition, partial credit is not awarded for incorrect responses. Students are told, if the problem stated: “A physician prescribed amoxicillin 90 mg/kg/day divided in 2 doses for a 2 y/o girl. You have amoxicillin 400 mg/5 mL suspension bottles. How many teaspoons every 12 hours would you tell the mother to give her daughter?” A mother would not care if you knew 1 teaspoon was 5 mL. Although many things in health care and pharmacy are debatable, calculations is not one of them.

Event 2: Inform students of the objectives. Objectives tell the learners what they will be able to do after instruction. Well-stated objectives provide a focus for instruction, guidelines for learning and assessment of learning, and criteria for assessment performance.¹⁷ Learning objectives should focus on student performance, the end-product for the learner, and not the subject matter or instructor’s process.¹¹ Objectives include the behavior, criterion, and condition. For example, “To correctly perform calculations relevant to various drug doses and dosing regimens on a written exam” includes the components of an objective. The behavior is to “perform calculations,” the criterion is “correctly,” which is sometimes implied, and the condition is on “a written exam." Objectives also need to be presented in a way that motivates learners to achieve that objective. When presenting the objectives for calculations, reminding students that a miscalculation in dose or "being off by a decimal point" could result in fatality of a patient, provides motivation for achieving that objective. Pre-instructional design change there were no formal written objectives.

Event 3: Stimulate recall of prior learning. Begin instruction by assisting students in recalling relevant prior learning. The first 5-10 minutes of each calculations class includes 2-3 questions to stimulate recall of related prior material. This foundational knowledge assists with the integration of new learning into existing cognitive structures (ie, schema). Because pharmacy calculations problems build in difficulty and concepts are commonly revisited, this step is particularly important. For example, in chapter 15 of Pharmaceutical Calculations, students learn dilutions of stock solutions.¹⁶ A problem reads, “A pharmacist mixes 100 mL of 38% w/w hydrochloric acid with enough purified water to make 360 mL. If the specific gravity of hydrochloric acid is 1.20, calculate the percentage strength (w/v) of the resulting dilution.”¹⁶ The main concept in the problem is “dilution”; however, recalling concepts learned earlier in the course (specific gravity taught in chapter 5 and concentration percent strength learned in chapter 6) makes the new learning more meaningful and memorable. Pre-instructional design change, there was no formal attempt to develop increasingly complex problem solving abilities by applying prior concepts from previous lessons.

Event 4: Present the content. New information is now presented in the lesson. Deficits in linking the mathematical approach to the practical application in pharmacy practice were identified; therefore, post-instructional design change put a stronger emphasis in this area. For a skill, such as calculations, instruction typically consists of explanation and demonstration. As an example, the concepts intravenous infusions, parenteral admixtures, and rate-of-flow calculations are presented in chapter 13. Describing the process of compounding the admixtures separately from dosing with the use of infusion pumps helps the students differentiate calculations going on in the inpatient pharmacy versus on the inpatient floor where the patient resides. Different instructional strategies support learning at different levels (eg, knowledge, application). Select a strategy that aligns with the desired level of learning outcome.

Event 5: Provide learner guidance. This part of instruction helps the learner encode the new information in a meaningful way. Meaningful learning can be applied in appropriate settings and transferred to other types of examples.¹⁸ Providing numerous examples and non-examples stimulates associations and promotes commitment to long-term memory. A common problem in pharmacy calculations is that students tend to learn how to perform the example in class and have difficulty using it when an appropriate opportunity arises or when the concepts could be applied to another situation/problem. An issue may be that students are trying to memorize specific problems and examples, rather than learning the underlying concepts. Using non-examples allows students to differentiate, when a specific concept should not be applied. Meaningful learning is the difference between understanding and comprehending versus memorizing.¹⁹ To overcome this issue in the calculations course, students are introduced to GUMPS. This acronym stands for: what are my Givens; what is the question asking me for in Units; is there Miscellaneous/extraneous information in the problem; what is the Path to solving this problem (ratio/proportions, dimensional analysis, use of an equation like C1V1=C2V2, etc.); and, lastly, does the answer make Sense.

Another method of facilitating meaningful learning is providing pictures and drawings of the problems. Visualization is an important strategy in meaningful learning. Moreover, visualizing the prescription order and the bottle on the community pharmacy shelf or the IV they are preparing in the inpatient pharmacy may increase the likelihood that the student will transfer that learning to problems of the same type, as opposed to committing a single example to memory. Another instructional support to facilitate transfer of learning is presenting and solving many problems in class. A scaffolding strategy is used in which a common pharmacy problem is broken down stepwise. As we advance through examples of calculations problems, less scaffolding is provided.

Event 6: Elicit performance. “Practice doesn’t necessarily make perfect, but it does make successful transfer and problem solving more likely.”¹⁹ For intellectual skills, such as calculations, the key to mastery is repetition. Students need to practice the skill repeatedly and in many different formats. In our calculations course, students are first encouraged to solve problems from the Pharmaceutical Calculations book.¹⁶ Each chapter has 20-50 problems. P1 students also are provided with online practice questions in the learning management system (LMS) to help simulate the computer-based exams they will take. In addition, students are given a 20-question practice exam in which concepts from multiple chapters are integrated in each problem. In the P2 and P3 years, students complete retention exams. To study for these retention exams, students have access to two practice exams with 20 questions each that reinforce concepts and allow students to practice skills. Prior to the instructional design change, team-based problem sets were used for practice. This was changed to individual practice sets. Students present with large baseline variability in calculations; therefore, team assessments may not be the appropriate platform to elicit performance.

Backward chaining is another strategy to elicit performance. This involves practicing a skill in a backward manner.¹¹ In the calculations course, for example, students perform calculations where a dose/answer that a nurse or pharmacy technician has calculated is given. The student then must use backward chaining to check the nurse’s or technician’s work. For example: A nurse calculated to give the patient 6 mL of acetaminophen and you know that acetaminophen 160 mg/5 mL should be dosed at 10-15 mg/kg/dose for a 30-lb. child. Is the 6 mL dose correct?

Lastly, two non-cumulative quizzes consisting of five questions each are administered in the P1 course and account for 20% of the calculations grade. This opportunity for practice is graded and bridges this performance instructional event and the next event – provide feedback.

Event 7: Provide feedback. Corrective, formative feedback is important when developing new skills or improving existing skills.¹¹ Ideally, feedback should be individualized and immediate; however, with 110-140 students in each of the P1, P2, and P3 classes, this can be difficult logistically. To overcome this challenge, problem sets are provided in the LMS along with solutions and stepwise approaches to solving the problems either on paper or through the use of a video (Doceri Desktop, 2012, San Francisco, CA). Sufficient detail is provided to enable students to follow the problem solution process and improve their performance.

A survey of student perceptions demonstrated that 75% of the P1 class agreed or strongly agreed that the online video tutorials providing formative and corrective feedback were beneficial to their learning.²⁰ A survey of the P2 class, for whom retention of learning is assessed but didactic instruction is not provided, found that 81% of students watched the online tutorials for formative feedback. Of those, 90% found the tutorials to be useful.²¹

If independent practice and the formative assessment feedback from the answer keys and tutorial videos are not sufficient, students can receive more individual and immediate feedback during tutoring sessions and professor office hours. Tutoring sessions became more formal in the academic year of 2014-15 when P4 students and graduate pharmaceutical sciences students provided scheduled tutoring/office hour sessions. Pre-instructional design change feedback was only through low-stake quizzes and team workshops.

Event 8: Assess performance. Summative assessment ensures that information learned is complete and relatively stable.¹¹ Many summative assessments are embedded throughout the longitudinal curriculum. P1 students are formally assessed with a cumulative examination approximately 1-2 weeks after the final didactic lecture. The examination has 20 questions and accounts for 80% of the calculations portion of the course grade. Students are required to pass the examination (not quizzes) with ≥ 80% score. Students who do not pass the calculations examination are given a retake examination. Failure to achieve ≥ 80% on the retake examination will result in a failing grade for the course. A student who fails a course is offered remediation. In the fall and spring of the P2 year and again in the spring of the P3 year, students complete a summative assessment in the integration courses of the curriculum. In each course in the P2 and P3 years, students are provided a 1-hour didactic review, two practice exams with 20 questions each (total of 40), and the two hour, 20-question examination occurring approximately 1 week after the didactic review. This examination is similar in format to the P1 examination and students must pass each retention examination with a score ≥ 80% or they will be given a retake. If they do not pass the retake with ≥ 80%, they must remediate the calculations examination.

Two important changes were made to the cumulative examination over the past several years. In 2010, the college changed from selected response multiple-choice answers to constructed response to mimic the North American Pharmacist Licensure Examination (NAPLEX). In the academic year of 2014-2015, to reflect the computer-based nature of the licensure examinations, the calculations examination was changed to a computer-based examination.

Event 9: Enhance retention and transfer to the pharmacy practice setting. The ultimate goal of instruction is to ensure students retain information and can perform the skill after a period of time has elapsed in an appropriate context.¹¹ In addition, “what is commonly observed is that learners often cannot transfer what is learned in one context or setting to another context or setting, and hence learning is situated in the original learning context.”¹⁸ The transfer of calculations is facilitated by using these concepts in therapeutic blocks, for example, calculating correct weight-based amoxicillin dosing for pediatric populations with acute otitis media to calculating chemotherapy doses using body surface area. Curricular changes included requesting faculty to integrate pharmacy calculations into therapeutic courses. Post-instructional design change also put a strong emphasis on integrating calculations into an objective structured clinical examination (OSCE) with calculating renal clearance and choosing appropriate dosing for correct counseling. Furthermore, retention examinations in the P2 and P3 years provide for summative assessment for calculations learning and retention as described in event 8.

This quality improvement educational study was approved by the WesternU institutional review board. To determine if the instructional design model improved performance in pharmacy calculations, we compared pharmacy calculations examination scores in the P1 fall, P2 fall, P2 spring, and P3 spring. In addition, we identified the number of student failures (score < 80% on the first examination attempt). All students from the classes of 2016-19 were eligible for inclusion in the study. Data were accessed through the Academic Progress Portal (Pomona, CA) used by the college of pharmacy to record grades. Students were excluded from these analyses if they repeated any year to ensure intendance of observations from year to year comparisons of examination scores. Statistical analysis was performed using GraphPad Software QuickCalcs (2016, La Jolla, CA). Categorical variables were analyzed using Chi Squared test and continuous variables were analyzed using student’s unpaired t-test. Two-tailed tests were used and a p value ≤ .05 was considered statistically significant.

RESULTS

Calculations scores were available for the classes of 2016 and 2017 for all four courses. Scores were available for three courses for the class of 2018 and for one course for the class of 2019. When excluding students who did not progress with their original class, the sample consisted of 109, 100, 108, and 109 students from the classes of 2016-2019, respectively. To compare before and after the calculations instructional redesign that occurred in the academic school year of 2014-2015 (for the starting class of 2018), classes of 2016 and 2017 vs. classes of 2018 and 2019 were compared on some of the following analyses.

Mean grades for classes of 2016 and 2017 fluctuated across the curriculum as shown in Figure 2. Means fluctuated up and down ranging from 85% to 91% for the class of 2016 and 83% to 89% for the class of 2017, but demonstrated an upward trend and tighter range of 88% to 91% for the class of 2018. With the implementation of the new instructional design for the P1 students in the spring of the 2014-2015 academic year, performance appeared to stabilize (class of 2018). Before the redesign, the classes of 2016 and 2017 together had a mean (SD) score of 87.1% (12.1) and, after the redesign, the classes of 2018 and 2019 demonstrated significantly higher mean scores of 89.5% (10.8) in the P1 course (Difference =2.3; 95% CI 0.2 to 4.4; p=.033).

Figure 2.

Mean Performance of Exams by Class Across the Longitudinal Curriculum.

To further explore student learning outcomes, the number of students failing the course examination on the first attempt was measured. With the addition of the longitudinal curriculum in 2010, students had increased exposure to calculations in the P2 and P3 years. By the spring of P3 year, the fourth calculations examination, less than 10% of the students failed the examination on the first attempt for the classes of 2016 and 2017 as shown in Figure 3.

Figure 3.

Rate of Failures by Class Across the Longitudinal Curriculum.

Consistent with the findings on mean grades, the percentage of students failing in each course decreased with the new instructional design as shown with the classes of 2018 and 2019 in Figure 3. The rate of failures for the classes of 2016 and 2017 was significantly higher than the rate of failures for the classes of 2018 and 2019 in P1 course (13.9% vs 7.3%; p=.045). Likewise, the rate of failure for the classes of 2016 and 2017 was significantly higher than the rate of failures for the class of 2018 in the fall P2 course (22.5% vs 11.1%; p=.01). For the retention examination in the spring P2 course, although the rates of failure were higher with the classes of 2016 and 2017 compared to the class of 2018, this was not significant (23.9% vs 15.7%; p=.09).

In addition, the number of students in the same cohort, who failed one examination, and failed in subsequent examinations was measured. This analysis assessed cumulative failures throughout the curriculum of four calculations examinations. The number of students experiencing multiple failures is lower than students experiencing a single failure. This demonstrates the 10% to 20% of the class failing each examination (Figure 3), tend to be different students within the same cohort for each examination. Table 3 presents the numbers/percentages of cumulative failures per class/cohort of students. At the time, it is difficult to compare classes since the classes of 2018 and 2019 have incomplete data sets.

Table 3.

Number and Percentage of Multiple Failures by Class Year and Pre- and Post-Instructional Design Change

DISCUSSION

Using a systematic approach to develop instruction allows for the intentional creation of prior learning and structure of teaching strategies in a way that supports learners’ internal cognitive process. The tutorial videos were identified as an effective means to provide more corrective and formative feedback to enhance retention (event 7). Other studies have found positive results when providing multiple opportunities for assessment and feedback. A University of Oklahoma study described a self-paced pharmacy calculations course with examinations of modules once a week.³ Students completed eight modules, one to two per week depending on their individual level of learning. Due to the self-paced nature of the instruction, the 140 students were all on different module examinations during any given week. Examinations were “high stakes” in that students had to pass with a 90% or higher or the examination counted as a missed attempt. The number of missed attempts correlated with student grades (0-1 missed attempt = A, 2-4 missed attempts = B, 4-7 missed attempts= C, 8 missed attempts = F). As compared to previous years with 20/127 student failures, there were no student failures with this instructional format in the first year of implementation. Immediate learning outcomes were better, retention of learning over time was not reported.³

Increased learner guidance and a wider variety of examples were a large part of the P1 redesign of instruction in the academic year of 2014-2015, which was geared to enhancing retention (event 5). Students’ pharmaceutical calculations skills may decline after instruction in the first year. Studies have implied that a “fall-off” in pharmaceutical calculations skills may be occurring in their programs.^5,6 Conway and colleagues, in a descriptive report of the development of their program’s curricular improvement process, identified consistent repetition and assessment of math skills throughout the curriculum as one of the top 5 priorities identified by their faculty.⁵ Hegener and colleagues reassessed pharmaceutical calculations skills 2 weeks and 6 months after completion of a standalone pharmaceutical calculations course in the first year.⁶ Students retained pharmaceutical calculations skills at 6 months following the course, although the mean score of 82.5% was lower than the 90% score required to pass the initial course.

Prior knowledge has been identified as an important influence on student learning.⁷ One challenge with pharmaceutical calculations reassessment as a method of ensuring skill retention and competence is that some students may have inadequate prior mathematical knowledge, or may have factual knowledge, but lack deeper procedural knowledge and the ability to integrate and apply the knowledge to problem solving. To improve transfer of calculations learning to these contexts, calculations in therapeutic blocks and in OSCE (event 9) were incorporated.

Overall, the more recent classes of students are demonstrating lower rates of failure on the calculation examination, on both the first examination in the P1 year when the material is first learned and on the retention summative assessments in the P2 year. Additionally, all classes demonstrated lower rates of failures on the fourth examination in the P3 year, as compared to the first through third examinations. This finding may demonstrate beneficial effects associated with reinforcement of the material throughout the curriculum. Furthermore, the mean scores for the later classes after the instructional redesign have significantly improved by 2.3%. Not only is this statistically significant, however, as this increase changed the average from a B to an A, but it also demonstrates mastery of the subject in the P1 year. For the class of 2018, there was a trend toward more stabilization in mean scores from P1 to P2 year and a slight upward trend, showing there may be overall improvement in calculations performance with the new instructional design. Although, retention may be improving with the new design, it should be noted that the model does require additional resources such as faculty time in the P2 and P3 years to develop multiple examinations and provide office hours. Furthermore, additional study is needed to draw more definite conclusions about the impact of the revised instructional delivery.

Table 3 shows the evaluated cumulative failures and if the same students were failing each examination throughout the curriculum. The 10% to 20% of students who fail each examination are not always the same students as seen by decreasing number of cumulative failures. This demonstrates that students who failed one calculations examination were perhaps learning from their mistakes, becoming more self-aware of their inadequacies, and/or studying more for the next exam. Although the classes of 2018 and 2019 (after the new instructional design) have incomplete data sets at the time of this study, it is expected that with the significant decreases in overall rates of failure (Figure 3), the number of multiple failures will decrease in the classes of 2018 and 2019 compared to 2016 and 2017.

There are limitations to this study. Because the examinations were different from year to year, an inherent difference in examination difficulty may have affected mean scores between classes. Additionally, with the instructional redesign in the academic year of 2014-2015, there was a change in the P1 year pharmacy calculations instruction from a pharmaceutical science faculty member to a pharmacy practice faculty member. Although the same material was covered from the same calculations textbook, concepts may have been emphasized differently. Additionally, although retention examinations in the P2 and P3 years have always been created by pharmacy practice faculty, a different faculty member was responsible for creating exams in the academic year of 2014-2015 and thereafter. Lastly, students’ means scores and rates of failures from different classes were compared, which may bias the results as the classes may have different baseline levels of academic achievement.

CONCLUSION

Accuracy in pharmaceutical calculations requires proficiency in mathematics and the ability to apply these skills in the clinical setting. Using an effective instructional design model to optimize immediate instructional outcomes may enhance retention of calculations skills over time.