Incorporation of Bloom’s Taxonomy into Multiple-Choice Examination Questions for a Pharmacotherapeutics Course

Abstract

Objective. To incorporate Bloom’s taxonomy into multiple-choice examination questions in a pharmacotherapeutics course and assess its effectiveness in detecting areas of improvement in learning.

Design. Bloom’s taxonomy was incorporated into examination questions through a multi-step process: Sample questions representing each learning domain within Bloom’s taxonomy (knowledge, comprehension, application, analysis, synthesis, and evaluation) were introduced to students during lecture presentations and discussions. Quiz and examination containing questions categorized according to Bloom’s taxonomy were administered to students. During review sessions following each quiz or examination, the categorization of each question was provided to students and feedback from students was gathered.

Assessment. The effect of the 5 types of test questions on the correct response fraction and discrimination index was determined after combining synthesis and evaluation. Correct response fractions for knowledge, comprehension, and application questions were significantly higher than those for analysis and synthesis/evaluation questions (p<0.05). However, discrimination index for application and synthesis/evaluation questions were significantly higher than those for knowledge and comprehension questions (p<0.05). In interviews with students who had requested learning assistance, the majority realized the importance of critical-thinking skills in the learning process.

Conclusion. Well-designed multiple-choice questions incorporating different learning domains of Bloom’s taxonomy may be a potential method of assessing critical-thinking skills in large classes of students.

INTRODUCTION

Accreditation standards and guidelines for the doctor of pharmacy degree program support the development of critical thinking throughout the curriculum.¹ The American Psychological Association Delphi Report defines critical thinking as “purposeful, self-regulatory judgment which results in interpretation, analysis, evaluation, and inference, as well as explanation of the evidential, conceptual, methodological, criteriological, or contextual considerations upon that which judgment is based.”²

Several pharmacy educators have measured critical-thinking skills of pharmacy students with standardized instruments such as the California Critical Thinking Skills Test (CCTST)^3-5 and the California Critical Thinking Skills Test Disposition Index (CCTDI).^3,4 Although Philips and colleagues⁴ reported improvement in CCTST and CCTDI scores throughout student academic progression in pharmacy school, Cisneros³ concluded that pharmacy students did not improve their critical-thinking skills during their pharmacy education. However, these measures of critical thinking were designed for students pursuing/receiving a general college education and are not specific for health science education. The apparent lack of student improvement may be partially attributable to the lack of sensitivity in the CCTST and CCTDI for detecting critical-thinking skills associated with pharmacy education. Another potential weakness of these tests is that they solely measure critical-thinking skills and do not train students to apply their therapeutic knowledge.

An innovative method was needed to test critical-thinking skills specific to students’ pharmacotherapeutics knowledge. In addition, this method needed to be linked to examinations and the final course grade in such a manner that it would influence both the materials and the methods by which students study.

Thus, we used Bloom’s taxonomy⁶ to incorporate critical-thinking skills into multiple-choice examinations with questions designed to span the full range of Bloom’s taxonomy categories. Bloom’s taxonomy was suggested by Benjamin Bloom who was dedicated to the study of educational objectives and intellectual behaviors important in pedagogy. He proposed a taxonomy of educational objectives to facilitate communication in order to precisely define and classify vaguely defined terms such as “thinking” and “problem solving.” Although Bloom’s taxonomy identifies 3 domains of learning (cognitive, affective, and psychomotor domain), the cognitive domain is the primary focus of classroom education. As shown in Figure 1, the cognitive domain is hierarchically classified as knowledge, comprehension, application, analysis, synthesis, and evaluation.⁶

Figure 1.

Cognitive Domain of Bloom’s Taxonomy⁶ in Pharmacotherapy

Including essay questions in an examination may be an ideal way to evaluate upper hierarchical cognitive levels of Bloom’s taxonomy that require critical-thinking skills based on knowledge taught in the classroom. However, such examinations can be time and labor intensive when administered in a large class setting and, therefore, may result in delayed feedback to students. Additionally, when multiple graders are involved, there is the potential for inter-rater variability.

The University of the Pacific School of Pharmacy admits approximately 200 students each year. Because the majority of graduate student assistants at the University of the Pacific do not have complete therapeutics knowledge, they cannot assist faculty members with grading essay examinations of pharmacotherapy courses. Thus, grading such an examination would be a tremendous burden on faculty members attempting to evaluate the upper hierarchical cognition associated with the critical-thinking skills of a class of 200 students.

This article: (1) introduces a methods for incorporating Bloom’s taxonomy⁶ and concept maps into examination questions in the General Pharmaceutical Care II course (2) shows the quantitative assessment of the correct response fraction and discrimination index of test questions classified by Bloom’s taxonomy, and (3) briefly describes students’ feedback on the incorporation of Bloom’s taxonomy into examinations during the course.

DESIGN

In this study we applied the principles of Bloom’s taxonomy to multiple-choice examination questions. The examinations were used to directly determine students’ course letter grades for a required therapeutics course, General Pharmaceutical Care II, from 2006 through 2010. The required course is 5 credit units and taught in the sixth semester (14 weeks in a semester) of the 3-year accelerated pharmacy program offered at the University of the Pacific (6 semesters [2 years] of classroom lecture courses, followed by 1 year of advanced pharmacy practice experiences). The sixth semester is the last semester that students spend on campus prior to their advanced pharmacy practice experiential training. The topics covered in the therapeutics course include: oncology, hematology, transplantation, supportive cares, as well as women’s and men’s health.

Incorporation of Bloom’s Taxonomy in the Context of Therapeutics

The cognitive domain of Bloom’s taxonomy has been well-explained and widely cited in many articles^7-11 as well as in Bloom’s original handbook.⁶ However, it was necessary to clarify the definition of each cognitive domain of Bloom’s taxonomy in the context of pharmacotherapeutics test questions for adoption into a required pharmacotherapeutics course. These clarifications helped differentiate test questions into the various Bloom’s taxonomy classification types.

As shown in Figure 1, knowledge is the lowest level in the cognitive hierarchy. It refers to acquisition, recognition, or recall of therapeutic knowledge and information. Examples of keywords used to classify these test questions were: define, describe, list, recall, and select. Although keywords were not found in the test questions, the context of each question was evaluated to classify taxonomy. In the next level in the hierarchical order, Bloom⁶ suggested 3 types of comprehension: translation, interpretation, and extrapolation. In the context of therapeutics, students who comprehend therapeutic knowledge should be able to take the given information and process it into their own language and interpret a given patient case. Examples of keywords were: interpret, estimate, and predict. ⁶

Application requires administering a concept in a new situation or use of abstraction to solve problems.⁶ In a therapeutics context, it is simply interpreted as applying a concept into a patient case as Bloom suggested to present a fictional situation to accomplish the application objectives. Examples of keywords were: apply, modify, prepare, and solve. Analysis refers to the ability to “break down the materials into its constituent parts and detect the relationships of the parts and of the way they are organized.”⁶ Examples of keywords were analyze, differentiate, and separate. As shown in Appendix 1, multiple factors are involved in solving analysis-type problems, as is also true with synthesis and evaluation-type problems. In most practice settings, multiple factors including disease states, drugs, and surgical or radiation procedures are considered to adequately manage patients. An example of an analysis questions (question A) is presented in Appendix 1. As shown in the 3 columns of the concept map for question A in Appendix 1, multiple factors should be considered when deriving the correct answer for question A through the hierarchical thinking process. The clinical manifestations of a patient may be caused by the disease itself (ie, acute promyelocytic leukemia, APML), pathological changes caused by treatment (ie, tumor lysis syndrome, TLS), and/or adverse drug reactions (ie, pulmonary infiltrates). Students should be able to analyze these multiple aspects of clinical manifestations. After students recall the 3 factors mentioned above, hierarchical thinking processes are required as shown in the 4 rows of the concept map, starting with knowledge as the bottom row and analysis as the top row.

Synthesis refers to the ability to put parts together to form a whole, with emphasis on creating a new meaning or structure.⁶ Examples of keywords were create, design, and plan. Synthesis skills in a therapeutics context could be interpreted as applying “multiple factors” to a patient case. An example of a synthesis question (question B) is presented in Appendix. As shown in the 3 columns of the knowledge row of the concept map for question B in Appendix 1, students are required to possess knowledge about pain management, anemia management, and neutropenia management as a foundational step. In the next steps (shown in the comprehension and application rows), students need to comprehend the concepts to enable them to apply each concept to the patient case. Then, they should be able to break down given information and identify multiple problems or issues (shown in the analysis row). Finally, they are required to suggest a therapeutic plan to solve the problems identified (shown in the synthesis row).

Evaluation refers to making judgments about the value of ideas, works, and/or solutions.⁶ Examples of keywords were assess, judge, and explain. Evaluation skills in a therapeutics context could be interpreted to evaluate current therapy or therapy plan, which was synthesized based on multiple factors. An example of an evaluation question (question C) and the related concept map are presented in Appendix 1.

Procedures Used to Incorporate Bloom’s Taxonomy into the Pharmaceutical Course

A stepwise approach was used to introduce Bloom’s taxonomy⁶ to students and integrate it into the course. Bloom’s taxonomy and sample practice questions with written explanations were introduced to students during lecture presentations and discussions. Additionally, frequent quizzes were administered throughout the semester to enhance students’ learning and help them prepare for the midterm and final examinations. The adoption of multiple-choice questions into quizzes allowed more immediate feedback on students’ test performance than if open-essay examination questions had been used. Students’ examination results were easily exported to educational Web-based sites, such as Blackboard (Blackboard incorporation, Washington, DC) or rSmart Sakai CLE (Sakai Foundation, Ann Arbor, MI).

Examination questions were written with explanations and categorization according to Bloom’s taxonomy. When writing the examination questions, concept mapping (Appendix 1) was used to classify each test question into the appropriate Bloom’s taxonomy. Non-faculty pharmacists were asked to peer review the questions to confirm categorization according to Bloom’s taxonomy. Summary descriptions and examples of Bloom’s taxonomy were abstracted from the handbook Taxonomy of Educational Objectives and provided to the peer reviewers. In instances of disagreement, test writers and peer reviewers discussed the question to determine the proper classification. Examinations were then administered to students without knowing in which category of Bloom’s taxonomy the questions were classified. An answer key with written explanations and categorization of questions was provided to students during examination review sessions. These sessions were held during office hours and a formal review session (outside regular class and discussion sessions). The course coordinator further explained questions about Bloom’s classification upon request. These review sessions enabled students to self-evaluate their performance based on Bloom’s taxonomy. In the final step, brief interviews with students were conducted to obtain feedback regarding the testing method and categorization of questions.

EVALUATION AND ASSESSMENT

Assessment of Test Questions Classified by Bloom’s Taxonomy

Six hundred thirty-eight questions were compiled from midterm and final examinations for 5 years, from 2006 to 2010. The questions were classified by pre-assigned Bloom’s taxonomy types. To assess difficulty level and discriminating level, a correct response fraction and a discrimination index (point-biserial correlation coefficient) for each test question were determined by using LXR TEST 6.0 (Logic eXtension Resources, Georgetown, South Carolina) and Scantron (Scantron Corporation, Eagan, MN). The correct response fraction is the fraction of students who answered the question correctly out of the total students who answered the question. The discrimination index is a point-biserial correlation comparing performance on an item to total test performance. The formula to calculate the discrimination index is r=(Mp-Mq)*(p*q)^0.5 *S⁻¹, where r = the discrimination index (point-biserial correlation coefficient); Mp = the test mean for those students who got the item right; Mq = the test mean for those students who got the item wrong, S = the standard deviation for the test scores, p = the fraction of students who got the item right, and q = the fraction of students who got the item wrong. Values can range from +1 to -1. A high discrimination index indicates that high-performing students correctly answered the question, and/or that low-performing students incorrectly answered. Similarly, a low or negative discrimination index indicates that high-performing students incorrectly answered a question, and/or that low-performing students correctly answered a question. Thus, it may suggest a flawed question or low ability to discern high-performing from low-performing students.¹² Correct response fractions and discrimination indexes for multiple-choice examinations are easily gained by using LXR TEST 6.0 and Scantron, compared to essay examinations which are usually manually graded. Thus, correct response fraction and a discrimination index computed through LXR TEST 6.0 and Scantron, may increase utility of multiple-choice examinations. After combining questions classified as synthesis types and evaluation types, a one-way multivariate analysis of variance (MANOVA) was conducted to determine the effect of the 5 types (knowledge, comprehension, application, analysis, and synthesis/evaluation) of test questions on the correct response fraction and discrimination index. Using the Bonferroni method, analyses of variance (ANOVA) on the dependent variables were conducted as follow-up tests to the MANOVA. PASW statistics, version18 (SPSS Inc. Chicago, IL) was used for statistical analysis.

Table 2 contains the mean and standard deviation on the 2 dependent variables (correct response fraction and discrimination index) for the 5 types of test questions (knowledge, comprehension, application, analysis, and synthesis/evaluation). A one-way MANOVA was conducted to determine the effect of the 5 types of test questions on the 2 dependent variables and significant differences were found on the dependent measurements (p<0.001).

Table 2.

The Item Correct Response and the Discrimination Index of Questions^a on a Therapeutics Examination Written Based on the 5 Types of Bloom's Taxonomy

Table 1.

Demographics of Students Participating

Analyses of variance on the dependent variables were conducted as follow-up tests to the MANOVA. Using the Bonferroni method, the ANOVA on the correct response fraction was significant (p<0.001). In addition, the difference found from the ANOVA performed on the discrimination index was also significant (p<0.001).

Post-hoc analyses to the univariate ANOVA for the correct response fraction and the discrimination index consisted of conducting pairwise comparisons. As shown in Figure 2, correct response fraction of knowledge, comprehension, and application was significantly higher than the same metrics for analysis and synthesis/evaluation (p<0.05). This indicates that the level of difficulty of test questions associated with multiple factors was significantly higher than that of test questions associated with a single factor. However, discrimination indices of knowledge and comprehension were significantly lower than those of application and synthesis/evaluation (p<0.05).

Figure 2.

Comparisons on the item correct response and discrimination index for the five types of Bloom's taxonomy of learning. Asterisks indicate a significant difference (p<0.05) between pairwise comparisons over knowledge and comprehension. Double asterisks indicate a significant difference (p<0.05) between pairwise comparisons over knowledge, comprehension, and application.

Description of Student Interviews as Feedback

Students who met all of the following criteria were asked to participate in an interview to obtain feedback related to the adoption of Bloom’s taxonomy: took the course in 2010, participated in a non-mandatory examination review session, and approached the course coordinator to seek advice on how to improve their academic performance or to ask for clarification of lecture materials related to the examinations. Forty students who met these criteria participated in the interview. Eighteen of the 40 (45%) students had midterm grades that were less than 60%.

Thirty-eight students (95%) said that it was helpful to them to identify which type of questions (knowledge, comprehension, application, analysis, and synthesis/evaluation) they most often missed. They also stated that it was helpful to understand the purpose of each examination question. The 18 students whose midterm percentage was less than 60% responded that the examination questions that they missed were scattered throughout all 5 levels of Bloom’s taxonomy. However, 16 (88%) of the 18 students responded that they realized the importance of critical-thinking skills in obtaining therapeutic knowledge.

DISCUSSION

In our assessment, synthesis and evaluation questions were combined before statistical tests were performed for 2 reasons. First, the hierarchical order of synthesis and evaluation could often be reversed. For example, it is difficult to synthesize drug therapy before evaluating current therapy-related problems. A well-accepted revision of Bloom’s taxonomy by Anderson and colleagues¹³ reversed the orders in their taxonomy by using “evaluation” and “creation,” rather than “synthesis” and “evaluation.” In fact, in practical situations, it is difficult to decide which should be higher because evaluation is not only required before synthesizing drug therapy, but also necessary after synthesis to ensure that the new regimen is adequate. Second, the number and variance of synthesis and evaluation questions were unequal. Combining the 2 categories into 1 better satisfied the assumption of MANOVA.

The assessment on the discrimination index of Bloom’s taxonomy question types indicates that application and synthesis questions, which require critical-thinking skills,¹⁴ are able to discern significantly better between high- and low-performing students than can knowledge and comprehension questions (p < 0.05). Thus, well-planned multiple-choice questions may be alternatives for essay examinations to evaluate critical-thinking skills of students in a large class. However, caution is needed to interpret the result of the study because other factors besides critical-thinking skills are also associated with higher domains of Bloom’s taxonomy. For example, a student may have critical-thinking skills, but not possess simple knowledge of multiple topics and therefore not select the correct answer for questions classified as analysis, synthesis, or evaluation. The focus of the present study was to incorporate Bloom’s taxonomy into a therapeutics course to facilitate critical thinking. However, it was not specifically designed to evaluate the relationship between critical-thinking skills in pharmacotherapy and students’ performance on higher domain Bloom’s taxonomy questions, although Bloom’s taxonomy was designed and has been utilized to incorporate critical thinking skills.¹⁴

The assessment of the correct response fractions showed that test questions involving analysis and synthesis/evaluation, which required multiple areas of knowledge were significantly more difficult than questions associated with a single concept. This can easily be understood as students who did not have the requisite were likely unable to derive the correct answer from a question associated with multiple topics. Although we tried to equally distribute topics (eg, cancers, gender health issues, etc) among different types of questions in Bloom taxonomy classification, statistical analysis of the topics was not performed. Thus, this study is not defending the argument that the statistical difference in difficulty levels of analysis and synthesis/evaluation was due in part to different topics.

Application questions significantly improved the discrimination index, compared to knowledge or comprehension questions. However, such questions did not lower the correct response fraction. In other words, application questions significantly improved the ability of the test to discern between high and low performers without significantly affecting the average examination score.

When classifying questions, we focused on “multiple factors” of Bloom’s upper categories (analysis, synthesis, and evaluation), as opposed to the lower categories (knowledge, comprehension, and application). For example, question A (Appendix 1) was classified as an analysis question because students should have considered multiple factors in deriving the correct answer. The clinical manifestations of a patient may be caused by the disease itself, pathological changes caused by treatment, and/or adverse drug reactions. Students should be able to analyze these multiple aspects of clinical manifestations in order to make prudent interventions. If the same question were asked in an essay format, it would be considered an analysis question without major controversy regarding classification. However, the answer choices provided may guide students’ thinking processes to detect the examination writers’ intensions. As such, the example could be argued to be a knowledge question, as it could be viewed as a combination of several independent “knowledge” questions. Similar arguments could be made about multiple-choice questions for “synthesis” and “evaluation.” This factor/aspect of multiple-choice questions, ie, that answer choices may guide students’ thinking processes, could be a limitation of incorporating critical-thinking skills into multiple-choice questions. We tried to minimize controversial classification of questions based on Bloom’s taxonomy by using peer reviewers. The reason we had clinician peer reviewers rather than full-time pharmacotherapy educators was to improve the practical usefulness of test questions. Because they were asked to peer-review questions for content and quality, they were also asked to review Bloom’s classification of questions. In instances where there was disagreement, the authors and peer reviewers reached agreement after discussion.

There are also limitations on the qualitative assessment of incorporating Bloom’s taxonomy into a pharmacotherapeutics course. Student interviews used in this study may have caused biases such as experimenter and/or subject effect, unlike a blinded formal evaluation. However, an overwhelmingly positive response was expressed by students regarding the usefulness of this method of testing. The majority of interviewed students expressed that they realized the importance of critical-thinking skills in obtaining therapeutic knowledge.

The other limitation of the qualitative assessment was selection of interviewed students. Students who were interviewed were selected from students who wanted to seek advice on how to improve their performance or to clarify test materials. Therefore, interview samples were skewed toward low-performing students. However, it is not uncommon for instructors to spend much of their out-of-class time to assist low-performing students with understanding the material so that they can successfully pass a course. During the interview, the majority of low-performing students responded that their incorrect questions were scattered throughout the entire classification scheme used by Bloom’s taxonomy. These students did not have sufficient foundational knowledge to answer basic knowledge questions or higher hierarchical questions. Bloom addressed in his handbook that “knowledge by itself is one of the most common educational objectives.”⁶ The emphasis on critical-thinking skills should not misdirect students from understanding the importance of the acquisition of knowledge. Student feedback suggested that sufficient acquisition of knowledge is necessary prior to applying, analyzing, synthesizing, and/or evaluating patients using critical-thinking skills.

This unique approach to adopting Bloom’s taxonomy drew the interest of university faculty members and ultimately led to presentation of the innovative method at a faculty development workshop. Participants indicated that they had heard of Bloom’s taxonomy, but had not conceptualized Bloom’s taxonomy enough to apply it to their teaching. However, the concept maps (Appendix 1) were helpful to them in sufficiently understanding and applying these concepts to their lectures and to examination questions.

SUMMARY

Adoption of Bloom’s Taxonomy into multiple-choice test questions was implemented in a pharmacotherapeutics course to incorporate critical-thinking skills. Statistical analysis on the correct response fraction and discrimination index of Bloom’s taxonomy (knowledge, comprehension, application, analysis, and synthesis/evaluation) showed that the difficulty level of test questions associated with multiple factors is significantly higher than that for test questions associated with a single factor. In addition, the discrimination index of test questions requiring application skills and thinking ability is higher than that of test questions requiring only memory and understanding. The majority of the interviewed students responded that they realized the importance of critical-thinking skills in obtaining therapeutics knowledge through the innovative application of Bloom’s taxonomy. This suggests that well-planned multiple-choice examinations using Bloom’s taxonomy may be a viable and effective alternative to essay examinations to assess the critical-thinking skills of a large class of students.

Appendix 1. Sample and Concept Map of the Three Types of Midterm and Final Examination Questions in Upper Hierarchical Domain (Analysis, Synthesis, and Evaluation) based on Bloom’s Taxonomy Classification (Correct answers are written in bold.)