Student Knowledge and Confidence in an Elective Clinical Toxicology Course Using Active-Learning Techniques

Abstract

Objective. To measure changes in students’ knowledge and confidence scores after completing an elective clinical toxicology course in an accelerated doctor of pharmacy (PharmD) program.

Design. Various active-learning techniques were used to create a learner-centered environment. Approximately two-thirds of the course used student-led presentations. Some of those not presenting were assigned to be evaluators, responsible for asking the presenter a question or writing quiz questions based on the presented material. Other learner-centered activities included weekly quizzes and discussions at the conclusion of each presented topic.

Assessment. A test instrument designed to measure students’ knowledge and associated level of confidence on each item was administered at the beginning and end of the course. Students’ knowledge and confidence scores increased significantly from pretest to posttest.

Conclusions. Students’ increased confidence and knowledge scores were well correlated after course completion, indicating students were better able to self-assess these areas. These findings suggest that confidence could be an additional measure of students' metacognitive skill development.

INTRODUCTION

Self-assessment is a valuable attribute of healthcare professionals¹ and is recognized by the Accreditation Council for Pharmacy Education as important in helping students assume responsibility for their own learning.² Much of the published medical literature on self-assessment has examined the relationship between global measures of confidence (eg, surveys after an intervention) to performance on written examinations or clinical simulation scenarios.³ Examples in the pharmacy education literature primarily describe self-assessed confidence as 1 of the metrics when evaluating instructional designs, but has been measured globally and by retrospective self-evaluation using survey questions.^4-6 Attempts to correlate performance on assessments and self-reported global measures of confidence in the medical and nursing education literature have varied, with correlations ranging from weak to no association.^3,7,8 One of the reasons postulated for a lack of correlation is that people have a difficult time accurately assigning summary assessments outside the context of the event.¹

A better measure of confidence may be to shift away from summative confidence assessments to a more concurrent approach.¹ There are few reports in the healthcare education literature on the relationship between knowledge and confidence, but several that examined confidence at the question level were identified.^9,10 Ultimately, pharmacists must be self-aware of their knowledge to make the best decisions for patients. Poor alignment between knowledge and confidence may result in poor practitioners who do not use their knowledge when they should (underconfidence) or pose a threat to patients by making errors (overconfidence).^10,11

The purpose of this study was to identify differences in student knowledge and confidence following completion of an elective clinical toxicology course using a multiple-choice assessment instrument and self-reported confidence at the question level. Additionally, this study examined the relationship between knowledge and confidence and distinguish if differences in these metrics were associated with who presented the material (instructor vs student).

DESIGN

The clinical toxicology elective course was first offered in 2010 in the sixth quarter of the accelerated doctor of pharmacy (PharmD) curriculum at South University School of Pharmacy. Starting in 2011, the elective course, which was capped at 20 students in 2011 and 25 students in 2012, was delivered across 2 campuses using synchronized video-conferencing technology. The course was designed to create a shared environment wherein some topics were presented by the instructor and others were presented by students enrolled in the course (Table 1). A clinical toxicology textbook was used to identify foundation toxicological principles and topics wherein drugs were the cause of a toxicological syndrome or pharmacotherapy was a focus in treatment.¹² A cost-effective handbook was the only required text for the course.¹³ The instructor-led topics were 90 minutes in length and were presented using a lecture format and embedded audience-response questions to check for understanding of important concepts. For the student-presented topics, each student chose an available topic and prepared a traditional 20-25 minute electronic slide presentation that covered the source of poisoning, clinical presentation, and treatment. Weekly 10-item multiple-choice quizzes were administered throughout the entire course based on the previous week’s topics and associated readings. When students presented topics, there were additional student responsibilities, including asking questions of the presenter (3 students), writing quiz questions (3 students), and evaluating peers (3 students). In all, there were 10 students actively involved in each student seminar. Given that 2 seminars were presented during each class, nearly every student was responsible for at least 1 activity. To ensure that responsibilities were clearly delineated and equitable, a student activity grid that outlined student responsibilities was created.

Table 1.

Topics in an Elective Clinical Toxicology Course Using Active-Learning Techniques

Students earned points for their assigned roles (ie, writing the quiz, questioning the presenter, or evaluating peers) based on timeliness, completeness, and quality. The presentation grade was based on the collective mean of the student evaluators and assigned by the instructor using a rubric that assessed content and delivery. Additionally, student evaluators were required to detail at least 1 positive aspect of the presentation and 1 constructive comment. These grading rubrics were returned to the presenter within 1-2 weeks of the presentation. A comprehensive final examination was given at the end of the course.

EVALUATION AND ASSESSMENT

The primary objectives were to determine if there were differences in student knowledge and confidence after course completion. To assess these outcomes, a 50-item multiple-choice assessment instrument was created based on the material to be covered in the course. Twenty-five multiple-choice items assessed student knowledge, and 25 assessed self-reported confidence. An electronic classroom management website (eCompanion, version 13.1.3.21, Pearson, Centennial, CO) used to build and administer the assessment. The knowledge questions had 4 possible answers but only 1 best answer.

The same assessment instrument was used for the pre- and post-course tests for both years of the study. The precourse test was completed before any clinical toxicology material had been presented. The postcourse test was completed after all material had been presented, but prior to the final examination. Students were allowed 60 minutes to complete the assessment instrument, which was done outside of class. The goal was to evaluate student knowledge and confidence in a low-stakes environment at baseline and after completion of the course. In order to encourage all to participate, completion of each assessment accounted for a small portion of the final grade. Students who completed the assessment received credit for this portion of the course.

A secondary objective was to determine the level of correlation between assessment scores and self-reported confidence. Mean composite knowledge and confidence scores were calculated for the pre- and post-course test. The level of correlation was determined by comparing the composite scores for knowledge and confidence on the precourse test. Likewise, the level of correlation was determined for the composite knowledge and confidence scores on the postcourse test. The third objective was to stratify and examine data based on instructor-led vs student-led presentation of material. Each assessment question was coded as presented primarily by the instructor or by the student. Twelve of the items were directly linked to instructor-taught material and 13 questions were linked to student seminars. Pre- and post-course test scores were analyzed separately based on who presented the material. Additionally, the relationship between knowledge and confidence was independently analyzed for instructor- or student-presented material.

Raw data from the precourse and postcourse tests were entered into an electronic spreadsheet. Knowledge scores were dichotomized into correct or incorrect responses, and the level of confidence for each question was entered using a number from 0 to 10. To maintain anonymity, each student was assigned a unique research identification number before being entered into the spreadsheet. Mean and standard deviation were calculated based on the number of correct responses for each student’s pre- and post-course tests to examine the distribution of the data with respect to knowledge and confidence. Composite knowledge and confidence scores on the pre- and post-course tests were calculated for the entire cohort. Paired t test was used for pairwise comparisons of knowledge and confidence scores on the pre- and post-course tests. Pearson correlation was applied to assess agreement between knowledge and confidence scores. A value of 0.05 was established a priori to test significance, and the data were analyzed using SPSS, version 21 (SPSS Inc, Chicago, IL). This project was approved by South University’s Investigational Review Board with the requirement of informed consent by participants.

The study period was 2011 to 2012. All 45 students who were eligible provided written informed consent to participate. There were 31 (68.9%) female students. Composite knowledge scores were calculated for each student by numerically summing the number of correct answers and converting them to percentage of correct scores by dividing the number of correct responses by 25 and multiplying the result by 100. Composite confidence scores were calculated by summing all of each individual student’s self-reported confidence for each multiple-choice question. There were 25 questions assessed for confidence, each of which could have a maximum confidence score of 10, yielding a maximum possible score of 250. For ease of data interpretation, these composite scores were converted to percentages by dividing the composite confidence score by 250 and multiplying the result by 100. The score for knowledge increased from 37.5%±12.2% to 68.8%±16.1% between the precourse test and the postcourse test (p<0.001). Likewise, the confidence score increased from 27.9%±15.5% on the precourse test to 67.6%±17.2% on the postcourse test (p<0.001).

The secondary research objective was to determine the level of correlation between assessment scores and self-reported confidence. There was no meaningful correlation between knowledge assessment and confidence on the precourse test; however, the postcourse test evaluation yielded a Pearson correlation coefficient of 0.631 (p<0.001). The third research objective was to stratify and examine data based on instructor-presented material vs student-presented material. When data were stratified, there was significant improvement in both knowledge and confidence scores (Table 2). Similar to the entire cohort, there was no meaningful correlation between knowledge and confidence on the precourse test for instructor-taught material or student-taught material. The postcourse test, however, revealed a positive correlation on both (Pearson correlation=0.459, p=0.002 and 0.672, p<0.01, respectively).

Table 2.

Stratified Knowledge and Confidence Scores Before and After Completion of Elective Clinical Toxicology Course Using Active-Learning Techniques

DISCUSSION

The course was deliberately designed to use a variety of active-learning modalities because of their ability to effectively engage students in activities that foster metacognitive skill development. Active learning is encouraged by the academy and has been shown to be more effective than traditional teaching methods.^2,14 Students in this study demonstrated significant improvement in both knowledge and confidence after completing the elective clinical toxicology course. While the results of this study confirmed what instructors hoped to demonstrate in a well-designed course, the relationship between the 2 variables is important in the teaching and learning process. Before any content was delivered, students performed poorly on the test. Although self-reported confidence overall was also rated low, there was no meaningful relationship between the 2 variables, indicating that students were not able to accurately self-assess. This lack of congruence may be a function of previous knowledge about a topic. For example, in comparing knowledge on a precourse test and question-level confidence in a nonprescription drug course, students with intermediate and high baseline knowledge had stronger correlation coefficients than did those with low baseline knowledge.⁹ Toxicology is an elective course at our school of pharmacy, and based on the knowledge scores, students generally did not have strong baseline knowledge.

At the completion of the course, there was significant improvement in both knowledge and confidence. While the average knowledge score on the postcourse test was disappointing, it was not surprising given that it was comprehensive. Considering that students were not provided any study aids or clues as to the content, the score may have been a more accurate reflection of knowledge acquired throughout the entire course. Confidence also significantly improved, indicating that students were more confident on answers they provided on the multiple-choice questions.

Using self-reported confidence and performance on multiple-choice assessments adds a level of assessment that could be useful to students and instructors alike. Although information could be given back to students after examinations, in the current investigation, score reports were not shared with students for the purpose of maintaining the integrity of the assessment. However, students could compare their performance with self-reported confidence to determine if further study was necessary. For example, a question with a high level of confidence, but poor performance by a student could indicate a misconception. Conversely, a low level of confidence but high achievement by a student suggests a need for reinforcement to ensure complete understanding. Usable knowledge occurs when answers are correct and there is a high degree of self-reported confidence at the question level. At the other extreme, hazardous ignorance occurs when there is a high degree of confidence, yet the answer is incorrect.¹⁰ When there is a lack of intellectual confidence, defined as the ability to act on one’s knowledge and skill, pharmacists may be poor practitioners, but knowledge without confidence may fail to prompt action when necessary, creating hazardous conditions for patients.¹⁵ Ultimately, practitioners will need to accurately self-assess as part of lifelong-learning efforts. After completing the course, students in this study were better able to self-assess, as evidenced by a strong correlation coefficient between knowledge and confidence.

Peer-to-peer presentations have been suggested as an effective instructional method.¹⁶ Given that the course was a shared environment with clear delineation of instructor-presented and student-presented material, a separate analysis was conducted to evaluate student performance on questions linked to both instructional approaches. Independent of who presented material in the class, there was improvement in knowledge and confidence. Additionally, there was strong correlation between the 2 variables on the postcourse test, lending support for student presentations as a method of achieving student engagement in an elective course environment.

Although this evaluation was not designed to measure the effect of active learning, various active-learning strategies were embedded in the course to create a learner-centered environment.¹⁷ The course was effective at improving knowledge, confidence, and the alignment between these measures. To create intellectual confidence, the academy must consider measuring the relationship between these variables beyond summative measures, including question-level confidence. Doing so may require an additional instrument to develop self-awareness and metacognition. Self-awareness is recognized as an important objective of pharmacy education to ensure pharmacists’ continual growth as professionals.¹⁸

There are several limitations to this study. The assessment was conducted outside of class to maximize instruction time. Incentives for academic dishonesty were controlled to as great a degree as possible through verbal instruction to the class that only participation counted toward their grade. The time allowed for completion of the assessment was limited to 60 minutes and was controlled by the classroom management software. Many of the questions required application rather than direct recall, which discouraged use of external resources in a time-limited environment. Had external resources been used, higher scores and perhaps a more narrow range between the precourse and postcourse test scores would have been expected. This analysis did not address whether there was a difference in knowledge and confidence based on the active learning that students were assigned for a given topic. Even though nearly every student had some type of active learning for each class period, there were presentations for which students were merely audience members. Another limitation was the number of questions used. The assessment was meant to be representative of class content but did not comprehensively cover every topic to the same degree. Because the goal was to examine the relationship between confidence and knowledge, a comprehensive and exhaustive assessment was unnecessary, resulting in minimal time requirements outside of class. A further limitation was that the cohort represented a single course over 2 years; however, 100% participation in the study helped to create a complete picture of this population.

SUMMARY

A significant increase in both knowledge and confidence was demonstrated as a result of an elective clinical toxicology course in an accelerated PharmD program. Students demonstrated an improved ability to accurately self-assess at the conclusion of the course. Similar outcomes were demonstrated regardless of whether material was presented by the instructor or through peer presentations. The concept of linking confidence at the question level with competence deserves further exploration within the academy as 1 potential avenue to metacognitive development.

Appendix 1. Example of a knowledge and confidence question

A 72 year-old unresponsive woman is brought to the emergency department from a local nursing home. Her vital signs are as follows: blood pressure 92/45 mm Hg, pulse 108 beats per minute, respiratory rate 6 breaths per minute, oxygen saturation 74%. What is the best initial treatment strategy for this patient?

• A) 1000mL 0.9% sodium chloride intravenously

• B) Norephinephrine infusion intravenously at a rate of 10 mcg/min

• C) Endotracheal intubation

• D) Bag-mask ventilation

• My level of confidence for answering the previous question correct is: ______ (0=no confidence and 10=complete confidence).