A Comparison of Pharmacy Students' Confidence and Test Performance

Abstract

Objectives

The objectives of this study were to estimate student retention of knowledge regarding the management of patients with hypertension and dyslipidemia, measure student clinical confidence, and identify the relationship between clinical confidence and actual performance on a knowledge assessment test.

Methods

This was a sequential cross-sectional study to evaluate knowledge retention and clinical confidence of second-year pharmacy students. To measure student clinical confidence, a 12-item clinical confidence questionnaire was administered. To measure student retention of knowledge, a 21-question knowledge assessment test was administered. At least 1 test question was related to each question asked in the clinical confidence questionnaire.

Results

One hundred eight students completed the study. The percentage of students correctly answering test questions decreased from a baseline of 70.4% ± 5.8% to 60.9% ± 5.8% four months later (p = 0.02) in spite of the students rating their clinical confidence from moderate to high in all areas. The proportion of students answering questions correctly was similar across the different levels of confidence.

Conclusion

Overall, retention of knowledge appears to decline over a 4-month period of time. Furthermore, while students perceived moderate to high confidence, student knowledge did not match perceived confidence.

INTRODUCTION

The existence of academic institutions is based, in large part, on the belief that students remember what they learn. While retention of knowledge gained in school is critical, there is a paucity of research regarding the interplay between academic instruction and retention of knowledge.^1,2 Students may pass an examination following an intense study session, but fail to retain much of this material in long-term memory. As described by Popovich, “Information overload does not nurture lifelong learning but actually memory ‘erasing’ or ‘dumping’ after an examination or a course.”³

Faculty members at schools of pharmacy strive to graduate students who are clinically competent and able to manage a variety of medical conditions. While examination performance may suggest that students are prepared to manage patients, it is likely that, in the absence of continual clinical application, student retention of knowledge will decrease over time. Brown et al demonstrated that a smoking cessation education module increased knowledge of pharmacists enrolled in a nontraditional doctor of pharmacy curriculum from a mean pretest score of 36.3% to a mean posttest score of 84.5%. While there was a decrease in mean 1-year posttest scores to 51.6%, the retention of knowledge was higher for students who had the ability to apply their knowledge clinically through patient contact.⁴ Although literature related to the retention of knowledge in healthcare professional students is limited, there is evidence that knowledge retention continues to be problematic in practicing healthcare professionals. Several studies have demonstrated that knowledge levels significantly increased immediately following training and/or continuing education courses, but declined to near baseline within 4-10 weeks.^5–7 Knowledge retention should be a focus of academic institutions, and it should be evaluated to assure the quality of programs. However, data on how to best achieve long-term retention of knowledge is limited and needs to be further studied.

In the context of pharmacy education, students receiving a good examination score may feel confident managing patients in a future clinical setting. However, false or inappropriate confidence, coupled with a lack of knowledge retention, is problematic for academic institutions educating future healthcare practitioners. After all, overestimating clinical confidence has the potential to cause patient harm. Unfortunately, little is known about pharmacy student perceptions of clinical confidence and how this relates to their actual knowledge. After performing searches on Ovid MEDLINE, CINAHL, Cochrane Database of Systematic Reviews (DSR), ACP Journal Club, Database of Abstracts of Reviews of Effects (DARE), Cochrane Central Register of Controlled Trials (CCTR), and PsycINFO using the search terms knowledge retention, retention of knowledge, educational measurement, clinical confidence, clinical competence, pharmacy students, medical students, health occupations students, and higher education, articles related to the correlation of clinical confidence and knowledge retention of pharmacy students and medical students were identified. These studies found a positive relationship between knowledge (as demonstrated by evaluation performance) and confidence.^8,9 Although Messmer et al evaluated methods to enhance knowledge and self-confidence of novice nurses, the authors did not correlate knowledge or knowledge retention with self-confidence.¹⁰ If academicians had sufficient data to illustrate how students' confidence is related to the retention of knowledge and how this relationship affects patient care, then initiatives could be maintained or incorporated to assure patient safety and educational goals.

Two important areas of pharmacy practice are treatment of dyslipidemia and hypertension. Current diagnostic and treatment guidelines are taught to students at the University of Colorado during the fall semester of the second year in the Integrated Organ Systems IV (IOS IV) course. The IOS IV course introduces students to the pathophysiology of hypertension and dyslipidemia and the medicinal chemistry, pharmacology, and pharmacotherapeutics of drugs used to treat these conditions. Given that hypertension and dyslipidemia are so common, this study was designed to focus on these areas for the investigation of retention of knowledge and confidence. The objectives of this study were to (1) estimate student retention of knowledge regarding the management of patients with hypertension and dyslipidemia, (2) measure student confidence when applying their knowledge to manage patients with hypertension and dyslipidemia, and (3) identify any relationship between student perception of clinical confidence and actual performance on a knowledge assessment test.

METHODS

This was a sequential cross-sectional educational research study that was reviewed and approved by the Colorado Multiple Institutional Review Board. Study participants were second-year (P2) pharmacy students who had received formal didactic lectures and examinations on dyslipidemia and hypertension based on the National Cholesterol Educational Program Adult Treatment Panel III (NCEP ATP III) and the Seventh Report of the Joint National Committee on the Prevention, Detection, Evaluation, and Treatment of High Blood Pressure (JNC 7) guidelines.

In January 2004, during the first day of Professional Skills Development IV (PSD IV) class, students were asked to complete a 3-part study set. Part 1 consisted of a written information sheet that described the purpose of the study and provided an opportunity for the students to accept or decline participation in the study. Students were excluded if they did not authorize the use of their data, if the student had transferred from another institution, or if the data set submitted by the student was incomplete. In addition to the written information sheet, students were verbally informed of the study and reassured that data collected for this study maintained anonymity, each student had the option to be excluded from the study, and there would be no repercussions for participating or not participating in the study.

Part 2 of the study set was a confidence questionnaire. This questionnaire assessed confidence in managing patients with hypertension and dyslipidemia. The 12 items included in the questionnaire were based on the application of critical concepts, as summarized in the JNC 7 Quick Reference Card and the NCEP ATP III At-A-Glance: Quick Desk Reference. All items were rated on a 5-point Likert scale and the rating scale was explained to the students. Confidence was rated as follows: 1 = low confidence, 2 = low-moderate confidence, 3 = moderate confidence, 4 = moderate-high confidence, 5 = high confidence. Upon completion of the questionnaire, students received part 3 of the study set, which was a knowledge assessment test. Students were instructed that once the knowledge assessment test had been administered, their responses on the clinical confidence questionnaire should not be modified. Because the clinical confidence questionnaire was administered prior to the knowledge assessment test, it was considered to provide a prospective perception of student confidence.

In part 3 (knowledge assessment test), student retention of knowledge related to the management of hypertension (based on JNC 7 guidelines) and dyslipidemia (based on NCEP ATP III guidelines) taught during the previous semester was evaluated. At least 1 test question was related to each question asked in the clinical confidence questionnaire. The knowledge assessment test consisted of 21 questions. To serve as control questions, 16 of the questions that assessed knowledge of hypertension and dyslipidemia management were obtained directly from the IOS IV examinations administered approximately 4 months earlier. Twelve of the 21 questions were related to hypertension, 7 of which were derived from the previous IOS IV examination and the remaining 5 of which were new questions. The new questions were included to assess student ability to apply critical concepts, as summarized in the JNC 7 Quick Reference Card, to a patient with hypertension. A faculty member, different from the instructor responsible for the IOS IV questions, wrote these new questions. Nevertheless, the appropriateness of the questions and the academic rigor of the questions relative to the IOS IV questions were verified by the IOS IV instructor. The remaining 9 of the 21 questions assessed dyslipidemia, and all were obtained from the previous IOS IV examination. (Contact the corresponding author for further information regarding the clinical confidence or knowledge assessment tests.)

Once the clinical confidence questionnaire and knowledge assessment test had been completed, students stapled all 3 documents (information sheet, clinical confidence questionnaire, and knowledge assessment test) together and placed the document packet on a table away from the instructors. Upon receipt of packets from all students, each packet was reviewed for completeness. All complete study set packets were assigned a study identification number to allow individualized data analysis.

To measure student retention of knowledge related to managing hypertension and dyslipidemia, pooled student performance data on each of the original IOS IV examination questions were compared with the pooled data for the same questions on the knowledge assessment test. The difference in performance for each question was then calculated (ie, IOS IV examination result minus knowledge assessment test result) for the dyslipidemia questions and for the hypertension questions.

Students' perceived confidence for each dyslipidemia and hypertension concept (ie, determining blood pressure goal, recommending drug therapy, identifying cardiovascular risk factors) was analyzed in several ways. First, each questionnaire item was analyzed to determine the frequency of student selection for each confidence level (ie, low = 1; low-moderate = 2; moderate = 3; moderate-high = 4; high = 5). In these analyses, responses for each item were enumerated and expressed as a percentage of the total number of responses for each item. This permitted a visual representation of student confidence. Second, confidence estimates were further analyzed in the context of the question that related to the particular concept of dyslipidemia or hypertension. In these analyses, confidence estimates made by students answering correctly were pooled and compared with confidence estimates made by students who answered incorrectly (ie, mean confidence associated with correct answer versus mean confidence associated with incorrect answer). Third, each assessment test question was analyzed to determine the relationship between knowledge retention and clinical confidence. In these analyses, each assessment test question was analyzed individually to determine the percentage of students who answered the question correctly for each confidence level. This was repeated for all levels of confidence. For each confidence level, the proportion of students answering correctly for each of the dyslipidemia questions were pooled and subjected to statistical analysis. Identical procedures were applied to the hypertension questions. These analyses permitted a direct comparison between the performances of students at different confidence levels.

Knowledge assessment test and clinical confidence questionnaire results for each student were incorporated into a Microsoft Excel database for analysis. To examine knowledge retention, IOS IV examination scores for each dyslipidemia (or hypertension) question were compared with the knowledge assessment scores for the same questions using the Student's paired t test. Student performance on the IOS IV questions on the knowledge assessment test was compared with their performance on the new questions using one-way analysis of variance (ANOVA). To evaluate the relationship between clinical confidence and knowledge retention, confidence levels obtained from students answering the assessment test question correctly for each questionnaire item and each assessment test question were pooled, as were those obtained from students answering incorrectly. The pooled confidence scores from these 2 groups were compared by ANOVA. Similarly, ANOVA was used to compare student performance (ie, correct vs. incorrect) pooled at each level of confidence for all questions. Statistical analyses were conducted using JMP version 5 (SAS Institute Inc, Cary, NC) statistical software. In all analyses, p < 0.05 was considered significant.

RESULTS

One hundred nineteen students met criteria for inclusion. Eleven students were excluded secondary to submission of an incomplete data set resulting in 108 students included in data analysis. In the original IOS IV examinations, the proportion of students answering both the dyslipidemia and hypertension questions correctly was 70.4% ± 5.8%. For the same questions in the knowledge assessment test (administered 4 months after the IOS IV examination), the proportion of students answering correctly decreased to 60.9% ± 5.8%, a reduction of 9.6% ± 3.6 % (p = 0.02). When the dyslipidemia and hypertension questions were considered separately, the proportion of students who correctly answered the dyslipidemia questions in the IOS IV examinations did not differ significantly from the proportion answering correctly in the knowledge assessment (p = 0.09; Table 1). A similar result was obtained for the hypertension questions (p = 0.13). Other, new questions relating to hypertension were included to examine the impact of student familiarity with questions on ability to answer correctly. Overall student performance on these questions was not significantly different from performance on the IOS IV hypertension questions (p = 0.08).

Table 1.

Pharmacy Students' Retention of Knowledge of Management of Hypertension and Dyslipidemia^*

^*Students were required to answer multiple-choice questions in Integrated Organ Systems IV (IOSIV) examinations. Four months later, 9 of the IOSIV questions relating to dyslipidemia and 7 of those relating to hypertension were administered to the same students in the knowledge assessment. The proportion (%) of the class answering each of these questions correctly in the IOSIV examination and in the knowledge assessment test are presented as means from a total of 108 student respondents

^†These are questions that involve several concepts and are considered “higher level” questions requiring analysis, synthesis, and/or evaluation

^‡New questions added to the knowledge assessment test

Analysis of the prospectively administered confidence survey indicated that for the majority of concept areas, students selected moderate to high confidence. Least confidence was expressed for “identifying causes of resistant hypertension” and most confidence was expressed for “determining blood pressure goal” and “recommending lifestyle modification for hypertension” (Figure 1).

Figure 1.

Prospective perception of clinical confidence of students about different aspects of National Cholesterol Educational Program (dyslipidemia) and Joint National Commission (hypertension) guidelines. Using a Likert scale, students were asked to estimate their level of confidence (1 = low confidence to 5 = high confidence) in answering questions relating to different aspects of dyslipidemia (shaded text) and hypertension using National Cholesterol Educational Program (NCEP ATP III) and Joint National Commission (JNC 7) guidelines, respectively. The proportion (percentage) of respondents (n = 108) selecting each level of confidence is shown as closed bar (1, low confidence), light grey bar (2, low-moderate confidence), dark grey bar (3, moderate confidence), cross-hatched bar (4, moderate-high confidence), and open bar (5, high confidence).

Each question in the knowledge assessment test related directly to concepts in the NCEP ATP III (dyslipidemia) or JNC 7 (hypertension) guidelines. Given that each student's level of confidence in answering questions for each concept area had been collected, it was possible to match the level of confidence and whether the student answered the question in that concept area correctly in the knowledge assessment test. Comparing the confidence estimates of students answering each question correctly with those made by students answering the same question incorrectly permitted an estimation of whether confidence translated to knowledge. For all of the dyslipidemia questions, students answering incorrectly were as confident as students answering correctly (Table 2). A similar trend was noticed for the hypertension questions (Table 3), with 2 exceptions. Students correctly answering questions relating to “classifying hypertension” and “determining blood pressure goal” correctly were more confident than students answering the same questions incorrectly (p = 0.017 and p = 0.005, respectively).

Table 2.

Prospective Perception of Clinical Confidence by Students Answering Questions About Dyslipidemia Correctly or Incorrectly

^*Clinical confidence relating to different aspects of dyslipidemia was estimated by students on a Likert scale (1 = low confidence to 5 = high confidence). Students were then required to answer multiple-choice questions about the different aspects (knowledge assessment). The clinical confidence of students answering dyslipidemia questions correctly (correct) or incorrectly (incorrect) were pooled for each questions expressed as mean ± standard error of the mean (SEM) from a total of 108 student respondents

^†percentage of students who selected correct or incorrect answer

^‡probability, compared to clinical confidence in students answering correctly (correct), analysis of variance (ANOVA)

Table 3.

Clinical Confidence of Students Answering Questions About Hypertension Correctly or Incorrectly

^*Clinical confidence relating to different aspects of hypertension was estimated by students on a Likert scale (1 = low confidence to 5 = high confidence). Students were then required to answer multiple-choice questions about the different aspects (knowledge assessment). The clinical confidence of students answering hypertension questions correctly (correct) or incorrectly (incorrect) were pooled for each questions expressed as mean ± standard error of the mean (SEM) from a total of 108 student respondents

^†percentage of students who selected correct or incorrect answer

^‡probability, compared to clinical confidence in students answering correctly (correct), analysis of variance (ANOVA)

To determine whether students expressing the highest level of confidence were more likely to answer questions correctly than those expressing lower levels of confidence, a separate analysis was conducted. This entailed segregating student data into groups based on confidence level and then determining the proportion of students who answered questions correctly. The results of this analysis demonstrated that the proportions of students answering questions correctly were similar across the different levels of confidence in both the dyslipidemia (p = 0.44) and the hypertension (p = 0.92) questions.

DISCUSSION

The results of the present study support the hypothesis that overall student retention of knowledge decreases over a 4-month period of time in second-year pharmacy students when evaluating the hypertension or dyslipidemia sections together. However, there was a not a significant change in knowledge when the 2 sections were analyzed independently. This was likely due to the small sample of questions for each section resulting in insufficient power to detect a difference when analyzed independently. More importantly, independent of statistical significance, had this been an actual examination for the Professional Skills Development IV (PSD IV) course, many students would have experienced a letter grade reduction. Specifically, in the dyslipidemia section, the average scores decreased from 66% to 58%, and in the hypertension section, the average scores decreased from 76% to 66%. In the academic setting, this would have resulted in an increased failure rate for both sections of the examination; hence, these findings are of academic significance. The results of the study also demonstrate that students maintain a relatively high level of confidence managing patient populations with hypertension or dyslipidemia. However, confidence does not appear to correlate with actual performance on a knowledge assessment test.

In the knowledge assessment test, students were asked to answer multiple-choice questions that were identical to those on IOS IV examinations administered 4 months previously. It is conceivable that recall bias may have obscured a decline in student retention of knowledge. To examine this possibility, new questions relating to hypertension were included in the knowledge assessment test. However, the proportion of students correctly answering these new questions was not different from the proportion answering the IOS IV questions correctly, arguing against recall bias impacting the IOS IV interpretation. It is important to note that there was considerable variability in these data that could serve to obscure differences. Whether the same conclusion bears true over a longer period remains to be determined.

Students reported moderate to high confidence in the majority of concept areas. What influences a student's perception of confidence is a source of speculation, although a passing examination score may carry significant weight. Inadvertently, faculty members may perpetuate the notion that successful examination performance illustrates growth in knowledge and objectively support clinical competence.

The results of the present study indicate a disconnect between confidence and knowledge in that student confidence fails to accurately reflect student knowledge. Of the 12 different concept areas assessed (4 for dyslipidemia and 8 for hypertension), only in 2 areas, classifying hypertension and determining blood pressure goal, did students who answered the questions correctly show higher confidence than those students who answered the questions incorrectly. For the other 10 categories, students who answered the questions correctly or incorrectly showed similar confidence. Furthermore, the proportion of students correctly answering questions was similar independent of the level of confidence. For example, the proportion of students with low confidence answering questions correctly was not different from that for students with high confidence.

Second-year pharmacy students are not able to accurately assess their knowledge. These results contrast with the findings of Ytterberg et al who evaluated student confidence and found that confidence correlated to student performance on an objective structured clinical examination (OSCE).⁹ This may be related to the fact that OSCEs are typically a true performance-based examination designed to evaluate student ability in recalling, applying, and physically demonstrating knowledge and skill whereas this written multiple-choice examination primarily evaluated student ability to either recall information or recall and apply information to patient cases. In addition, OSCEs tend to evaluate skill and knowledge on a more general level whereas multiple-choice questions tend to evaluate knowledge on a more specific level. Interestingly, the results from this study also differ from Popovich's study, which concluded that there was a relationship between confidence and performance on a multiple-choice examination.⁸ However, both Ytterberg's and Popovich's studies correlated student confidence in relation to knowledge retention as determined by actual examination performance for a course in which students were aware of the impending examination and likely prepared for it.^8,9 In the present study, student confidence was assessed and the knowledge assessment examination was administered without allowing students time to prepare or review material. Thus, the variance in results between Ytterberg's and Popovich's studies and the present study could result from different examination formats, different education levels of students or, most likely, not allowing students to study for the examination.^8,9 The consequences of an inability to accurately estimate confidence in our pharmacy students are difficult to judge but are certainly problematic. In a situation in which confidence greatly exceeds knowledge and clinical competence, there is concern that students may inadvertently practice beyond their capabilities, potentially leading to adverse patient outcomes.

Limitations

There are some limitations that need to be considered when evaluating this study. First, the relatively low number of IOS IV and new questions limited the power of the knowledge assessment test. The large interquestion variability associated with the proportion of students answering correctly, ranging from 19%-96% for IOS IV questions and 2%-92% for new questions, may have obscured any differences that may have otherwise been apparent had a larger number of questions been asked. The probability value obtained for the comparison of IOS IV and new question scores, P = 0.08, may have reached significance had more questions been used. A similar consideration applies to the evaluation of knowledge retention using the IOS IV questions wherein the difference in scores between the original IOS IV examination and the knowledge assessment test ranged from -20% to +24%. Second, the knowledge retention aspect of this study only applied to a 4-month period. It is important to know what happens to knowledge retention over a longer duration. For example, dyslipidemia and hypertension are revisited in our curriculum in the Comprehensive Patient Care course, some 16 months after the students have completed these topics in the IOS IV course. It remains to be established whether knowledge retention continues to decrease with time or a baseline level of knowledge is retained, leading to a plateau of knowledge retention. Lastly, this study used multiple-choice examination questions to assess knowledge retention, which may not accurately reflect actual knowledge for all students. This examination was not considered to be “high stakes” in that it did not count towards or against student grades, and therefore, some students may not have been motivated to answer questions to the best of their ability. Furthermore, students who experience test anxiety or difficulty with answering multiple-choice questions may actually possess greater knowledge than that reflected in the examination score.

Implications

The important finding of this study is the apparent disconnect between student knowledge and confidence. Our challenge as pharmacy educators, therefore, is to enhance our students' abilities to perceive their confidence more accurately. This could be accomplished by providing more frequent opportunities to recall, apply, and demonstrate their knowledge and skill to clinical situations. Like many pharmacy schools, our curriculum is based on an organ system or disease-specific approach. As such, once the cardiovascular system is examined (in the fall of the P2 year), it is superseded in sequence by IOS courses dealing with pulmonary, immunological, gastrointestinal, neurological, genitourinary/reproductive systems, and finally by infectious disease/virology, leading to a “teach, evaluate, and move-on” approach to pharmacy topics. Our PSD courses integrate with the IOS courses to provide problem-based and active-learning opportunities that permit application of knowledge being attained in the didactic IOS lectures and are temporally linked. Accordingly, the PSD courses also foster the “teach, evaluate, and move-on” approach. Only in Comprehensive Patient Care (in the Spring of the P3 year) are dyslipidemia and hypertension (and other previously taught material) revisited in any significant way. Provision of problem-based learning opportunities that continually integrate previously learned material into active-learning activities (ie, longitudinal patient cases, simulated patients with complex medical histories) throughout the curriculum might be beneficial to enhance knowledge retention. Future evaluation is needed to assess the impact of continued vertical integration of key concepts throughout the curriculum of pharmacy schools as a method of improving student retention of knowledge.

CONCLUSION

Overall, retention of knowledge appears to decline in second-year pharmacy students at the University of Colorado School of Pharmacy over a 4-month period of time. However, this decline in knowledge was not significant when the results from the hypertension section and dyslipidemia section were analyzed individually. While students perceive moderate to high confidence in their ability to manage patients with hypertension or dyslipidemia, student knowledge demonstrated on an impromptu examination does not match their perceived confidence. Education approaches should be developed that promote knowledge retention and make students more accurately aware of their knowledge assets and liabilities. These data will serve as the foundation for further research regarding ways to enhance knowledge retention and methods to enhance knowledge retention and clinical confidence.