Assessing the Effectiveness of an Evidence-based Practice Pharmacology Course Using the Fresno Test

Abstract

Objective. To assess the effectiveness of an evidence-based practice (EBP) pharmacology elective course to teach EBP skills using the Fresno Test (FT).

Methods. Pharmacy faculty members and medical librarians developed the elective course and offered it to two cohorts of doctor of pharmacy (PharmD) students. A pre/post intervention study design was used. Seven of 12 FT items were chosen to measure specific EBP skills: Ask, Access, Appraise and Apply. Pre/postcomposite and FT item mean scores were compared using Student’s t test with p<0.05 set as significant a priori.

Results. Composite FT mean scores increased significantly for both cohorts. Mean scores for both cohorts increased significantly in four of the seven FT items but on different FT items.

Conclusion. As a profession that commonly uses evidence-based guidelines, developing and integrating an EBP course in the PharmD curriculum is worth considering.

INTRODUCTION

Health professionals are expected to use current best evidence in their clinical practice. Evidence-based practice (EBP) is an approach that integrates three components in clinical decision-making: current best evidence, clinical expertise, and patient perspective. Evidence-based practice requires a different skill set of the clinical practitioner, namely, the ability to identify, access, appraise, and integrate research or scientific evidence into clinical decisions. Development and assessment of EBP knowledge and skills in health professions education, therefore, is necessary to ascertain foundation knowledge in evidence-based approaches to health care. Mastery of EBP can be developed over time and can serve as a strategy to meet health professionals’ life-long learning needs.^1,2

The principles of EBP are taught across health professions programs at the undergraduate and graduate levels, as a component of a course, as a stand-alone course, or integrated throughout the curriculum.³ Educators have experimented with a variety of teaching approaches, including the development of online courses and continuing education classes.⁴ Health professionals need more EBP knowledge and skills than what is taught currently.^5,6 There is also an increased demand to improve the effectiveness of EBP teaching.⁷ Teaching EBP should be clinically integrated to improve not only knowledge but also skills, attitudes, or behavior.³ Teaching EBP using hypothetical patient problems prior to clinical experiences can also be a valid method for students transitioning from didactic to more clinical experiences.⁸

Self-perceived competency in EBP among health professions students is often inflated.⁹ For example, medical students’ beliefs regarding their knowledge and skills related to searching and evaluating the medical literature are exaggerated.¹⁰ Thus, it is critical that competence in EBP be assessed objectively. The Fresno Test (FT) is a widely used and validated instrument for objectively assessing competence in EBP and demonstrating true EBP skills.¹¹ The psychometric properties of the FT have been reported elsewhere.¹¹ Ramos et al developed the FT to primarily evaluate EBP skills among medical students.¹¹ The FT is comprised of 12 items in different formats: short-answer questions, mathematical calculations, and fill-in-the-blank questions relating to two pediatric clinical scenarios. The test is taken in one sitting of approximately 30 minutes. Ramos et al also developed a standardized grading rubric to score EBP skills.¹¹ The FT scoring system is documented in the “Fresno Test of Evidence Based Medicine, Grading Rubrics (Form A)” and the grading rubric standardizes and makes it easier to score the test objectively.¹¹ It includes examples of acceptable answers and specifies four or five grading categories (not evident, minimal and/or limited, strong, excellent), each of which is associated with a point value.

Pharmacists are the medication experts on the health care team and are the leading providers of medication therapy management services.¹² To optimize pharmacotherapy for improved patient outcomes, the profession uses evidence-based guidelines. For this study, an elective course was developed to teach EBP to pharmacy students, specifically the ability to identify, access, appraise, and integrate evidence in the medical literature into hypothetical clinical pharmacological problems. The goal of this study was to assess the effectiveness of the elective course in teaching EBP to pharmacy students as measured by the FT. In this study, we only used seven of the 12 FT items. Items 8-10 are mathematical calculations unrelated to assessment of the four EBP domains of interest. Items 11 and 12 pertain to the identification of study designs appropriate for prognosis and diagnosis not generally taught to pharmacists.¹¹

METHODS

The 2-semester credit hour elective course for PharmD students, Evidence-Based Pharmacology, was developed through collaborations between two pharmacy faculty members and two medical librarians at two institutions (pharmacy and medical schools). Our rationale for incorporating EBP in the context of pharmacology was that pharmacology is an integral and necessary component for determining therapeutic strategies in drug consultations by physicians and other prescribers. The elective course was offered to second-year pharmacy students who had completed two of the four required pharmacology courses in an accelerated PharmD program and were concurrently taking the third required pharmacology course. Two student cohorts, from consecutive entering classes participated in the study.

The elective course focused on developing knowledge and skills for the following EBP domains or steps: (1) Ask: the ability to translate uncertainty and formulate a focused, searchable clinical question; (2) Access: the ability to conduct an efficient literature search and retrieve research evidence; (3) Appraise: the ability to evaluate the relevance and clinical importance of the research evidence; and (4) Apply: the ability to integrate the research evidence with patient perspectives and clinical expertise and apply it to clinical practice. These domains are outlined in Table 1. The course was offered during the 14-week spring semester. The class met two hours each week. The same group of instructors participated in the design and delivery of the two course offerings.

Table 1.

Comparison Fresno Test Results of Cohorts 1 and 2 Used to Measure Evidence-based Practice (EBP) Domain Skills

The structure of the course differed slightly between the two course offerings in terms of number of students, student groups, and case-based clinical pharmacology problems, and the 14-week semester schedule (Table 2). In cohort 2, the number of clinical pharmacology problems was reduced from four to three (biotransformation and pharmacogenomics were merged into one case), freeing up three class sessions for an introductory hands-on instruction on the EBP approach. Over these first three weeks, the students were immersed in the four EBP steps using a workbook as a guide.¹³

Table 2.

Comparison of Cohort 1 and 2 of a 14-week, Evidence-Based Pharmacology Course

Students were randomly assigned to groups of three or four and remained in the same group for the duration of the course. They had three weeks to work on and apply the four EBP steps as they analyzed each hypothetical case-based clinical pharmacology problem (see Appendix 1 for an example). Instructors were readily available to provide guidance and feedback when needed. On the third week designated for each case study, students presented their findings and submitted their EBP worksheets and written search histories. Students were graded as a group and all instructors participated in grading students’ work. Instructors provided written and oral feedback and improvement strategies; librarians gave feedback on search strategies and results.

To further apply and hone their EBP skills, during weeks 13 and 14 of the course, students developed a poster project on a pharmacological case or question of interest to them. Students presented their posters at a forum that simulated a poster session at a scientific conference. Instructors used a rubric to grade the poster project and presentation.

To assess the effectiveness of the elective course in developing pharmacy students’ EBP knowledge and skills, a pre/post intervention study design was used. Seven of the 12 FT items were used to specifically assess pharmacy students’ performance on the four EBP domains. The FT was used because the hypothetical clinical scenarios contained pharmacology-related questions relevant to pharmacy students. The FT was administered in week 1 (preintervention) and week 14 (postintervention) of the course. The MCPHS University Institutional Review Board deemed the study to be exempt from the statutory requirements for review.

Faculty members in pharmacology, pharmacy administration, and medical librarianship scored the students’ FT results using the grading rubric developed for the test.¹¹ If scoring differences occurred, the scorers met and decided on an appropriate score. Pre/postintervention FT mean scores with standard error of mean (SEM) were compared using the Student’s t test for paired samples. SPSS, v21.0 (IBM, Armonk, NY) was used for data analysis. An a priori p value <0.05 was considered significant.

RESULTS

Eleven (cohort 1) and 9 (cohort 2) pharmacy students enrolled in the two offerings of the elective course. Figure 1 shows the pre/postintervention FT mean scores (SEM) for both cohorts, as composite scores. Mean composite FT scores for both cohorts increased significantly (p<0.001). There was a larger percentage increase in cohort 1 [pre=60.1 (6.7); post=95.9 (4.7)] compared to cohort 2 [pre=93.7 (5.2); post=122.0 (4.8)], 59.6% and 30.2%, respectively.

Figure 1.

Pre/Postintervention Fresno Test (FT) Composite Mean Scores. Fresno Test scores for questions 1 through 7 for cohort 1(n=11) and Cohort 2 (n=9). Scores were assessed by three independent raters using the standard FT rubric¹¹. ***p<0.001 pretest vs posttest.

Table 1 shows mean scores by EBP domain or FT item. Following FT guidance, the EBP domain Ask was assessed using FT Question 1 (Q1). The mean scores for Q1 increased for both cohort 1 and cohort 2; however, only the increase in cohort 2 mean scores was significant (p<0.001). The EBP domain Access was assessed using FT Questions 2, 3, and 4 (Q2, Q3, and Q4). For Q2, the mean scores increased significantly for cohort 1 (p<0.05) but not for cohort 2. For Q3, mean scores increased for cohort 1 and decreased for cohort 2; however, neither mean score change was significant. For Q4, the mean scores increased significantly for both cohort 1 (p<0.01) and cohort 2 (p<0.05). The EBP domain Appraise was assessed using FT Questions 5, 6, and 7 (Q5, Q6, and Q7). For Q5, the mean scores increased significantly for both cohort 1 (p<0.01) and cohort 2 (p<0.001). For Q6, the mean scores increased for cohort 1 and cohort 2, but neither mean score change was significant. For Q7, the mean scores increased significantly for both cohort 1 (p<0.001) and cohort 2 (p<0.01). Question 5 was used to measure the EBP domain Apply. The mean score for Q5 increased significantly for both cohort 1 (p<0.01) and cohort 2 (p<0.001).

Cohort 2 had higher pre-intervention mean scores than cohort 1, compositely and by FT item. On further analysis, the preintervention scores for cohort 2 were significantly higher than those of cohort 1 on the following FT items: Q2, Q4, Q5, and Q7 (p<0.0001).

DISCUSSION

We measured the effectiveness of the elective course in developing pharmacy students’ EBP knowledge and skills using seven of 12 items of the FT. Overall, the course seemed to be effective, as indicated by significant increases in mean composite FT scores in both cohorts. The course also seemed to have had an impact on pharmacy students’ knowledge and skills in searching MEDLINE for original research (Q4), determining the relevance of an original research report (Q5), and determining the size and meaning of an effect reported in an original research study (Q7); the mean scores for Q4, Q5, and Q7 increased significantly in both course offerings.

We made changes to the course for cohort 2 based on cohort 1 FT results. Even though there was a significant increase in the mean composite score in cohort 1, when we further analyzed the mean scores by FT item, we found that the cohort 1 course was not as effective in developing students’ knowledge and skills in formulating a focused, searchable clinical question using the population, intervention, comparator/control, outcome (PICO) format (Q1), determining the appropriateness of study designs (Q3), and determining the characteristics that affect a study’s internal validity (Q6). We felt that students would benefit from a more structured instruction on these aspects of the EBP process. Thus, in cohort 2, the number of hypothetical clinical pharmacology problems was reduced from four to three cases, freeing up the first three class sessions for focused hands-on instruction on the EBP skills. Another change was adding a second medical librarian instructor. Both librarians were more involved in teaching, analyzing students’ search strategies, and providing feedback to students. These two changes seemed to have yielded mixed results in cohort 2. Question 1 mean scores increased significantly; the students did better at formulating a focused, searchable clinical question using the PICO format. The magnitude of change in Q1 mean scores was also larger than in cohort 1. For Q3 and Q6 mean scores, there were no significant changes. In fact, Q3 mean scores decreased in cohort 2 but not significantly. Interestingly, the course changes in cohort 2 did not seem to have had a significant effect on students’ awareness of the strengths and weaknesses of common information sources in clinical practice (Q2). We expected otherwise considering that the updated edition of the Glasziou workbook expanded the scope of coverage of “useful sources of evidence” to include sources such as BMJ Clinical Evidence and Embase,¹⁴ in addition to PubMed and MEDLINE. In terms of EBP domain, cohort 2 did not show a significant change in the Access domain, which was measured by Q2, Q3, and Q4; only Q4 mean scores increased significantly.

We found that cohort 2 had higher pre-intervention mean scores than cohort 1, compositely and by EBP domain or FT item. These findings may partly be a result of changes implemented in the PharmD curriculum, specifically the Drug Literature Evaluation and Informatics in Health Care course offered to first-year pharmacy students. More concepts and applications of EBP were incorporated in this course when cohort 2 took it as first-year students. Tilson demonstrated that EBP-trained physical therapy students had significantly higher mean scores than EBP-novice physical therapy students on a 13-item modified FT,¹⁵ thus supporting the notion that previous EBP training improves knowledge and skills.

Our study has a few limitations. We had a small sample of study subjects for both course offerings. We used convenience sampling (second-year pharmacy students voluntarily enrolled in the elective course), so we did not know if they were representative of their respective classes. We also used the FT to assess the effectiveness of the elective course in teaching EBP. However, its grading rubric may not have been suitable to truly assess the EBP domain Access. Question 2 is an open-ended question asking test takers to “name as many possible types or categories of information sources as you can” as well as list strengths, weaknesses, advantages, and disadvantages of each information source. The test also favors a MEDLINE search. Question 3 asks test takers to describe their search strategy if they were to search MEDLINE. A more holistic EBP search strategy is to search Embase and/or Cumulative Index of Nursing and Allied Health Literature (CINAHL) database in addition to MEDLINE. Embase is the most comprehensive database for international biomedical literature;¹⁴ it is a key resource for finding evidence-based pharmacological data. The largest and most in-depth nursing research database,¹⁶ CINAHL captures key concepts from resources unique to this database. Developing a grading rubric that captures the use of a more holistic EBP search strategy and truly assesses the EBP domain Access may be necessary to supplement the FT grading rubric.

CONCLUSION

Development of an elective EBP pharmacology elective course was an effective method in developing PharmD students’ EBP skills. Our findings also support the notion that EBP knowledge and skills can be critically appraised between two distinct cohorts in a PharmD program using the FT as a reliable and validated test for detecting the effect of EBP instruction. Furthermore, to enhance the development of a hierarchy of search strategies, creation of a search skills grading rubric would be more informative to assess the strengths and weaknesses of information sources. While adapted FTs have been developed to assess a comprehensive range of EBP knowledge and skills in novice and trained users in other health professions,^15,17 the FT is still appropriate for use among pharmacy students as pharmacotherapy is a focus of the FT items.

Appendix 1. Example of Case Study and Evidence-based Practice (EBP) Format: Drugs and the Elderly

Mr. McC is an 85-year-old who lives at home with his wife. He had just taken his evening medications and had already gone to bed when he remembered that he had forgotten to let in their dog. When he got up he blacked out and fell. Upon regaining consciousness, he had a sudden surge of pain in his back and thigh. In the ER, the attending physician found a BP of 80/45 mm Hg and a heart rate of 130 bpm. An X-ray indicated an intertrochanteric fracture, and surgery was scheduled. After the surgery, the orthopedic surgeon suggests changing his alpha-blocker medication to Tamsulosin as this will decrease his dizziness and fainting spells. However, the patient is reluctant to change as this new medication may be too expensive for him. The orthopedic surgeon asks you, the pharmacist, for a recommendation.

Medical history: BPH; moderate (AUA score-14) Rx: Finasteride (Proscar) 5 mg po qd; Doxazosin 8 mg po