Abstract
Objective
To describe and evaluate the transition of a pharmacy residency and fellowship (PRF) elective course to a hybrid online platform.
Innovation
In 2016, the 1-credit hour PRF elective was transitioned from a live, synchronous course to a hybrid online platform. Over the course of the semester, students completed eight modules along with assignments that pertained to a different component of PRF. Course grades and evaluations, as well as PRF placement rates, were compared between 2015 (live, synchronous course) and 2016 (hybrid online course). There were no differences in overall course grades or student evaluations of individual relevant course objectives between the two course formats. However, more students rated the course as excellent during the 2015 live, synchronous course. Placement rates were similar between students who took the course in 2015 and 2016.
Critical Analysis
Following the transition of a PRF elective to a hybrid online platform, course grades, evaluation of individual relevant course objectives, and PRF placement rates remained similar to previous years. Creative educational venues can help meet the student demand while simultaneously allowing faculty to manage their time. However, instructors should balance this with desire of students to have more face-to-face in class time.
Keywords: professional elective, online learning, teaching, post-graduate education, residency training
DESCRIPTION OF THE PROBLEM
In the mid-2000s, multiple national pharmacy organizations issued position statements recommending requirement of post-graduate year one (PGY1) residency for all pharmacists involved in direct patient care by 2020.1, 2 With a national match rate of 67%, colleges and schools of pharmacy nationwide have worked to increase their students’ success rates in securing pharmacy residencies or fellowships (PRF).3 The method most often described in the literature is elective coursework, with reported benefits including enhanced perception of knowledge and confidence in PRF and increased number of students obtaining PRF positions.4, 5
At our institution, a PRF elective course was first implemented in 2013. Studies have demonstrated student satisfaction with online course formats, which tailors the learning environment to millennials.6, 7 Additionally, faculty have reported that although organization and preparation for online coursework is time-consuming, the upfront preparation reduces time needed to facilitate the course during the semester.8 Therefore, the PRF course was transitioned to a hybrid online format in 2016 with goals of increasing flexibility and engagement for students while simultaneously decreasing workload for faculty.
THE INNOVATION
Pursuit of Post-Graduate Opportunities is a 1-credit hour elective course offered for second (P2) and third (P3) year pharmacy students, which is designed to develop skills and tools needed for successful completion of the PRF application and interview process. Traditionally, the course design consisted of weekly, hour-long lectures that were facilitated by a designated instructor. Students completed assignments or reflections following each class period, along with longitudinal assignments, culminating in a final “application packet.” This process gave students the opportunity to reflect, incorporate feedback into their curriculum vitae (CV) and letter of intent (LOI), and meet the learning objectives, which included: 1) identifying opportunities prior to graduation to increase marketability, 2) explaining postgraduate employment opportunities including, but not limited to: residencies, fellowship, industry, hospital, and community, 3) strengthening oral and written communication skills necessary to attain postgraduate employment, 4) interpreting strengths and weaknesses of residency and fellowship programs which can help students achieve their career goals, 5) developing a timeline for the postgraduate application process.
Course content is centered on development and refinement of many soft skills, such as goal analysis, refinement of professional writing, and strategies for showcasing strengths. As such, the PRF course is ideally suited to allow students time for thoughtful reflection and to work at their own pace. The transition to a hybrid online course started after the 2015 spring semester and was implemented during the spring 2016 semester. Course design began with preparing a student workbook, recording online modules, revising and generating new assessment modalities, and altering the syllabus. Course learning outcomes, objectives, and content remained consistent (Figure 1).
Figure 1. Outline of Topics, Assessments, and Mapped Objectives from Table 1 for the Spring 2016 PRF Electivea.
Improving student engagement was a key focus. Instructional methods to target millennial students were explored. Videos were limited to 20 minutes in length to allow time for activities. Online, interactive platforms were used to engage students in thoughtful reflection and discussion, including posting activities, discussions, and announcements in the Moodle (Perth, Australia) course management system, as well as using an individual workbook in Google Docs (Alphabet, Inc., Mountain View, California). The workbook included instructions on modules, activities, due dates, and supplemental documents and served as a continuous document that allowed students to submit certain assignments, collaborate one-on-one with faculty, receive feedback on assignments via track changes and comments, and reflect on their progress in the course. Live class sessions were conducted twice during the semester: an introduction session and resident panel at the end of the semester, along with course evaluation. Additionally, one-on-one mock interviews were scheduled between students and course instructors.
The module format included viewing a pre-recorded lecture followed by an assignment, activity, or reflection. Many innovative activities were developed for the hybrid format. To glean perspectives outside of the pharmacy profession, students completed a mock, videotaped interview with the University’s Internship and Career Services (ICS) prior to their final mock interview. Students viewed the recording, self-assessed, and received feedback from the ICS staff member. Diigo, an online social bookmarking site, was utilized for students to evaluate with PRF programs (https://www.diigo.com/). This platform allowed students to bookmark PRF program sites, annotate webpages, type in pros/cons of the program, and ultimately share with their peers. Another unique learning activity included Quandary clinical cases (http://www.halfbakedsoftware.com/quandary.php). These were designed to mimic case presentations that may be encountered in interviews. This web-based decision making program helped students walk through scenarios by asking questions and providing immediate feedback on basic thought processes. Students were asked to reflect on how they approached the case and any modifications they would implement prior to interviews. Finally, they were also required to complete a shadowing experience with a local pharmacy resident and complete a written or video blog post reflecting on their experience. Course enrollment was capped at 20 students in 2016 due to the new delivery method.
CRITICAL ANALYSIS
Thirty-three students participated in the live PRF elective course in 2015 compared to 20 in the hybrid online course in 2016. The majority were P3 students, with 76% and 80% for each year, respectively.
Evaluation Methods
Three methods of evaluation were used to assess the transition from a live, synchronous course (2015) to a hybrid online platform (2016): 1) grades, 2) evaluations, and 3) PRF placement rates. Individual assignment and overall course grades were compared between 2015 and 2016. Although the assessment methods differed slightly between years, the overall content of the assessments did not and assignment point allocation remained similar. After each semester, evaluations were completed utilizing the mandatory University-wide evaluation tool, the IDEA Student Ratings of Instruction.9 Three essential teaching objectives were identified by the IDEA service: 1) learning to apply course material, 2) developing specific skills, competencies, and points of view needed by professionals in the field most closely related to this course, and 3) developing skill in expressing myself orally or in writing. The individual ratings of these objectives, their aggregate scores, and overall rating of the course were compared between 2015 and 2016. PRF placement of students who completed the course in 2015 were compared with students who completed it in 2016, as the National Matching Service (NMS) did not report individual match data to colleges prior to 2015. Participants were only included in the analysis if they actively sought PRF the year of their graduation. Data was collected via NMS reports, self-reported student exit surveys, and personal communication with course faculty.
Individual assignment grades, overall course grades, and IDEA evaluation scores were evaluated by Independent Samples Mann-Whitney U tests. PRF placement rates were evaluated by Fisher’s exact test. Statistical tests were performed in SPSS v24.0 (IBM, Armonk, NY). A p-value of <0.05 was considered statistically significant. This study was granted exempt status by the University’s Institutional Review Board.
Evaluation Results
There was no significant difference in overall course grades between semesters (p=0.28). Individual assignment grades were also similar (Table 1). The only significant difference was on the video/blog reflection assignment for the resident/fellow shadow experience (p<0.001).
Table 1. Course Grades and IDEA Score Ratings from 2015 and 2016a.
| Median (IQR) | ||||
|---|---|---|---|---|
| Assignment Grades (Points)b,c | 2015 (n=33) | 2016 (n=20) | Raw Difference (2015 to 2016)d | p-value |
| Professional Goals Reflection (10) | 10 (10,10) | 10 (10,10) | No change | NS |
| Program Evaluation (10) | 10 (10,10) | 10 (10,10) | Increase | 0.27 |
| Strengthen Self as Candidate (10) | 10 (10,10) | 10 (10,10) | Increase | 0.11 |
| CV Peer Feedback (5) | 5 (5,5) | 5 (5,5) | No change | NS |
| LOI Peer Feedback (5) | 5 (5,5) | 5 (5,5) | No change | NS |
| LOI Final (10) | 10 (10,10) | 10 (10,10) | No change | NS |
| Interview Questions (5) | 5 (5,5) | 5 (5,5) | Increase | 0.55 |
| Video Blog of Resident/Fellow Shadow Experience (10) | 10 (10,10) | 10 (10,10) | Decrease | <0.001 |
| ICS Mock Interview (10) | 10 (10,10) | 10 (10,10) | Increase | 0.17 |
| Overall Grades (100) | 100 (100,100) | 100 (100,100) | Increase | 0.28 |
| IDEA Score Ratings on Relevant Objectivesd | 2015 (n=29) | 2016 (n=19) | Raw Difference (2015 to 2016) | p-value |
|---|---|---|---|---|
| 1. Learning to apply course material (to improve thinking, problem solving, and decisions) | 5 (5,5) | 4 (4,5) | Decrease | 0.14 |
| 2. Developing specific skills, competencies, and points of view needed by professionals in the field most closely related to this course | 4 (4,5) | 4 (4,5) | Increase | 0.67 |
| 3. Developing skill in expressing myself orally or in writing | 4 (4,5) | 4 (4,5) | Decrease | 0.16 |
| 4. Average ratings from objectives 1-3e | 5 (5,5) | 4 (4,5) | Decrease | 0.04 |
| 5. Overall rating of course as excellent | 5 (5,5) | 5 (5,5) | Decrease | 0.02 |
IQR – interquartile range; NS – not significant; CV – curriculum vitae; LOI – letter of intent; ICS – Internship and Career Services
Professional Organization assignment (5 points) was added in 2016; all students received 100%
Final Interview/Application was worth 20 points in 2016 due to addition of Professional Organization assignment and worth 25 points in 2015. All students received 100% both years
Ratings were on a 5-point Likert scale (1-no apparent progress/definitely false, 2-slight progress/more false than true, 3-moderate progress/in between, 4-substantial progress/more true than false, 5-exceptional progress/definitely true)
n=87 for 2015, n=57 for 2016
Student evaluations of the IDEA essential objectives were similar between years (p>0.05). There were significant differences in the combined rating of these three objectives and overall rating of the course as excellent, which were lower in 2016 (p<0.05). However, median scores of at least a 4 or 5 (highest marks) were achieved in all areas during both years. Twenty-four of 33 students (73%) from the 2015 class and 17 of 20 students (80%) from the 2016 class pursued PRF positions. PRF positions were secured by a total of 21 students (3 fellowship, 18 residency) from the 2015 class and 14 of 17 (all residency) from 2016 class, with no statistically significant difference between groups (88% vs 80%, respectively; p=0.66).
Key Issues/Discussion
The PRF elective course was successfully delivered in a hybrid online format and still resulted in an overall positive experience. Students continued to actively participate and complete assignments in 2016 despite the self-directed format. Overall course grades and most individual assignment grades did not change between 2015 and 2016, although students were primarily graded on participation and completion of assignments, making it difficult to compare due to consistently high marks. Students performed better on the video/blog reflection in 2015 than 2016, though this did not influence overall grades. This may have been due to students being unclear on everything to include in the reflection, resulting in incomplete assignments and reduced grades, highlighting the need for clear instructions in online formats. In addition, student ratings of individual objectives did not change during the transition, which can likely be attributed to consistent course content. However, ratings for overall course and combined ratings of these objectives were significantly lower in 2016. When evaluating the specific data, this seems to be influenced most heavily by the objective “developing skill in expressing oneself orally or in writing.” Lower ratings may be attributed to the course involving a heavy online component, minimizing the need for regular verbal communication, or it may reflect the need for more face-to-face interaction. We suspect that transitioning to a new method of delivery resulted in lower evaluation scores. Additionally, the majority of coursework at the University is delivered live, so it is possible that students were not accustomed to a more autonomous, self-paced course. The relevance of these ratings may be questioned after evaluating the median and interquartile range for these endpoints, as median ratings of relevant objectives and overall course were similar between years.
PRF attainment rates between students completing various course formats did not differ, although this was a small sample limited to two years of data. It also did not represent students who sought alternative opportunities, such as employment or additional degrees, or those that secured residency and did not take the class (about 50% of each cohort). While students could take the class during P2 or P3 year, the authors do not believe engagement level was affected. Although not analyzed separately, it is worth noting that seven of the 12 total students who took the class as P2s pursued PRF positions and all secured residencies.
Creation of online modules allowed for a self-paced format that afforded flexibility and autonomy. This model can be easily implemented at schools or colleges with online platforms. However, it is not without unique considerations. Though time intensive to develop, this format allows instructors to serve several cohorts of learners simultaneously. Although our perceived time commitment was not lower due to preparation of initial materials and troubleshooting issues, it is anticipated to decrease in future semesters to facilitation, grading, fielding questions and troubleshooting issues. Recorded lectures can be utilized repeatedly, as most of the topics do not require updating from year-to-year. Additionally, utilizing multiple faculty to grade assignments allows for individualized feedback with low overall time commitment.
Unique online tools, such as Quandarys and Diigo were used to build some course assessments. These tools were interactive and engaging for learners. Students particularly appreciated the Diigo PRF evaluation assignment as it allowed them to view all their classmates’ assessments of programs on a single thread. Although these tools were engaging, optimal use of a learning management system (e.g. Blackboard, Moodle) may remove the need for additional resources and effort. In addition, use of faculty advisors and preceptors, as well as University resources (e.g. career services), may allow for alternative perspectives and expansion to larger numbers of students with minimal workload additions. In the subsequent year, enrollment was expanded to 24 students with no subjective increase in workload, despite a decrease in the number of evaluators. Enrollment will continue to be increased in future years.
NEXT STEPS
The evaluation of the hybrid online PRF course highlight the need for continual course improvement. In evaluations, some students indicated they preferred to have more live sessions. Future semesters will include additional live sessions to build on online coursework and provide students a venue for group discussion. Additionally, student feedback will continue to be monitored to ensure balance of online and face-to-face sessions.
Overall, the transition of the PRF elective to a hybrid online course did not significantly affect course grades, student evaluations, or PRF placement rates. As the first iteration of this course in a hybrid model, continuous quality improvement is necessary to ensure positive student experiences.
Furthermore, use of the hybrid model for this type of elective may improve downstream workload commitments and is replicable at other institutions.
Acknowledgements
The authors would like to thank Alex N. Isaacs and Meredith L. Howard for their contributions to the original course content.
Funding/Support
None
Other Disclosures
None
References
- 1.Murphy JE, Nappi JM, Bosso JA, et al. American College of Clinical Pharmacy’s vision of the future: postgraduate pharmacy residency training as a prerequisite for direct patient care practice. Pharmacotherapy. 2006;26:1816–1825. doi: 10.1592/phco.26.5.722. doi: 10.1592/phco.26.5.722. [DOI] [PubMed] [Google Scholar]
- 2.American Society of Health-System Pharmacists ASHP long-range vision for the pharmacy work force in hospitals and health systems. Am J Health-Syst Pharm. 2007;64:1320–1330. doi: 10.2146/ajhp070057. [DOI] [PubMed] [Google Scholar]
- 3.National Matching Services Inc Summary results of the match for positions beginning in 2016 combined phase I and phase II. [Accessed Jul 12;2018 ]. https://www.natmatch.com/ashprmp/stats/2016applstats.html Available at:
- 4.Phillips BB, Bourg CA, Guffey WJ, Phillips BG. An elective course in postgraduate residency training. Am J Pharm Educ. 2012;76(9) doi: 10.5688/ajpe769174. Article 174. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Caballero J, Benavides S, Clauson KA, et al. Role of residency interview preparatory activities as a determinant on pharmacy residency. J Pharm Pract. 2016;30(2):219–222. doi: 10.1177/0897190016632127. doi: 10.1177/0897190016632127. [DOI] [PubMed] [Google Scholar]
- 6.Porter AL, Pitterle ME, Hayney MS. Comparison of online versus classroom delivery of an immunization elective course. Am J Pharm Educ. 2014;78(5) doi: 10.5688/ajpe78596. Article 96. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.King AE, Egras AM. A required online course with a public health focus for third professional year pharmacy students. Am J Pharm Educ. 2015;79(5) doi: 10.5688/ajpe79568. Article 68. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Brahm NC, Davis T. Psychiatric pharmacy delivery via an online course: psychoactive substances. Curr Pharm Teach Learn. 2010;2(4):204–210. doi: 10.1016/j.cptl.2010.07.007. [DOI] [Google Scholar]
- 9.IDEA Student Ratings of Instruction. [Accessed Jul 12;2018 ]. http://www.ideaedu.org/services/student-ratings-of-instruction/