Pharmacy Student Perceptions and Preferences of In-person Versus Video-Recorded Evaluations in Skills-Based Courses

Abstract

Objective. To determine pharmacy students’ preferences for and perceptions of in-person and video evaluations.

Methods. A mixed methods survey was administered to 447 first-, second-, and third-year students enrolled in a public US Doctor of Pharmacy program. A survey instrument with 14 quantitative items and four qualitative items was used to measure student perceptions. Eight response choice items measured preferences. Paired t tests were used to compare students’ perceptions. Independent t tests were used to compare perceptions between students who experienced and had not experienced video evaluations. Two researchers performed thematic content analysis of the qualitative responses.

Results. Students (n=444, 99.3% response rate) perceived in-person evaluations more positively for all items except nervousness. Students who experienced video technology felt significantly more positive about video evaluations than students who had little or no experience using video technology on nine items. The students who experienced video technology felt significantly less positive toward video evaluations in terms of quality (1.24 vs. 0.83) and amount (1.14 vs 0.77) of written feedback. Although students valued the interaction with a larger, more diverse pool of evaluators that was made possible by video evaluations, they did not view video technology as applicable to their future practice.

Conclusion. Students viewed in-person evaluations significantly more positively than video evaluations. This effect was mitigated by greater exposure to video technology, suggesting that concerns regarding video evaluations are based on conjecture rather than experience. This study highlights the need to reduce the technological issues associated with video evaluations and improve the written feedback provided to students.

INTRODUCTION

As technology becomes increasingly integrated into classrooms and pharmacy practice, it is necessary to study the impact of technology on Doctor of Pharmacy (PharmD) students. With the growth in the number of pharmacy schools with multiple campuses, the use of live streaming or video recordings to reach students on satellite campuses has increased.^1,2 Several studies have compared the study habits, grades, and North American Pharmacist Licensure Examination (NAPLEX) pass rates of students on main and satellite campuses, yet none has found significant differences in outcomes between traditional classroom learning and distance learning.^3-6 Similar results have been found in other academic programs, including in physical therapy and medicine programs, that use technology to bridge multiple campuses.^7-9

Technology is not only pervading pharmacy curricula, but also proliferating within pharmacy practice with the growing popularity of services such as telepharmacy. The American Society of Health-System Pharmacists’ statement on telepharmacy calls for “appropriately trained and equipped pharmacists” to use telepharmacy to provide patient care to those who do not have easy access to a health care facility.¹⁰ As telepharmacy becomes increasingly common, it is necessary to familiarize pharmacy students with the process of providing pharmacy services from a remote location. While some pharmacy students have mixed feelings about viewing video recordings of lectures, many have reported a willingness to use technology to conduct on-campus patient assessments with mannequins and virtual patient assessments using computer programs.¹¹ However, little if any research has explored pharmacy instructors’ use of teleconferencing or video streaming to evaluate student performance.

Student perceptions and preferences regarding video evaluation of their performance are of interest as student satisfaction with the evaluation process has been linked to positive learning outcomes.^12-14 Ramsden and Entwistle found a correlation between positive student perceptions of courses and better learning outcomes across several areas of study.¹⁴ Lizzio and colleagues studied the relationship between student perceptions and learning outcomes and found that student perceptions of higher levels of teaching quality and appropriateness of workload were associated with better academic achievement.¹²

Despite the prevalence of research conducted to examine the effects on pharmacy students of using technology to access educational content, no research has explored the impact of instructors using remote technology to evaluate and provide feedback to pharmacy students rather than using traditional, in-person techniques. While some research has focused on the effects of remote teaching and assessment on student learning and achievement, little research has examined student perceptions or preferences regarding the two modalities of teaching and evaluation.¹⁵ This study addresses this knowledge gap by assessing pharmacy student perceptions of and preferences regarding video vs in-person evaluations.

METHODS

A mixed methods design was used to assess pharmacy students’ perceptions and preferences regarding in-person vs video evaluations within a skills laboratory curriculum at one public, single-campus US PharmD program. The skills laboratory curriculum was comprised of six stand-alone, required courses occurring each semester of the first through third professional years of the program. For each laboratory course, students met weekly for three hours throughout the semester and participated in various patient counseling sessions, patient interviews, and provider communication activities. Instructors evaluated and provided both formative and summative feedback to students either in person or via video. In this study, “in-person evaluations” referred to evaluations in which the instructor was physically present in the same room as the student during the activity. “Video evaluations” referred to evaluations in which the instructor viewed the activity using an audio-video recording device or live-streaming tool such as Skype from a remote location. The majority of video evaluations occurred synchronously, with instructors using personal computers with webcams that connected to monitors and speakers in the rooms where the students performed the laboratory activities.

When faculty members agreed to evaluate students at the start of the semester, they chose whether to evaluate students in person or remotely via video for each activity. Therefore, students were, in effect, randomly designated to be evaluated either in-person or via video. The video option was available to allow faculty participation in student skills-based evaluations while supporting faculty members in meeting their other obligations, eg, clinical practice, as the majority of faculty members evaluating students in the laboratory sessions were based off campus.

To assess student perceptions and preferences, an anonymous, online survey was developed using Qualtrics (Qualtrics, Provo, UT). The survey included 18 perception items, eight response choice preference items, and four demographic items. Several of the preference items were adapted from the literature.^16,17 Students assessed 14 of the 18 perception items using a five-point Likert-type scale with five indicating the highest level of agreement and one indicating the lowest level of agreement. The remaining four perception items were open-ended questions designed to solicit more in-depth insights into student perceptions of in-person and video evaluations. The three response choices for the eight preference items were in-person evaluations, video evaluations, and both are equal. Demographic questions collected participant age, gender, year in the pharmacy program, and previous exposure to video evaluations.

An email with a link to the survey on Qualtrics was distributed to all (N=447) first, second, and third year PharmD students enrolled during the spring 2019 semester. The voluntary survey was administered at the beginning of their designated laboratory meeting time at the beginning of the spring semester by which time most students had experienced both in-person and video evaluations during the prior semester. If a student had not experienced any video evaluations by the time the survey was administered, they were asked to answer the questions anyway based on their general perceptions of video evaluations. No incentives were provided for study participation.

For the quantitative data, paired t tests were used to compare student perceptions of in-person and video evaluations. Independent t tests were used to compare the differences in perceptions between students who had experienced video evaluations and students who had not experienced video evaluations within each of the three class cohorts. Given the variability between each class, no analysis across the classes was performed. All distributions of data had an approximately normal distribution. An a priori alpha of .05 was used for all statistical tests, and pairwise exclusion was used for missing data. For the qualitative data, two researchers independently reviewed all responses to the open-ended questions and identified themes using an inductive approach.¹⁸ After reconciling and defining the themes, the researchers then independently coded all responses. Each response may have included more than one theme. The researchers then reconciled and finalized all coding. Study approval was received by Purdue University’s Institutional Review Board.

RESULTS

Of the 447 first-, second- and third-year pharmacy students invited to participate, 444 (99.3%) responded. Demographics of study participants may be found in Table 1.

Table 1.

Demographics of Participants in a Study to Assess Pharmacy Students’ Preferences and Perceptions Regarding In-Person and Video Evaluations in Pharmacy Skills-Based Laboratory Courses

In the quantitative portion of the survey, we found that students perceived in-person and video evaluations as significantly different on all 14 perception items (p<.05) (Table 2). Students had significantly more positive perceptions of in-person evaluations for 13 of the 14 items. The only item on which students rated video evaluations significantly more positively (p=.028) was the one that assessed nervousness, ie, students felt less nervous during video evaluations than in-person evaluations. When separated by year in the pharmacy program, only first-year students perceived in-person and video evaluations equally in terms of causing nervousness. All other items for all three years were perceived differently (p<.05). For perception items that were significantly different, in-person evaluations were perceived more positively for all items except nervousness (video=2.7 vs in person=2.5 for first-year students; video=2.5 vs in person=2.3 for second- and third-year students, respectively).

Table 2.

Perceptions of In-Person^a versus Video Evaluations^b by Professional Year and Total

Although all students had previously experienced in-person evaluations, not all students had experienced video evaluations (Table 1). As the students progressed through the curriculum, they were more likely to have experienced video evaluations (29.1% of first-year students experienced vs 91.2% and 95.8% of second- and third-year students experienced, respectively). Overall, 71.2% (n=316) of the participants had previously experienced video evaluations. The difference between student perceptions of in-person and video evaluations was significantly (p<.05) different between those who had experienced and those who had not experienced video evaluations for 11 of the 14 perception items (Table 3). Of these 11 items, students who had not experienced video evaluation perceived a greater difference between the two evaluation methods, with a more positive perception of in-person evaluations for all but two items (quality and amount of written feedback). Exposure to video evaluations had no significant effect on students’ perceptions about the likelihood of needing to remediate an activity, evaluation method appropriateness for high-stakes assessments, or nervousness.

Table 3.

Comparison of Mean Differences in Perceptions^a Between Exposure and Non-Exposure to Video Evaluations by Professional Year and Total

When asked to directly compare in-person and video evaluations, students preferred in-person evaluations to video evaluations for seven of eight items when participants who reported no preference were excluded (Figure 1). Students only preferred video evaluations when asked to compare which method made them less nervous. This finding aligned with the results from the perceptions section of the survey.

Figure 1.

Preferred Type of Evaluation Method

The response rates for the four open-ended items in the qualitative portion of the survey were as follows: What do you like about in-person evaluations? (n=436, 97.5%), What do you dislike about in-person evaluations? (n=423, 94.6%), What do you like about video evaluations? (n=429, 96.0%), What do you dislike about video evaluations? (n=433, 96.9%). The response rates were similar to those for the quantitative portion of the survey. Student responses to the four questions resulted in 2,565 codes (interrater agreement=94.8%) categorized into 11 themes. Through thematic content analysis, we identified 11 themes: convenience, presence, comfort, variety of evaluators, feedback, applicability, technology, focus, performance, lack of preferences or perceptions, and other. Each response (N=1721) was coded to at least one of the identified themes. Illustrative quotes and frequencies of codes are provided in Table 4. Each theme contains relevant student comments, whether positive or negative.

Table 4.

Pharmacy Student Perceptions of In-Person and Video Evaluations

Presence was the most commonly coded theme (n=513), with the majority of comments appearing in response to what the students liked about in-person evaluations (n=203, 46.6% of responses to this item) and disliked about video evaluations (n=134, 30.9% of responses). Similar results were found for the second- and fourth-most commonly coded themes: feedback (n=470) and applicability (n=291), respectively. Comfort was the third most commonly coded theme (n=443), with the majority of codes from responses to what the students disliked about in-person evaluations (n=216, 51.1% of responses). Performance was the least commonly mentioned theme, with only 43 responses across all four items reflecting this theme.

DISCUSSION

Although PharmD students preferred in-person evaluations across almost all aspects of the evaluation process, these preferences were mitigated by exposure to video evaluations as the students advanced through the pharmacy program. This finding suggests that a substantial amount of student concerns regarding video evaluations may be rooted in fear of the unknown rather than in having had negative experiences. Based on previous findings that suggest student preferences and learning outcomes are positively correlated, it may be necessary to introduce all students to video evaluations early in the curriculum to reduce negative perceptions.^12-14 In this study, student exposure to video evaluations improved perceptions in all areas except for perceptions of quality and amount of written feedback. Notably, student responses to the open-ended questions suggested that the students had experienced long delays between participating in the laboratory activity and receiving their written feedback when evaluated via video. One way to address this discrepancy is to provide all students with their written feedback at the same time, regardless of which evaluation method is used. All evaluators would be required to give their written feedback to the course coordinator immediately following the activity, either in hardcopy or electronic form.

The qualitative findings of this study reinforced and expanded upon the quantitative findings. The finding that students preferred in-person evaluations were largely confirmed by students’ responses to the open-ended survey items. New findings revealed by the qualitative portion of the survey included the importance of presence, applicability, and diversity of evaluator pools.

When discussing reasons for liking or disliking in-person and video evaluations, students most commonly mentioned the physical distance between the student and evaluator and its impact on establishing a personal connection or professional rapport. Some students felt more comfortable with the distance provided by video evaluations, which likely explains why they reported feeling significantly less nervous during video evaluations in the quantitative portion of the survey. However, most students preferred the closeness afforded by in-person evaluations. Responses revealed this closeness served to improve communication, provide comfort, and even build relationships between students and evaluators.

Another notable finding was that students felt video evaluations were less realistic and less applicable to their future practice sites. Some students claimed they would never communicate with patients through video conferencing. However, as the number of states allowing the practice of telepharmacy increases, the use of video and teleconferencing to communicate with patients will become increasingly common in the practice of pharmacy.¹⁹ Social distancing restrictions imposed as a result of the COVID-19 pandemic have made the practice of telepharmacy even more prevalent and further supports the need to familiarize students with distance communication strategies.^20,21 Addressing students’ misconceptions will likely resolve the negative perceptions they have regarding video evaluation. Furthermore, the increased societal need for distance communication because of COVID-19 may provide students with a more accurate view of the future of pharmacy practice, ie, that telepharmacy may become the norm.

Finally, the qualitative portion of the survey revealed that students valued video evaluations for their ability to increase the number and types of evaluators involved in laboratory activities. Students reported they appreciated and enjoyed the chance to interact with a more diverse population of evaluators, as video evaluations encouraged more faculty members from the remote campus to participate. Several students noted they felt they received better feedback by interacting with a greater variety of evaluators. However, the benefits of interacting with a broader range of evaluators needs to be balanced with the technical issues associated with using distance technology, such as poor connections, video lag time, and poor-quality audio that students reported experiencing.

The high student response rate to the survey is a significant strength of this study. The use of a mixed methods approach, which helped us to explore student perceptions and preferences in greater breadth and depth, is another strength. Because pharmacy students at only one university were surveyed, our findings may not be generalizable to students at other schools. Also, we did not assess actual student outcomes. Therefore, this study was unable to reaffirm or refute findings of previous studies in which correlations were found between perceptions and learning outcomes.^12-14 Finally, including responses from both students who had and students who had not undergone video evaluations was both a strength and a limitation of this study. This combination may have clouded the aggregate results and made it more difficult for us to understand how students experienced video evaluations. However, collecting data from both groups of students also made it possible for us to make an initial assessment of how student perceptions may change before and after exposure to video evaluations and as they progress through the curriculum.

Several ways to improve educational practices regarding evaluation of skills-based activities and potential areas for future research can be drawn from this study. The quality and amount of written feedback provided to students who undergo video evaluations need to be improved. In order to maintain efficiency, manage student stress, and provide more equitable learning experiences, the number and frequency of technological problems associated with video evaluations need to be addressed. Finally, more emphasis on the prevalence and expected future expansion of telepharmacy may be needed in the PharmD curriculum. Furthermore, faculty members and administrators need to better understand how distance evaluations can better prepare students to eventually use this method of communication with patients. Future research should assess whether any grade discrepancies exist between the two methods of evaluation.

CONCLUSION

Pharmacy students prefer in-person evaluations to video evaluations; however, more exposure to video evaluations is associated with students having a more positive perception of video evaluations. The findings of this study highlight the need to reduce technological issues and improve written feedback associated with video evaluations. Students valued the opportunity to learn from a more diverse set of evaluators that video evaluations provided. As telepharmacy becomes increasingly prevalent, acclimating students to video conferencing is paramount.