Effects of Remote Proctoring on Composite Examination Performance Among Doctor of Pharmacy Students

Abstract

Objective. To determine the impact of remote proctoring on the academic performance of Doctor of Pharmacy (PharmD) students.

Methods. This was a retrospective, observational study that compared first professional year (P1) and second professional year (P2) pharmacy students’ scores on eight composite examinations administered in spring 2020 (n = 387), the final three of which were proctored remotely, to that of a historical cohort of pharmacy students who took the same examinations in spring 2019 (n = 368). To assess whether remote proctoring affected academic performance, spring 2020 scores for examinations 6, 7, and 8 were compared to those of a historical cohort who took the same examinations in person with a proctor present in spring 2019. Academic performance on examinations 1 through 4 was also compared between the two cohorts to evaluate any possible year-to-year variation in academic performance during non-remote circumstances. Mann Whitney tests were used to compare scores between the two cohorts.

Results. The median scores of students in the spring 2020 cohort were significantly lower than the scores of the historical cohort on the first composite examination administered to P1 students after the implementation of remote proctoring. In contrast, median scores were significantly higher on two of the three examinations administered to P2 students using remote proctoring.

Conclusion. Remote proctoring has minimal impact on pharmacy students’ examination performance and its use should be considered to ensure academic honesty and security of testing content in a distance learning environment.

INTRODUCTION

In response to the coronavirus disease 2019 (COVID-19) global pandemic, many educational institutions rapidly transitioned to distance learning. This learning environment presents several challenges, particularly with student assessment and online examinations. These challenges may include technical issues, adapting to a novel learning environment, and ensuring academic integrity.^1-3 One method to ensure academic integrity for an examination taken remotely is to institute surveillance measures, such as remote proctoring.^4,5 Real-time online proctoring services, such as ProctorU (ProctorU, Inc), have gained in popularity in recent years. In their study of student pharmacists’ perceptions and the impact of ProctorU, Milone and colleagues reported that 88.95% of student pharmacists were satisfied with their experience. The most frequently encountered issues were technical difficulties, personnel issues, and taking too long to start the examination.⁵ The service also has a fee per examination, which may further limit its use.

Because of the COVID-19 pandemic, on March 16, 2020, the University of Tennessee Health Science Center (UTHSC) transitioned to remote learning. Demand for online proctoring services was at an all-time high, and proctor availability was limited. Consequently, in April 2020, the UTHSC College of Pharmacy developed and executed a novel plan to remotely proctor multiple-choice question examinations using only internal and existing resources. As remote proctoring was implemented, students anecdotally reported increased levels of stress that they felt were negatively affecting their examination scores.

There are no publications to date investigating how remote proctoring may affect pharmacy students’ academic performance. Similarly, the impact of remote proctoring on student performance has not been well described in other health professions. Weiner and colleagues investigated live vs remote proctoring for three high stakes professional licensing examinations and found no significant difference in overall scores between the two cohorts.⁶ Given the paucity of existing data, the current study aimed to determine the impact of remote proctoring on academic performance among first professional year (P1) and second professional year (P2) Doctor of Pharmacy (PharmD) students.

METHODS

The remote proctoring process developed and implemented at UTHSC College of Pharmacy used internal resources and existing technology. Remote proctoring required pharmacy students to use two devices. On the first device, the student was monitored with videoconferencing software (Zoom Video Communications, Inc). On the second device, the computer-based examination was administered securely via ExamSoft (ExamSoft Worldwide, Inc). Secure examinations in ExamSoft restrict access to all other applications and disable all device functions during the examination.

Proctors were UTHSC College of Pharmacy faculty and staff who completed a one-hour training that discussed procedures before, during, and after the examination along with a written procedures guide. Additional one-on-one sessions with IT staff were available if needed for adequate orientation. Students also attended a one-hour informational session to learn what to expect on examination day and could consult with IT staff as needed before the examination.

Prior to the examination, the proctor checked in on each student, having a 360° view of the student’s workspace and with ID verification. Throughout the examination, students maintained audio and visual connections. Each proctor observed up to 24 students and logged any suspicious activities, which were reported to the Office of Academic Affairs. If needed, the college’s coordinator of testing joined Zoom sessions to intervene on any suspicious activities in real-time using a breakout room to avoid interrupting other students. In addition, IT staff were available to help with any technical issues during the examination.

Computerized composite examinations at UTHSC are administered every two weeks (eight total examinations per semester) with questions from several courses combined into a single three-hour examination and has been previously described.⁷ In both spring 2019 and spring 2020, all courses included on the composite examination had three to four questions per lecture hour.

The study population of interest included P1 (n = 192) and P2 (n = 195) students enrolled at UTHSC College of Pharmacy in spring 2020. Cohorts of P1 (n = 203) and P2 (n = 165) students enrolled at UTHSC College of Pharmacy in spring 2019 served as historical comparators. Remote proctoring began with examination 6 in the spring 2020 semester for both P1 and P2 students. To assess whether remote proctoring impacted academic performance, spring 2020 scores on examinations 6, 7, and 8 were compared to scores on the same examinations taken by a historical cohort of students who had been observed live by a proctor in spring 2019. Academic performance on examinations 1 through 4 was also compared between the two cohorts to evaluate year-to-year variation in performance during in-person learning. Student scores on all composite examinations administered in both semesters were collected retrospectively. Student demographics, including age, gender, and race were also obtained retrospectively.

Descriptive statistics for all variables were calculated. Shapiro Wilks tests and scatter plots were examined to assess normality of data. Mann Whitney tests compared the distribution of scores on each of the composite examinations administered in spring 2020 to its corresponding examination administered in spring 2019. All tests were two-tailed, and p values  < .05 were considered significant. All analyses were conducted using SPSS, version 25.0 (IBM Corporation). This study was deemed exempt by the UTHSC Institutional Review Board.

RESULTS

Characteristics of the study participants from both cohorts are presented in Table 1. Most participants were female and White with a mean age of approximately 25 years. There were significant differences in the mean age of students in the 2020 and 2019 P2 cohorts (p < .001) and significant differences in race in students in the 2020 and 2019 P1 cohorts (p = .008).

Table 1.

Comparison of Demographic Characteristics of Doctor of Pharmacy Students Who Completed Examinations with Remote Proctoring (Spring 2020) Versus In-Person Proctoring (Spring 2019)

All composite examination scores for spring 2020 and spring 2019 are presented in Table 2. Examinations 1 through 4 were administered in 2019 and 2020 with live proctoring for both P1 and P2 students, and no significant differences in academic performance were noted on four of the eight (50%) examinations. As compared to spring 2019, spring 2020 academic performance was significantly worse on the P1 examination 1 (p  =  .04), P1 examination 2 (p < .001), and P2 examination 1 (p < .001), but significantly better on P2 examination 2 (p = .02). Median scores on examination 5 were significantly higher in spring 2020 vs spring 2019 for both P1 (p = .02) and P2 (p < .001) students. Academic performance was significantly worse in spring 2020 vs spring 2019 for P1 examination 6 (p < .001), but no significant difference was noted for P1 examination 7 or P1 examination 8. Median scores for P2 examination 6 and P2 examination 8 were significantly higher in spring 2020 as compared to spring 2019 (p = .04 and p < .001, respectively), but scores were significantly lower in spring 2020 for P2 examination 7 (p < .001).

Table 2.

Comparison of Examination Scores Among Doctor of Pharmacy Students Who Completed Examinations with Remote Proctoring (Spring 2020) Versus In-Person Proctoring (Spring 2019)

DISCUSSION

This study aimed to examine the effects of remote proctoring on examination performance among P1 and P2 PharmD students. The results indicate a variable impact on examination scores. On the first composite examination with remote proctoring, academic performance declined for P1 students but not for P2 students. The P1 students may have been more nervous about the change in proctoring modality. Additionally, P2 students may have been more accustomed to remote learning technology as Zoom was used throughout the PharmD program to facilitate communications between the three campuses, even prior to the pandemic. After the first examination with remote proctoring, P1 student performance did not differ significantly from the historical cohort, suggesting there may be an adjustment period.

For educators who are considering adopting a similar process for remote proctoring, we highly recommend including an orientation for both students and proctors and the opportunity to participate in a mock examination process before the first remotely proctored examination. Based on our experience in spring 2020, we added a practice remote proctoring session to acquaint students with the process beforehand for our students matriculating in fall 2020, and many upperclassmen commented that they wish this had been provided for them. In addition, we believe the human resources used in our approach were vital to ensure success. On examination days, four IT staff members and one academic affairs staff member were available if any issues arose during the examination. In addition, a faculty member within academic affairs monitored the success of the remote proctoring process and followed up on any reports from the proctors about academic dishonesty.

When examinations must be administered remotely, institutions often consider remote proctoring as an option to reduce academic dishonesty. Academic dishonesty was a concern voiced by several faculty and students at our institution during this transition to remote learning, especially after examination 5 was administered without any proctoring and significant improvements in examination scores were noticed. Substantive changes in examination design, such as using essays or other alternatives to multiple-choice questions, were not possible because of our large class size, limited resources, and constrained timeline. Developing and implementing these remote proctoring procedures allowed our program to maintain our assessment approach in a secure manner. Furthermore, we were able to remotely proctor examinations without any additional cost to the student because our procedures only used technology already in place prior to the pandemic and faculty and staff already employed by our institution.

Although remote proctoring offers benefits in terms of reducing academic dishonesty, there are also potentially negative implications to consider. Remote proctoring may be considered intrusive by some students. During our experience, only a few students voiced these concerns, specifically regarding the check-in procedures. Those students were offered the option to be checked in first so that their classmates did not see a 360° view of their workspace. Inclusivity is another consideration when conducting examinations remotely. Remote proctoring requires a high-speed, stable internet connection and a device with video capabilities, and some students may not have access to these. Such students were offered the option to test on campus, and a video capable device was loaned to any students needing one. Based on unsolicited feedback from students, the convenience benefits, including being able to choose where to take the examination, outweighed the negative aspects of the experience, and students generally enjoyed the autonomy offered by remote proctoring.

This study is not without limitations. There may have been confounding factors affecting examination performance that were not elucidated by this study. One potential confounding factor was the program’s simultaneous change to remote instructional delivery in spring 2020, with some lectures delivered synchronously and others asynchronously, depending on the lecturer and course director. However, the course content, examination content, and examination construction (ie, multiple choice) did not change with the transition to remote learning in spring 2020 and were the same as that used in spring 2019. Another limitation to consider is whether there were differences in the academic ability of the two cohorts that in turn impacted our findings. Academic performance on most examinations administered in spring 2020 prior to the transition to remote learning (ie, examinations 1 through 4) was observed to be comparable to that of the historical cohort, which lessens the potential impact of this limitation. The generalizability of the study may also be limited as the sample only includes students at one college of pharmacy. However, the likelihood of sample bias is low given that the characteristics of our study sample reflects the broader population of pharmacy students in the United States.

Although this study was conducted amidst a global pandemic, the implications will still be relevant in a post-pandemic environment. Distance education and online learning programs were already becoming more common. As of 2020, five PharmD programs in the United States were offering distance pathway options, which was a significant increase from just three programs offering this option the previous year.^8,9 Strategies to reduce academic dishonesty on high stakes examinations taken remotely while also minimizing any negative impact on a student’s performance will be paramount with distance pathway options increasing. Standard 10.17. Academic Integrity of the Accreditation Council for Pharmacy Education’s Standards 2016 further emphasizes the importance of minimizing academic misconduct opportunities for all, including distance students.¹⁰ Based on the findings from our study, remote proctoring may offer a satisfactory solution worthy of consideration.

CONCLUSION

The composite examination scores of P1 and P2 PharmD students did not consistently improve or decline with the implementation of remote proctoring at one college of pharmacy in spring 2020. Remote proctoring may be a reasonable strategy to ensure academic honesty and security of testing content in a remote educational environment with minimal impact on a student’s examination performance.