Abstract
INTRODUCTION
COVID-19 altered medical education systems worldwide as many medical schools quickly changed to online assessment systems. However, the feasibility of online assessment and how it compares to traditional examinations is unclear.
METHODS
We compared 4th year medical students' online surgery clerkship assessment scores to the traditional written examinations. The percent of correct scores using online open-book examination was compared to the results of the traditional closed-book examination in the previous three rotations. Additional correlation between grade point average(GPA) and examination performance were reviewed.
RESULTS
Compared with the traditional groups, medical students who took the online, open-book examination had a significantly higher mean score in both MCQ(85.21 vs. 77.36, 72.43, 83.00, p<0.001) and essay examinations (187.36 vs. 158.77, 152.17, 152.29, p<0.001), but a significantly lower mean score in short answer examination (60.09 vs. 66.79, 67.73, 64.82, p<0.001). The online open-book examination group had a significantly lower correlation between the essay score and their GPA than the previous traditional groups (z=2.81 p=0.005, z=2.23 p=0.026, z=2.19 p=0.029).
CONCLUSION
Although an online, open-book examination was feasible during the COVID-19 pandemic, this study indicates that mean scores are significantly different which has important implications regarding grading and standard setting. More research is required to assess other effects of this new assessment on long-term knowledge retention and application.
Key words: Online examination, Open-book examination, Surgery clerkship education, Medical education, COVID-19
Competencies: Medical Knowledge, Systems-Based Practice, Practice-Based Learning and Improvement
What Problem Was Addressed?
COVID-19 altered medical education worldwide. Due to social distancing policies,1 many medical schools adopted online classrooms and assessments. In Thailand, the outbreak occurred 2 weeks before the end of the academic year. Course directors at Khon Kaen University School of Medicine discussed an online examination or a proctored, written examination after the outbreak. Our faculty believed this pandemic could persist for several months. Therefore, we decided to create an online examination as postponing the examination could have disruptive effects on subsequent courses.
What Was Tried?
We created an online final examination for 4th-year medical students in their 12-week surgery clerkship at the time of the outbreak. Students were sent home at the beginning of the 11th week. Only 1 week of clinical experience was affected/cancelled by the pandemic. Since we did not have the technological resources to conduct online monitoring for cheating, we decided that the examination would be open-book. This study aimed to compare students’ scores from the online open-book and the traditional closed-book examinations in the previous 3 rotations.
We used our in-house question bank software to deliver an online examination with 120 multiple-choice, 6 essay, and 10 short answer items. This is the same item bank used for prepandemic examination, the online test had same number of items of each type as those in the prepandemic examination. Students wrote text answers on blank sheets of paper, took pictures of the papers, and then uploaded the pictures electronically when finished. The online examination had the same time limits, examination metrics, and level of difficulty as the closed-book examinations in the previous 3 rotations. MCQs in all the examinations were taken from the same question bank; however, the questions used in each rotation were different. We surveyed all 47 students who took the online open-book examination. Forty-five medical students (95%) completed the survey.
Data Analyzed
We compared the percent correct scores across 4 rotations from 2019 to 2020 (3 traditional in-person, closed-book end-of-clerkship surgery examinations and 1 online, open-book examination). We also looked at correlation between student grade point average (GPA) and examination performance. Continuous variables between 4 rotations were tested by 1-way analysis of variance. A linear association between 2 variables was tested by Pearson's correlation coefficient. Fisher's z-test was used to compare 2 correlation coefficients.
Results and Lesson Learned?
Compared with the 3 traditional groups, medical students who took the online, open-book examination had a higher mean score in both MCQ and essay examinations, but a lower mean score in the short answer examination (Table 1 ). These results were different to previous published studies which demonstrated equivalence between open- and closed-book examination.2 , 3 Interestingly, the online, open-book examination group had a significantly lower correlation between the essay scores (Table 2 ) and their GPA than the 3 previous traditional groups (r = 0.191). However, the correlation between the MCQ item scores and short answer item scores with prior GPAs in each clerkship was not statistically significant. These results may indicate that students with lower GPAs may benefit more from the open-book format such they were able to do better on these items than prior GPA predicts. We did not collect data about what type of reference materials were used by the students to assist in answering the test questions.
TABLE 1.
Comparison of Scores for Different Item Formats Between Block Rotations
| Tradition 1 (N = 73) | Tradition 2 (N = 60) | Tradition 3 (N = 47) | Online Open Book (N = 47) | p Value | |
|---|---|---|---|---|---|
| MCQ final | 77.36 ± 7.16 | 72.43 ± 6.79 | 83.00 ± 6.99 | 85.21 ± 6.05 | <0.001 |
| Tradition 2 | −4.92 (p < 0.001) | ||||
| Tradition 3 | 5.64 (p < 0.001) | 10.57 (p < 0.001) | |||
| Online open book | 7.86 (p < 0.001) | 12.78 (p < 0.001) | 2.21 (p = 0.700) | ||
| Essay final | 158.77 ± 22.47 | 152.17 ± 21.72 | 152.29 ± 15.32 | 187.36 ± 7.50 | <0.001 |
| Tradition 2 | −6.60 (p = 0.261) | ||||
| Tradition 3 | −6.48 (p = 0.387) | 0.12 (p = 1.000) | |||
| Online open book | 28.59 (p < 0.001) | 35.195 (p < 0.001) | 35.08 (p < 0.001) | ||
| Short answer | 66.79 ± 10.12 | 67.73 ± 7.71 | 64.82 ± 8.54 | 60.09 ± 7.89 | <0.001 |
| Tradition 2 | 0.94 (p = 1.000) | ||||
| Tradition 3 | −1.975 (p = 1.000) | −2.91 (p = 0.533) | |||
| Online open book | −6.709 (p < 0.001) | −7.65 (p < 0.001) | −4.73 (p = 0.056) |
TABLE 2.
Correlation Between GPA Before Surgery Rotation and Final Examination Scores and Comparison of 2 Correlation Coefficients Between Block Rotations
| Tradition 1 (N = 73) | Tradition 2 (N = 60) | Tradition 3 (N = 47) | Online Open Book (N = 47) | |
|---|---|---|---|---|
| MCQ final exam & GPA before | 0.653 (p < 0.001) | 0.540 (p < 0.001) | 0.463 (p = 0.001) | 0.445 (p = 0.002) |
| Tradition 2 | z = −0.98 (p = 0.322) | |||
| Tradition 3 | z = −1.44 (p = 0.147) | z = −0.51 (p = 0.610) | ||
| Online open book | z = −1.56 (p = 0.116) | z = −0.63 (p = 0.529) | z = −0.11 (p = 0.912) | |
| Essay final exam & GPA before | 0.625 (p < 0.001) | 0.566 (p < 0.001) | 0.578 (p < 0.001) | 0.191 (p = 0.199) |
| Tradition 2 | z = −0.51 (p = 0.610) | |||
| Tradition 3 | z = −0.38 (p = 0.704) | z = 0.28 (p = 0.780) | ||
| Online open book | z = −2.81 (p = 0.005) | z = −2.23 (p = 0.026) | z = −2.19 (p = 0.029) | |
| Short answer & GPA before | 0.528 (p < 0.001) | 0.410 (p = 0.001) | 0.458 (p = 0.001) | 0.583 (p < 0.001) |
| Tradition 2 | z = −0.85 (p = 0.395) | |||
| Tradition 3 | z = −0.48 (p = 0.631) | z = 0.29 (p = 0.772) | ||
| Online open book | z = 0.41 (p = 0.682) | z = 1.15 (p = 0.250) | z = 0.81 (p = 0.418) |
Surprisingly, medical students preferred a traditional closed-book examination over an online, open-book examination (62.2%). However, students generally believed (66.7%) that our approach was the best solution to prevent contagion during the COVID-19 pandemic.
Although an online, open-book examination was feasible during the COVID-19 pandemic, this study indicates that mean MCQ and essay scores may be higher and short answer scores lower than for closed-book examinations. Short answer scores had conversely higher correlation with GPAs than the others. This result is important for setting a comparable pass mark for closed- and open-book examinations and should be repeated at other institutions who have switched examination formats. Also, clerkships may want to use data from several open-book examinations and recalculate passing scores prior to assigning final grades. Further research is required to assess the downstream effects of online, open-book examinations on long-term knowledge retention and application.
References
- 1.Rose S. Medical student education in the time of COVID-19. JAMA. 2020 doi: 10.1001/jama.2020.5227. [DOI] [PubMed] [Google Scholar]
- 2.Durning S.J., Dong T., Ratcliffe T. Comparing open-book and closed-book examinations: a systematic review. Acad Med. 2016;91:583–599. doi: 10.1097/ACM.0000000000000977. [DOI] [PubMed] [Google Scholar]
- 3.Heijne‐Penninga M., Kuks J.B.M., Hofman W.H.A., Cohen‐Schotanus J. Influence of open- and closed-book tests on medical students’ learning approaches. Med Educ. 2008;42:967–974. doi: 10.1111/j.1365-2923.2008.03125.x. [DOI] [PubMed] [Google Scholar]