Abstract
Objective
This cross-sectional national study determined which educational approaches are associated with the effectiveness of online clerkship for medical students.
Method
A survey was conducted for medical students at 78 medical schools in Japan from May 29 to June 14, 2020. It comprised the following aspects: (a) participants' profiles, (b) number of opportunities to learn from each educational approach (lecture, medical quiz, assignment, oral presentation, observation of a physician's practice, clinical skill practice, participation in interprofessional meetings, and interactive discussions with physicians) in online clerkship, (c) frequency of technical problems, and (d) educational outcome measurement (satisfaction, motivation, knowledge acquisition, skill acquisition, change in self-study time, and understanding of the importance of medical care team).
Results
Of the 2,640 respondents, 2,594 (98.3%) agreed to cooperate. Ultimately, 1,711 matched our inclusion criteria. All educational approaches but assignments were positively associated with satisfaction and motivation. All educational approaches excluding assignment submission and interprofessional meeting were positively associated with knowledge acquisition. Observation, practice, and interprofessional meeting were positively associated with skill acquisition. Only assignment submission was positively associated with the change in self-study time. Educational approaches excluding medical quizzes were positively associated with understanding the importance of the medical care team. Technical problems were negatively associated with motivation, knowledge acquisition, and skill acquisition.
Conclusions
Educators should implement various educational approaches, especially observation and practice, even in online clinical clerkship. They also need to minimize the technical problems associated with the Internet, as they reduce the effectiveness of online clerkship.
Keywords: COVID-19, clinical clerkship, tele-education, online, medical education
Introduction
Coronavirus disease 2019 (COVID-19) has significantly affected societies globally, particularly restricting face-to-face interactions (1). COVID-19 has also greatly impacted medical education (2-6), specifically face-to-face clinical clerkship. Some medical schools have been forced to suspend clinical clerkships following a city lockdown or a national emergency declaration (7). Globally, online clinical clerkship was proposed to replace face-to-face clinical clerkship during the COVID-19 pandemic (mid-COVID-19) (7). Various opinions and practical innovations have been presented to make online clinical clerkship a valuable alternative to face-to-face clerkship (8-10). However, which educational approaches can best enhance online clinical clerkship is unclear at present.
This study surveyed Japanese medical students to examine practical educational approaches to improving online clinical clerkship. We hypothesized that educational approaches would enhance the effectiveness of online clinical clerkship mid-COVID-19. Therefore, we investigated which educational approaches were associated with online clinical clerkship effectiveness for medical students as an alternative to face-to-face clerkship.
Materials and Methods
Survey design and data collection
This study adhered to the principles of the Declaration of Helsinki and was approved by the ethics committee of Kyushu University. We conducted a cross-sectional survey study to achieve our research objective.
Our participants were undergraduate medical school students in Japan. The study was conducted from May 29 to June 14, 2020. We investigated medical education for the period from April to May 2020, when a state of emergency was declared in Japan. The questionnaire included an informative letter summarizing the purpose of the research and an informed consent form, which was written briefly and clearly in Japanese to avoid misinterpretation. The questionnaire took 3-5 minutes to complete and was prepared using Google Forms and distributed via the LINE application.
This study's inclusion criteria were students 1) who had switched from face-to-face to online clinical clerkship during COVID-19, 2) who were in their fifth or sixth year (final two years) in a Japanese medical school, and 3) who had completed all the survey questions.
Questions included in the survey
The questionnaire consisted of two parts. In the first part, participants were asked about their profile. The second part of the survey was conducted for medical students who had switched from face-to-face to online clinical clerkship mid-COVID-19. It asked students how they felt about the effectiveness of online clinical clerkship mid-COVID-19 compared with the face-to-face clerkship they had been part of before the COVID-19 pandemic (pre-COVID-19). It also asked them what they experienced via online clinical clerkship and how frequently they experienced it.
Participant profile
The first part of this survey asked about participants' profile details, including their sex, year in medical school, university, and education style. To incorporate the variable of whether or not an area was infected with COVID-19 into the analysis, we categorized the location of universities into two groups according to the number of COVID-19-positive cases as of May 28, 2020 (Fig. 1). We also asked participants about their self-study time pre-COVID-19 (Supplementary material 1).
Figure 1.
Number of COVID-19-positive people in each prefecture of Japan on May 28, 2020. Universities were categorized into two groups: universities in prefectures with high numbers (≥1,000) of COVID-19 patients (red bar) and universities in prefectures that did not have high numbers (<1,000) of COVID-19 patients (blue bar; https://web.sapmed.ac.jp/canmol/coronavirus/japan.html). Specifically, universities located in Tokyo, Osaka, Kanagawa, Hokkaido, and Saitama prefectures (red on the map) were defined as COVID-19-infected areas. COVID-19: coronavirus disease 2019
Exposure and other factors
We examined the extent of students' exposure to factors associated with the effectiveness of online clinical clerkship. We decided which factors should be included in our survey based on the previous literature and recommendations for medical education. We asked about factors such as the department into which students rotated for their online clinical clerkship. Concerning the department, they were categorized into two groups: departments that were considered as difficult to conduct online clinical clerkship and those that did not (11). Specifically, general surgery (12), neurosurgery (13), orthopedics (14), anesthesiology (15), and otolaryngology (16) were categorized as departments for which online clinical clerkship was considered difficult. Factors related to educational approaches included lecture duration (17-19), lecture frequency per week (17,20,21), and opportunities to take quizzes (22,23), submit assignments (24,25), give oral presentations (26,27), observe physician practices (8,28), practice clinical skills (8,28), participate in interprofessional meetings (28-30), and discuss with physicians interactively (28,31). In addition, we incorporated the frequency of technical problems related to the Internet as a variable (32). The details of the choices for each answer are presented in Supplementary material 1.
Outcome measurement (Table 1)
Table 1.
Educational Outcomes Based on Kirkpatrick’s Assessment Model.
| Kirkpatrick’s assessment model (34,36) | Outcome measurement in our study | |
|---|---|---|
| Level 1 | Learners’ satisfaction | Satisfaction level with online clinical clerkship compared with previous face-to-face clerkship |
| Level 2a | Changes in learner attitudes | Motivation level for online clinical clerkship compared with previous face-to-face clerkship |
| Level 2b | Measures of learner knowledge and skill | 1. Amount of knowledge acquired in online clinical clerkship compared with previous face-to-face clerkship |
| 2. Amount of skill acquired in online clinical clerkship compared with previous face-to-face clerkship | ||
| Level 3 | Changes in learner behavior | Change in self-study time between during face-to-face clerkship and during online clinical clerkship |
| Level 4a | Changes to clinical processes | NA |
| Level 4b | Benefits to patients | NA |
| NA | Understanding level of the importance of the medical care team in online clinical clerkship compared with previous face-to-face clerkship | |
NA: not applicable
Based on Kirkpatrick's model for assessing the usefulness of medical education and curriculum (30,33-35), we analyzed the following five outcome measurements of the effectiveness of online clinical clerkship: 1) degree of satisfaction, 2) degree of motivation, 3) amount of knowledge acquisition, 4) amount of skill acquisition, and 5) change in self-study time. We could not assess level 4a/4b (clinical processes/benefit to patients) in Kirkpatrick's assessment model, which is the highest level, because a long follow-up time is required to assess the clinical benefit for patients (30,35). In addition to these five outcomes, we incorporated the level of understanding the importance of the medical care team as the sixth outcome (36-38). Although this measurement was not incorporated into Kirkpatrick's evaluation model, an awareness of the importance of the medical care team is one of the important things medical students can learn via clinical clerkship.
We calculated the change in self-study time as the mid-COVID-19 time minus the pre-COVID-19 time. Answers to other questions were based on a 5-point Likert scale ranging from 1 (the level/amount during face-to-face clerkship was much higher than that during online clinical clerkship) to 5 (the level/amount during online clinical clerkship was much higher than that during face-to-face clerkship).
Total number of respondents and those included in this study (Fig. 2)
Figure 2.
A flowchart showing the selection of suitable patients for inclusion in this study. We collected responses from 2,640 Japanese medical students. Of the 2,640 respondents, 2,594 (98.3%) consented to cooperate in this study. Respondents with missing answers for their profiles were excluded from the analysis (n=122). In Japan, fifth- and sixth-year students have completed the equivalent years of practice, so we also excluded responses from students in fourth year or below (n=19). The next step was to categorize the responses with respect to the type of education received mid-COVID-19. Sixteen respondents were excluded for inappropriate response content. Of the 2,432 remaining respondents, 1,758 reported that they had experienced face-to-face clerkship in hospitals pre-COVID-19 and had switched to online clinical clerkship due to the COVID-19 pandemic. Of the 1,758 respondents, those with missing responses regarding the content of the education exposed were excluded (n=87). Finally, 1,671 responses were used for the analysis of this study. COVID-19: coronavirus disease 2019, pre-COVID-19: before the COVID-19 pandemic, mid-COVID-19: during the COVID-19 pandemic
Of the 2,640 Japanese medical students who viewed the questionnaire, 2,594 (98.3%) consented to cooperate with the study. The sample size was calculated based on a 99% confidence interval and a 5% margin of error. A total of 18,195 students enrolled in the fifth and sixth years of Japanese medical universities in 2020. The 2,594 students who agreed to participate was four-fold more than the required sample size of 642. Respondents with missing answers were excluded. Of the remaining respondents, we included those who answered that they had experienced face-to-face clerkship in hospitals pre-COVID-19 and had switched to online clinical clerkship due to the COVID-19 pandemic. Ultimately, 1,671 responses were analyzed (Fig. 2).
Statistical analyses
We used the SPSS ver. 27 software program (IBM, Armonk, USA) for the analysis. A two-sided p value of 0.05 was considered significant.
The linear mixed model analysis was used to determine which factors were associated with each outcome measurement of the effectiveness of online clinical clerkship mid-COVID-19. We assumed that the Likert scale provided linear ordinal data. Fixed effect predictors were (a) sex, (b) year in medical school, (c) location of university, (d) self-study time pre-COVID-19, (e) rotating departments with difficulty in online clinical clerkship, (f) lecture duration, (g) lecture frequency, (h) quiz, (i) assignment submission, (j) student oral presentation, (k) observation of practice, (l) clinical skill practice, (m) participation in interprofessional meeting, (n) interactive discussion with physicians, and (o) technical problems related to the Internet. Intercept and university were treated as random factors.
We conducted a sensitivity analysis to assess the robustness of our results (Supplementary material 2).
Patient and public involvement
Patients and/or the public were not involved in the design, conduct, reporting, or dissemination of this research.
Results
Summary of participants' profiles and their responses
Table 2 presents participants' profiles and the proportion of each answer to every question.
Table 2.
Summary of Participants’ Profiles and Answers.
| Participants (n=1,671) | |
| Profile of participants | |
| Sex, No. (%) | Male: 979 (58.6), Female: 692 (41.4) |
| Year in medical school, No. (%) | 5th: 707 (42.3), 6th: 964 (57.7) |
| Location of university, No. (%) | Infected area: 566 (33.9), Non-infected area: 1,105 (66.1) |
| Self-study time per day pre-COVID-19 (*1) No. (%) | 1: 307 (18.4), 2: 480 (28.7), 3: 418 (25.0), 4: 208 (12.4), 5: 107 (6.4), 6: 60 (3.6), 7: 43 (2.6), 8: 20 (1.2), 9: 8 (0.5), 10: 5 (0.3), 11: 8 (0.5), 12: 2 (0.1), 13: 5 (0.3) |
| Factors in web-based education | |
| Rotating departments with difficulty teaching online clinical clerkship or not, No. (%) | Yes: 1,104 (66.1), No: 567 (33.9) |
| Lecture duration (*2) No. (%) | 1: 287 (17.2), 2: 54 (3.2), 3: 139 (8.3), 4: 250 (15.0), 5: 406 (24.3), 6: 329 (19.7), 7: 118(7.1), 8: 88 (5.3) |
| Lecture frequency per week (*3) No. (%) | 1: 287 (17.2), 2: 141 (8.4), 3: 166 (9.9), 4: 138 (8.3), 5: 181 (10.8), 6: 139 (8.3), 7: 357 (21.4), 8: 61 (3.7), 9: 50 (3.0), 10: 28 (1.7), 11: 12 (0.7), 12: 63 (3.8), 13: 48 (2.9) |
| Opportunity to take quizzes (*4) No. (%) | 1: 357 (21.4), 2: 240 (14.4), 3: 753 (45.1), 4: 206 (12.3), 5: 115 (6.9) |
| Opportunity to submit assignments (*4) No. (%) | 1: 212 (12.7), 2: 171 (10.2), 3: 473 (28.3), 4: 382 (22.9), 5: 433 (25.9) |
| Opportunity to give oral presentations (*4) No. (%) | 1: 700 (41.9), 2: 370 (22.1), 3: 400 (23.9), 4: 119 (7.1), 5: 82 (4.9) |
| Opportunity to observe physician practices (*4) No. (%) | 1: 1,286 (77.0), 2: 238 (14.2), 3: 87 (5.2), 4: 25 (1.5), 5: 35 (2.1) |
| Opportunity to practice clinical skills (*4) No. (%) | 1: 1,250 (74.8), 2: 263 (15.7), 3: 124 (7.4), 4: 14 (0.8), 5: 20 (1.2) |
| Opportunity to participate in interprofessional meetings (*4) No. (%) | 1: 1,202 (71.9), 2: 244 (14.6), 3: 152 (9.1), 4: 37 (2.2), 5: 36 (2.2) |
| Opportunity to discuss with physicians interactively (*4) No. (%) | 1: 798 (47.8), 2: 382 (22.9), 3: 303 (18.1), 4: 108 (6.5), 5: 80 (4.8) |
| Frequency of technical problems related to the Internet (*5) No. (%) | 1: 575 (34.4), 2: 416 (24.9), 3: 296 (17.7), 4: 297 (17.8), 5: 87 (5.2) |
| Outcome measurement | |
| Satisfaction level (*6) No. (%) | 1: 517 (30.9), 2: 570 (34.1), 3: 348 (20.8), 4: 138 (8.3), 5: 98 (5.9) |
| Motivation level (*6) No. (%) | 1: 543 (32.5), 2: 525 (31.4), 3: 381 (22.8), 4: 108 (6.5), 5: 114 (6.8) |
| Acquired knowledge (*6) No. (%) | 1: 300 (18.0), 2: 503 (30.1), 3: 475 (28.4), 4: 260 (15.6), 5: 133 (8.0) |
| Acquired clinical skill (*6) No. (%) | 1: 1,093 (65.4), 2: 363 (21.7), 3: 146 (8.7), 4: 35 (2.1), 5: 34 (2.0) |
| Change in self-study time (*7) No. (%): | -12: 1 (0.1), -11: 0 (0), -10: 1 (0.1), -9: 1 (0.1), -8: 1 (0.1), -7: 2 (0.1), -6: 4 (0.2), -5: 2 (0.1), -4: 10 (0.6), -3: 25 (1.5), -2: 45 (2.7), -1: 105 (6.3), 0: 302 (18.1), 1: 321 (19.2), 2: 342 (20.5), 3: 206 (12.3), 4: 129 (7.7), 5: 100 (6.0), 6: 31 (1.9), 7: 13 (0.8), 8: 12 (0.7), 9: 7 (0.4), 10: 6 (0.4), 11: 1 (0.1), 12: 4 (0.2) |
| Understanding level of the importance of the medical care team (*6) No. (%) | 1: 901 (53.9), 2: 414 (24.8), 3: 277 (16.6), 4: 37 (2.2), 5: 42 (2.5) |
*1: 13 choices in hourly increments: 1) <one hour per day, 2) ≥one hour and <two hours per day, 3) ≥two hours and <three hours per day, 4) ≥three hours and <four hours per day, 5) ≥four hours and <five hours per day, 6) ≥five hours and <six hours per day, 7) ≥six hours and <seven hours per day, 8) ≥seven hours and <eight hours per day, 9) ≥eight hours and <nine hours per day, 10) ≥nine hours and <10 hours per day, 11) ≥10 hours and <11 hours per day, 12) ≥11 hours and <12 hours per day, and 13) over 12 hours per day.
*2: 8 choices in 15-minute increments: 1) not at all, 2) ≤15 minutes per lecture, 3) ≥15 minutes and ≤30 minutes per lecture, 4) ≥30 minutes and ≤45 minutes per lecture, 5) ≥45 minutes and ≤60 minutes per lecture, 6) ≥60 minutes and ≤75 minutes per lecture, 7) ≥75 minutes and ≤90 minutes per lecture, and 8) ≥90 minutes per lecture.
*3: 13 choices: 1) not at all, 2) less than once a week, 3) about once a week, 4) about twice a week, 5) about three times a week, 6) about four times a week, 7) about five times a week, 8) about six times a week, 9) about seven times a week, 10) about eight times a week, 11) about nine times a week, 12) about 10 times a week, and 13) over 10 times a week.
*4: 5-point Likert scale: 1) the opportunities during face-to-face clerkship were much more than during online clinical clerkship to 5) the opportunities during online clinical clerkship were much more than during face-to-face clerkship.
*5: 5-point Likert scale: 1)=not at all, to 5)=very frequent.
*6: 5-point Likert scale: 1) the level/amount during face-to-face clerkship was much higher than that during online clinical clerkship to 5) the level/amount during online clinical clerkship was much higher than that during face-to-face clerkship.
*7: Subtraction of self-study time during face-to-face clerkship from that during online clinical clerkship
*pre-COVID-19: before the COVID-19 pandemic
Of the 1,671 participants included in the analysis, 692 (41.4%) were women, and 964 (57.7%) were in their final year of medical education. Overall, 1,105 (66.1%) students attended universities located in not COVID-19-infected areas. Concerning self-study time, 1,205 (72.1%) students studied fewer than 3 hours per day during the pre-COVID-19 clerkship period. Regarding department rotation, 1,104 (66.1%) students rotated within departments with difficulty in conducting online clinical clerkship. The most common lecture duration was ≥45 minutes and ＜60 minutes, with 406 (24.3%) students' responses falling within this time range. The most common lecture frequency was 5 times a week, with 357 (21.4%) students selecting this option. The number of students who believed that online clinical clerkship was more effective than face-to-face clerkship regarding the 5 outcome measures analyzed were as follows: satisfaction, 236 (14.2%); motivation, 208 (13.3%); acquired knowledge, 393 (23.6%); acquired skill, 69 (4.2%); and understanding the importance of the medical care team, 79 (4.9%).
Factors associated with the degree of satisfaction with online clinical clerkship
Supplementary material 3 shows the results of the mixed model analysis for degree of satisfaction with online clinical clerkship. The positive factors associated with satisfaction were gender (estimate=-0.12; p=0.012), year in medical school (estimate=0.21; p＜0.001), lecture frequency (estimate=0.04; p＜0.001), quizzes (estimate=0.10; p＜0.001), student oral presentation (estimate=0.08; p=0.003), observation of practice (estimate=0.17; p＜0.001), clinical skill practice (estimate=0.17; p＜0.001), participation in interprofessional meeting (estimate=0.15; p＜0.001), and interactive discussion with physicians (estimate=0.15; p＜0.001). The only associated negative factor was self-study time pre-COVID-19 (estimate=-0.03; p=0.028).
Factors associated with the degree of motivation for online clinical clerkship
Supplementary material 4 shows the results of the mixed model analysis for degree of motivation for online clinical clerkship. The positive factors associated with motivation were gender (estimate=-0.11; p=0.037), year in medical school (estimate=0.15; p=0.006), lecture frequency (estimate=0.02; p=0.033), quizzes (estimate=0.11; p＜0.001), student oral presentation (estimate=0.11; p＜0.001), observation of practice (estimate=0.19; p＜0.001), clinical skill practice (estimate=0.15; p＜0.001), participation in interprofessional meeting (estimate=0.13; p＜0.001), and interactive discussion with physicians (estimate=0.13; p＜0.001). The associated negative factors were self-study time pre-COVID-19 (estimate=-0.03; p=0.021) and technical problems related to the Internet (estimate=-0.07; p＜0.001).
Factors associated with the amount of acquired knowledge and clinical skill
Supplementary material 5 shows the results of the mixed model analysis for the amount of knowledge acquired in online clinical clerkship. The positive factors associated with acquired knowledge were year in medical school (estimate=0.12; p=0.028), lecture frequency (estimate=0.04; p＜0.001), quiz (estimate=0.17; p＜0.001), student oral presentation (estimate=0.09; p=0.001), observation of practice (estimate=0.22; p＜0.001), clinical skill practice (estimate=0.09; p=0.032), and interactive discussion with physicians (estimate=0.11; p＜0.001). The associated negative factors were self-study time pre-COVID-19 (estimate=-0.03; p=0.023) and technical problems related to the Internet (estimate=-0.05; p=0.011).
Supplementary material 6 shows the results of the mixed model analysis for the amount of clinical skill acquired in online clinical clerkship. The associated positive factors were rotating departments with difficulty in online clinical clerkship (estimate=0.08; p=0.024), observation of practice (estimate=0.26; p＜0.001), clinical skill practice (estimate=0.52; p＜0.001), and participation in interprofessional meeting (estimate=0.07; p=0.009).
Factors associated with the change in self-study time
Supplementary material 7 shows the results of the mixed model analysis for the change in self-study time. The associated positive factors were year in medical school (estimate=1.06; p＜0.001) and assignment submission (estimate=0.10; p=0.026). The only associated negative factor was self-study time pre-COVID-19 (estimate=-0.56; p＜0.001).
Factors associated with understanding the importance of the medical care team
Supplementary material 8 shows the results of the mixed model analysis for the understanding level of the importance of the medical care team. The associated positive factors were lecture frequency (estimate=0.02; p=0.030), assignment submission (estimate=0.04; p=0.031), student oral presentation (estimate=0.07; p＜0.001), observation of practice (estimate=0.25; p＜0.001), clinical skill practice (estimate=0.24; p＜0.001), participation in interprofessional meeting (estimate=0.16; p＜0.001), and interactive discussion with physicians (estimate=0.08; p＜0.001). The only associated negative factor was longer self-study time pre-COVID-19 (estimate=-0.02; p=0.034).
Fig. 3 summarizes the 95% confidence intervals of the mixed model estimate of each online clinical clerkship factor at each level based on Kirkpatrick's assessment model.
Figure 3.
Graphic summary of the 95% confidence intervals of the estimates of the independent variables in the linear mixed model analysis for each outcome. Variables with red plots and bars, whose 95% confidence intervals of the estimate calculated with the linear mixed model analysis were over 0, represent the positive factors associated with each outcome. Variables with blue plots and bars, whose 95% confidence intervals of the estimate were under 0, represent the negative factors associated with each outcome. (A) Degree of satisfaction with online clinical clerkship compared with previous face-to-face clerkship. (B) Degree of motivation for online clinical clerkship compared with previous face-to-face clerkship. (C) Amount of knowledge acquired in online clinical clerkship compared with previous face-to-face clerkship. (D) Amount of skill acquired in online clinical clerkship compared with previous face-to-face clerkship. (E) Change in self-study time between during online clinical clerkship and during face-to-face clerkship. (F) Degree of understanding of the importance of the medical care team in online clinical clerkship compared with previous face-to-face clerkship. COVID-19: coronavirus disease 2019
Results of a sensitivity analysis
Supplementary material 9 summarizes the 95% confidence intervals of the coefficients in the generalized linear mixed model analysis. The sensitivity analysis produced similar results to the original analysis of our study, thus confirming the robustness of our results.
Discussion
No previous studies have examined educational approaches that enhance the effectiveness of online practice. Our national-level cross-sectional study is the first to identify educational approaches in online clinical clerkship using Kirkpatrick's assessment model with 1,671 medical students in 87 medical schools across Japan.
Our study showed that lecture frequency contributes to enhancing online clinical clerkship more than lecture duration. When transitioning from face-to-face lectures to an online form, a 3-hour face-to-face lecture needs to be summarized in a 30-minute video, as research shows that students get bored with videos if they are too long (39). In our study mid-COVID-19, lecture frequency was found to be related to satisfaction, motivation, the acquisition of medical knowledge, and understanding the importance of the medical care team. In short, our results support the above-mentioned research finding. In addition, an increased lecture frequency may have helped students feel more connected to society mid-COVID-19 (40). Our results indicate that it is more effective to divide lectures into small sessions and increase their frequency than to deliver a singly long lecture all at once in online clinical clerkship mid-COVID-19.
Quizzes, assignment submissions, and student oral presentations contributed in different manners to online clinical clerkship's effectiveness, although they all reflected knowledge output. Quizzes contributed to satisfaction, motivation, and knowledge the most among the three output activities (estimates: 0.10, 0.11, 0.17). Student oral presentations also contributed to satisfaction, motivation, and knowledge. However, student oral presentations differed from quizzes, as they contributed to an understanding of the importance of the medical care team. Unlike the other two, assignment submission contributed to an increase in self-study time, indicating that self-study is encouraged by assignments. Conversely, it did not contribute to satisfaction, motivation, knowledge acquisition, or skill acquisition. Considering the contribution of each of the above factors to online clinical clerkship, educators and universities should incorporate all three factors, keeping in mind the features of each educational approach. Specifically, student oral presentations can only be undertaken by one person/team at a time and are time-limited, as they can only be commenced once everyone is online and present. Quizzes, however, have fewer time restrictions, as they can be administered to all medical students at once and can be completed by each student at their own pace. Assignment submissions are less time-sensitive in this respect, similar to quizzes, but the burden on physicians to provide appropriate feedback to medical students is greater for assignment submissions than with the other two educational approaches. Therefore, it is possible that adequate feedback was not able to be provided to medical students, as physicians were under immense strain during the early phase of the pandemic. This may also be why assignment submissions did not contribute to medical student satisfaction and motivation.
Observation of practice and clinical skill practice are important approaches in both online clinical and face-to-face clerkship. These approaches have traditionally only been feasible with on-site practice (11), but mid-COVID-19, educators have attempted to provide them online (8-10). Our results showed that observation and practice similarly contributed to satisfaction, motivation, and the understanding of the importance of the medical care team (estimate, observation vs. practice: 0.17 vs. 0.17, 0.19 vs. 0.15, 0.25 vs. 0.24). These factors also contributed to knowledge and skill. However, observation contributed more to the acquisition of knowledge than practice (estimate: 0.22 vs. 0.09), and practice contributed more to the acquisition of skill than observation (estimate: 0.26 vs. 0.52). Participation in interprofessional meetings is also an important activity in online clinical clerkship. Our results showed that it contributed to satisfaction, motivation, the acquisition of skill, and understanding of the medical care team's importance. All three approaches were found to be important educational content in online clinical clerkship. Observation and practice contributed to the greatest number of educational outcomes. Considering the features of each educational approach online, practice would have been restricted by technical issues. Specifically, practice often requires the presence of certain people and special equipment, such as physical examination practices that use models; it can thus be difficult to practice properly in an online setting. Furthermore, for observation, issues such as the protection of patient privacy and the reliability of online tools must be considered when observing online. Finally, regarding participation in interprofessional meetings, in addition to ensuring the protection of patient privacy, the implementation of interprofessional meetings itself may have been restricted at each facility during the early phase of the pandemic.
Interactive discussions with a physician play a great role in clinical clerkship and online clinical clerkship. They contributed to satisfaction, motivation, knowledge acquisition, and understanding of the medical care team's importance. Interactive discussions with physicians would be easier to perform on an online setting than observation, practice, and participation in interprofessional meetings.
Online technical problems are some of the most feared obstacles in web education. They negatively contributed to motivation and knowledge acquisition. Educators and universities must implement interactive discussions between physicians and medical students and prevent technical problems related to the Internet.
Regarding profile factors, women were less satisfied and motivated by online clinical clerkship than men. A previous study concluded that men were more likely to express higher confidence in self-assessments than women (34). Our study may support this conclusion, although there were no significant differences between men and women in other outcomes. The final year in medical school was also positively associated with satisfaction, motivation, acquired knowledge, and change in self-study time. An interpretation of this observation is that students in their final year of medical school must be fully prepared to become doctors. Therefore, they put in the most effort to learn mid-COVID-19 (41). Their longer self-study time pre-COVID-19 was negatively associated with satisfaction, motivation, acquired knowledge, change in self-study time, and understanding of the importance of medical care team. Students who spend too long self-studying may have higher medical education expectations than those who spend a shorter time.
Regarding changes in self-study time, students who had spent less time on self-study pre-COVID-19 increased their self-study time mid-COVID-19. This results may have been influenced by the restriction of extracurricular activities (e.g. sports, part-time job) due to the lockdown. Although departments such as surgery reportedly find it difficult to educate students on the web because of the focus on hospital visits (11), rotating such departments helped improve students' clinical skills in our study, possibly due to these departments implementing innovations in educating students online (e.g. video observations of surgery) (8-10).
Methodological considerations
Several limitations associated with the present study warrant mention. First, this cross-sectional questionnaire survey captures subjects' experiences and perceptions at only one point in time, retrospectively. Thus, recall bias is unavoidable. In particular, with respect to the level 2 Kirkpatrick's assessment model, objective outcome measurements should be made pre-COVID-19, and a quantitative comparison should be conducted using the same outcome measurement mid-COVID-19. However, because the COVID-19 pandemic was unexpected, quantitative measurements could not be made pre-COVID-19. For this reason, our study design had to be a cross-sectional survey. Even though this study is based on self-reported outcomes, our results could contribute to future online education because self-reported outcomes, especially satisfaction and motivation, are important factors affecting the effectiveness of education. In fact, one of the interests of medical educators is how to maintain or enhance students' satisfaction and motivation for online education (42-44). Second, we focused on medical students in Japan. Future research must consider the applicability of these results to other countries. Online clinical clerkship may be less useful in countries with limited healthcare resources or a lack of an Internet infrastructure. The impact of COVID-19 also varies across countries. Our study compared the utility of online clinical clerkship mid-COVID-19 and face-to-face clerkship pre-COVID-19. Therefore, we excluded students who had not experienced face-to-face clerkship pre-COVID-19. Further studies are needed to determine whether or not similar results are found for students without any experience of face-to-face clerkship pre-COVID-19. In addition, some communities continued face-to-face clerkship in hospitals mid-COVID-19, so studies evaluating the effectiveness of clinical clerkship mid-COVID-19 will be necessary. Finally, we did not account for all confounding factors. For example, the educational system and content differ among universities and departments. In addition, the departments in which students rotated differed between pre- and mid-COVID-19 and could not be compared directly.
Conclusion
The present web-based national-level survey for 1,671 medical students clarified that educational approaches, especially observation and practice, made online clinical clerkship efficient. Our results may help improve the online clinical clerkship program. Educators should implement various educational approaches, even in online clinical clerkship. Furthermore, they should aim to minimize technical problems, as these reduce the effectiveness of online clinical clerkship.
The authors state that they have no Conflict of Interest (COI).
Financial Support
This work was supported by JSPS KAKENHI (Grant Number 17H04097).
Supplementary Materials
Detailed description of the choices for each answer.
Sensitivity analysis.
Linear mixed model analysis to characterize the satisfaction level of medical students with online clerkship (level 1 in Kirkpatrick’s assessment model).
Linear mixed model analysis to characterize the motivation level of medical students for online clerkship (level 2a in Kirkpatrick’s assessment model).
Linear mixed model analysis to characterize the amount of knowledge acquired in online clerkship (knowledge part of level 2b in Kirkpatrick’s assessment model).
Linear mixed model analysis to characterize the amount of clinical skill acquired in online clerkship (skill part of level 2b in Kirkpatrick’s assessment model).
Linear mixed model analysis to characterize the change in self-study time during online clerkship compared with during previous face-to-face clerkship pre-COVID-19 (level 3 in Kirkpatrick’s assessment model).
Linear mixed model analysis to characterize the understanding level of the importance of the medical care team in online clerkship.
Graphic summary of the 95% confidence interval of the coefficients of the independent variables in each outcome with the sensitivity analysis.
Acknowledgments
The authors are grateful to Eishi Asano at Children's Hospital of Michigan, Detroit Medical Center, Wayne State University for the advice on the statistical analysis. The authors are also grateful to all participants of this survey-based study.
References
- 1.Guan WJ, Ni ZY, Hu Y, et al. Clinical characteristics of coronavirus disease 2019 in China. N Engl J Med 382: 1708-1720, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Daniel M, Gordon M, Patricio M, et al. An update on developments in medical education in response to the COVID-19 pandemic: a BEME scoping review: BEME guide No. 64. Med Teach 43: 253-271, 2021. [DOI] [PubMed] [Google Scholar]
- 3.Gibbs T. The COVID-19 pandemic: provoking thought and encouraging change. Med Teach 42: 738-740, 2020. [DOI] [PubMed] [Google Scholar]
- 4.Gordon M, Patricio M, Horne L, et al. Developments in medical education in response to the COVID-19 pandemic: a rapid BEME systematic review: BEME Guide No. 63. Med Teach 42: 1202-1215, 2020. [DOI] [PubMed] [Google Scholar]
- 5.Tolsgaard MG, Cleland J, Wilkinson T, Ellaway RH. How we make choices and sacrifices in medical education during the COVID-19 pandemic. Med Teach 42: 741-743, 2020. [DOI] [PubMed] [Google Scholar]
- 6.Yang DY, Cheng SY, Wang SZ, et al. Preparedness of medical education in China: lessons from the COVID-19 outbreak. Med Teach 42: 787-790, 2020. [DOI] [PubMed] [Google Scholar]
- 7.Rose S. Medical student education in the time of COVID-19. JAMA 323: 2131-2132, 2020. [DOI] [PubMed] [Google Scholar]
- 8.Chao TN, Frost AS, Brody RM, et al. Creation of an interactive virtual surgical rotation for undergraduate medical education during the COVID-19 pandemic. J Surg Educ 78: 346-350, 2021. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Co M, Chu KM. Distant surgical teaching during COVID-19 - a pilot study on final year medical students. Surg Pract 24: 105-109, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Jack MM, Gattozzi DA, Camarata PJ, Shah KJ. Live-streaming surgery for medical student education - educational solutions in neurosurgery during the COVID-19 pandemic. J Surg Educ 78: 99-103, 2021. [DOI] [PubMed] [Google Scholar]
- 11.Khan S, Mian A. Medical education: COVID-19 and surgery. Br J Surg 107: e269, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Daodu O, Panda N, Lopushinsky S, Varghese TK Jr, Brindle M. COVID-19 - considerations and implications for surgical learners. Ann Surg 272: e22-e23, 2020. [DOI] [PubMed] [Google Scholar]
- 13.Guadix SW, Winston GM, Chae JK, et al. Medical student concerns relating to neurosurgery education during COVID-19. World Neurosurg 139: e836-e847, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Kogan M, Klein SE, Hannon CP, Nolte MT. Orthopaedic education during the COVID-19 pandemic. J Am Acad Orthop Surg 28: e456-e464, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Eusuf DV, England EL, Charlesworth M, Shelton CL, Thornton SJ. Maintaining education and professional development for anaesthesia trainees during the COVID-19 pandemic: the self-isolating virtual education (SAVEd) project. Br J Anaesth 125: e432-e434, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Ruthberg JS, Quereshy HA, Ahmadmehrabi S, et al. A Multimodal multi-institutional solution to remote medical student education for otolaryngology during COVID-19. Otolaryngol Head Neck Surg 163: 707-709, 2020. [DOI] [PubMed] [Google Scholar]
- 17.Mansouri M, Lockyer J. A meta-analysis of continuing medical education effectiveness. J Contin Educ Health Prof 27: 6-15, 2007. [DOI] [PubMed] [Google Scholar]
- 18.Mustafa T, Farooq Z, Asad Z, et al. Lectures in medical education: what students think? J Ayub Med Coll Abbottabad 26: 21-25, 2014. [PubMed] [Google Scholar]
- 19.Hashmi NR, Daud S, Manzoor I. Medical education: views and recommendations by final year MBBS students of a private medical college in Lahore. J Coll Physicians Surg Pak 20: 93-97, 2010. [PubMed] [Google Scholar]
- 20.Najmi RS. Lecture as a mode of instruction in undergraduate medical education. J Pak Med Assoc 49: 30-33, 1999. [PubMed] [Google Scholar]
- 21.Manzoor I, Mumtaz A, Habib M, Tariq S, Elahee M, Javaid I. Lectures in medical education: students' views. J Ayub Med Coll Abbottabad 23: 118-121, 2011. [PubMed] [Google Scholar]
- 22.Karpicke JD, Roediger HL 3rd. The critical importance of retrieval for learning. Science 319: 966-968, 2008. [DOI] [PubMed] [Google Scholar]
- 23.Larsen DP, Butler AC, Roediger HL 3rd. Repeated testing improves long-term retention relative to repeated study: a randomised controlled trial. Med Educ 43: 1174-1181, 2009. [DOI] [PubMed] [Google Scholar]
- 24.Epp S. The value of reflective journaling in undergraduate nursing education: a literature review. Int J Nurs Stud 45: 1379-1388, 2008. [DOI] [PubMed] [Google Scholar]
- 25.August E, Burke K, Fleischer C, Trostle JA. Writing assignments in epidemiology courses: how many and how good? Public Health Rep 134: 441-446, 2019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Plack MM, Driscoll M, Marquez M, Greenberg L. Peer-facilitated virtual action learning: reflecting on critical incidents during a pediatric clerkship. Acad Pediatr 10: 146-152, 2010. [DOI] [PubMed] [Google Scholar]
- 27.Singh K, Bharatha A, Sa B, Adams OP, Majumder MAA. Teaching anatomy using an active and engaging learning strategy. BMC Med Educ 19: 149, 2019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Reed T, Pirotte M, McHugh M, et al. Simulation-based mastery learning improves medical student performance and retention of core clinical skills. Simul Healthc 11: 173-180, 2016. [DOI] [PubMed] [Google Scholar]
- 29.Mann KV, Mcfetridge-Durdle J, Martin-Misener R, et al. Interprofessional education for students of the health professions: the “seamless care” model. J Interprof Care 23: 224-233, 2009. [DOI] [PubMed] [Google Scholar]
- 30.Peiris-John R, Selak V, Robb G, et al. The state of quality improvement teaching in medical schools: a systematic review. J Surg Educ 77: 889-904, 2020. [DOI] [PubMed] [Google Scholar]
- 31.Alsoufi A, Alsuyihili A, Msherghi A, et al. Impact of the COVID-19 pandemic on medical education: medical students' knowledge, attitudes, and practices regarding electronic learning. PLoS One 15: e0242905, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Dost S, Hossain A, Shehab M, Abdelwahed A, Al-Nusair L. Perceptions of medical students towards online teaching during the COVID-19 pandemic: a national cross-sectional survey of 2721 UK medical students. BMJ Open 10: e042378, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Kirkpatrick DL. Evaluation of training. In: Training and Development Handbook. Craig RL, Bittel LR, Eds. McGraw-Hill, New York, 1967: 87-112. [Google Scholar]
- 34.Colthart I, Bagnall G, Evans A, et al. The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10. Med Teach 30: 124-145, 2008. [DOI] [PubMed] [Google Scholar]
- 35.Wong BM, Etchells EE, Kuper A, Levinson W, Shojania KG. Teaching quality improvement and patient safety to trainees: a systematic review. Acad Med 85: 1425-1439, 2010. [DOI] [PubMed] [Google Scholar]
- 36.Kearney AJ. Facilitating interprofessional education and practice. Can Nurse 104: 22-26, 2008. [PubMed] [Google Scholar]
- 37.Wright MC, Phillips-Bute BG, Petrusa ER, Griffin KL, Hobbs GW, Taekman JM. Assessing teamwork in medical education and practice: relating behavioural teamwork ratings and clinical performance. Med Teach 31: 30-38, 2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Morrison G, Goldfarb S, Lanken PN. Team training of medical students in the 21st century: would Flexner approve? Acad Med 85: 254-259, 2010. [DOI] [PubMed] [Google Scholar]
- 39.Gewin V. Five tips for moving teaching online as COVID-19 takes hold. Nature 580: 295-296, 2020. [DOI] [PubMed] [Google Scholar]
- 40.Lyons Z, Wilcox H, Leung L, Dearsley O. COVID-19 and the mental well-being of Australian medical students: impact, concerns and coping strategies used. Australas Psychiatry 28: 649-652, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Çalışkan F, Mıdık Ö, Baykan Z, et al. The knowledge level and perceptions toward COVID-19 among Turkish final year medical students. Postgrad Med 132: 764-772, 2020. [DOI] [PubMed] [Google Scholar]
- 42.Jiang Z, Wu H, Cheng H, Wang W, Xie A, Fitzgerald SR. Twelve tips for teaching medical students online under COVID-19. Med Educ Online 26: 1854066, 2021. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Rahm AK, Töllner M, Hubert MO, et al. Effects of realistic e-learning cases on students' learning motivation during COVID-19. PLoS One 16: e0249425, 2021. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Elshami W, Taha MH, Abuzaid M, Saravanan C, Al Kawas S, Abdalla ME. Satisfaction with online learning in the new normal: perspective of students and faculty at medical and health sciences colleges. Med Educ Online 26: 1920090, 2021. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Detailed description of the choices for each answer.
Sensitivity analysis.
Linear mixed model analysis to characterize the satisfaction level of medical students with online clerkship (level 1 in Kirkpatrick’s assessment model).
Linear mixed model analysis to characterize the motivation level of medical students for online clerkship (level 2a in Kirkpatrick’s assessment model).
Linear mixed model analysis to characterize the amount of knowledge acquired in online clerkship (knowledge part of level 2b in Kirkpatrick’s assessment model).
Linear mixed model analysis to characterize the amount of clinical skill acquired in online clerkship (skill part of level 2b in Kirkpatrick’s assessment model).
Linear mixed model analysis to characterize the change in self-study time during online clerkship compared with during previous face-to-face clerkship pre-COVID-19 (level 3 in Kirkpatrick’s assessment model).
Linear mixed model analysis to characterize the understanding level of the importance of the medical care team in online clerkship.
Graphic summary of the 95% confidence interval of the coefficients of the independent variables in each outcome with the sensitivity analysis.