Abstract
Background
There is an urgent need for more diverse methods for student evaluation, given the sudden shift to online learning necessitated by the coronavirus disease 2019 (COVID-19) pandemic. Innovative assessment tools will need to cover the required competencies and should be used to drive self-learning. Self-assessments and peer assessments may be added to the traditional classroom-based evaluations to identify individual insecurities or overconfidence. Identification of these factors is essential to medical education and is a focus of current research.
Methods
A modified operational assessment was introduced for the evaluation of third-year medical students. This intervention has facilitated sustained education and has promoted interactive student learning. Members of the entering class of 2017 participated in an integrated team and a competency-based online project that involved innovative item creation and case presentation methods.
Results
The new assessment process has been implemented successfully with positive feedback from all the participants; a usable product has been generated.
Conclusions
We created new assessment tools in response to the COVID-19 pandemic that have been used successfully at our institution. These tools have provided a framework for integrated and interactive evaluations that can be used to facilitate the modification of traditional assessment methods.
Keywords: Assessment, e-assignment, Competency-based assessment, Self-assessment, Student peer assessment
Introduction
The outbreak of coronavirus disease 2019 (COVID-19) was declared a pandemic by the World Health Organization on March 12, 2020. In response to this declaration, on March 15, 2020, the central authority in Egypt announced several important and necessary actions, including the closure of schools, universities, sporting clubs, gyms, museums, and public gardens. Similarly, arrangements were put in place to limit the number of employees working on-site at government institutions; employees were instructed to work from home. The unplanned closure of all medical schools imposed a substantial challenge as it created the need for a complete and obligatory shift to online teaching and learning. As such, there was an urgent need to remodel all medical school curricula and to include novel and creative methods for student teaching and assessment.1 Of interest, even before this date, the medical school curriculum reform worldwide focused on and emphasized the need for independent learning, development of interpersonal skills, and improved capacity for problem-solving.2
The sudden shift from face-to-face to fully online education and assessment carried many challenges for the faculty, students, and administration. One of the main challenges was determining how to develop online assessment tools for curricula that had been designed to accommodate traditional assessments.3 The transition to online assessments was inevitable. It is expected to continue into the post–COVID-19 era owing to the limited number of staff members and insufficient infrastructure and educational resources at our medical school.4
Case-based learning (CBL) had been in use for several years and was designed to promote a deeper understanding and drive self-learning. This type of curriculum provided specific and proactive student preparation before their preclerkship years and additional phases of their clinical training. Despite the disruptions associated with the COVID-19 pandemic, a set of learning objectives must be efficiently conveyed, and student learning needed to be assessed.5
With respect to the design of the integrated curriculum, enhanced guidance must be provided during online learning sessions. Adherence to learning objectives and the need for suitable methods for student assessment are both essential features of any new program.6 To achieve this goal, a CBL approach is to be adapted to the online format. Online interactive CBL sessions were delivered to students, and video cases of virtual patients were introduced as critical components of interactive seminars that required student participation and engagement.
CBL had the advantage of flexibility and the capacity to cover numerous integrated topics. The learning objectives of each CBL session were stated clearly. As such, acquisition of skills and knowledge related to these objectives were assessed clearly and directly. Training provided to both students and tutors was the cornerstone of online CBL success.
In this study, a hands-on experience is described adopting these new learning modalities. Case item creation and case presentation videos were implemented as methods for evaluation that were generated with reference to Kirkpatrick's hierarchy of learning evaluation.7 Competencies (health-care providers, health-care promoters, professionals, scholars and scientists, a member of the health team and a part of the lifelong learner, and researchers) must be articulated and mapped to perform directed, competency-based assessments.8 The design of these learning strategies should define expectations with respect to skills that can be measured specifically for each task. A wider scope of competency was achieved with incorporation of new abilities and knowledge.8 Moreover, team-based assignments included student peer assessments that added value to the process.9,10
Material and methods
Aims
This article presents an innovative assessment tool as a post–COVID-19 adaptation that can be used in the post–COVID-19 era as an ancillary assessment method.
Objectives
Primary objectives
The primary objective was to implement an alternative reliable assessment tool that can replace the currently unattainable traditional methods during the pandemic.
Secondary objectives
The secondary objective was to gain a usable product by investing on the creativity and productivity of young brilliant medical students (aged 20–21 years).
Curriculum design
The adopted integrated modular curricula were targeted at a multidisciplinary level,11 including CBL through an enquiry-based approach during the prepandemic era. Objectives were aligned to the skills and competencies required.3 However, a complete shift to online learning was adopted in light of the ongoing COVID-19 pandemic.
We applied a competency-based plan; a competency matrix was mapped. The targeted competences aimed at preparing the medical student as a health-care provider, a health-care promoter, a professional, a scholar and scientist, a member of the health team and a part of the lifelong learner, and a researcher. We aligned competency-based training with the assessment tools via the methodologies implemented.
Integrated, interactive CBL sessions were delivered, wherein the case scenario was presented online; the students were asked to answer it individually through a link of an online poll. The results of the poll were discussed starting from the least chosen option to the most chosen one.
Poster seminars, virtual patient seminars, and video creation that included case scenarios on integrated topics had all been implemented as educational tools throughout the module (Fig. 2).
Fig. 2.
Flowchart depicting the steps of e-assignment design. CBL = case-based learning.
Assignment design
Encouraging students to participate in active learning with a higher level of engagement in the learning process was achieved via written case scenarios and preparing videos. Both written case studies and video simulations were ways to engage students and encourage active learning.3,11 Students were also asked to include histological images in their case studies; this provided students with a means to connect normal and abnormal histological features with questions addressed by clinically integrated problem-solving.11
We agreed to formulate a project as the final summative assessment module for the third-year students (203 students), including an integrated e-assignment.
Two modes were implemented, specifically, case item and video creation. The following steps were involved:
-
i.
Training sessions on the initiative modality were delivered for staff members (Supplementary file 1).
-
ii.
A template for case scenarios was created.
-
iii.
The project was fragmented into smaller tasks with a defined role for each student within the team; this had facilitated evaluation of individual students.
-
iv.The rubrics were set with task/grade compatibility clearly defined:
-
-Respect the structure of the case scenario MCQ (stem, lead-in, distractors, and key).
-
-Respect the structure and integrity of the case.
-
-The case should be aligned with the delivered intended learning outcomes (ILOs).
-
-Stick to question formatting in the lead-in.
-
-The distractors and the key should avoid the common flaws presented.
-
-The case should be a relevant case, consistent scenario, and as per the presented template.
-
-
-
v.
An open online discussion with students was prepared to define the purpose of the assessment; the output and anticipated final outcomes were both explained carefully. The students were provided with examples in the form of previous output from their colleagues (the created students' Q-bank) (Supplementary file 2).
-
vi.
Students were engaged in planning to ensure sustainability.
Implementation
-
i.
Biochemistry, histology, microbiology, and pathology departments asked the third-year medical students to create mini-case scenarios covering the objectives of the previous module.
-
ii.
Two demo sessions for the staff members were delivered by a delegate of the medical education unit. Online meetings on Zoom Cloud (https://zoom.us) were scheduled, with each session lasting 90 min.
-
iii.
It is worth mentioning that a pilot study was previously performed by the Department of Pathology. It was in the form of a summer elective project that provided students with practice in generating case scenario creation3 (Supplementary file 2).
-
iv.
Demo for students: The contributing departments provided mentorship for students via a preorganized schedule. Using online meetings on Zoom Cloud, the project rubrics were discussed thoroughly with the students. Some reading resources and materials were suggested as well. The meetings were recorded and sent to students who were unable to participate in the Zoom Cloud session.
Delivering the material to students with suitable mentorship
-
•
Item creation assignment: The students received the file including instructions, rubrics, and templates to be used to prepare each case scenario. Each student was assigned a specified number of cases on distinct topics associated with the integrated objectives. Mentorship was provided throughout the process. The final case presentations were then uploaded. Worth mentioning, the mentorship provided was to guide and advise students through the preparatory phase of their project and facilitate any difficulty encountered. However, this did not include providing aid during case or video production.
-
•
Video creation assignment: After the meeting, students were asked to organize themselves into groups of 10. Each group chose its own team leader; a focus group was created with 20 team leaders. A WhatsApp group was created for this focus group that included two staff members as facilitators (https://www.whatsapp.com/). Mentorship was provided for three consecutive days to help students navigate through any difficulties they encountered. Students were asked to present their cases through videos that were uploaded with the assignment as an integral part of the project. Videos could be presented in the form of a virtual patient approach. The Skillgym tool (https://kbank.skillgym.biz/) was used to provide guidance to students. Animation video case presentation was also suggested as an option for case presentations. An audio-visual presentation using the Zoom Cloud application was suggested as well. It is worth mentioning that for the preceding two months (March and April 2020), students were preparing, presenting, and attending online seminars under the supervision of faculty members as part of adaptation to the lockdown required by the pandemic. Via these seminars, students presented their case studies using the virtual patient tool.
Assessment
Assessments were as follows:
-
•
A committee of three instructors was assigned for the assessment and scoring of the assignments.
-
•
The case reports were checked for plagiarism.
-
•
Individual evaluation of each student was performed by focusing on individual assigned tasks.
-
•
For video creation, the group was assessed.
-
•
This method of assessment was used as a pass or fail method.
-
•
A sample of the students' cases was included (Supplementary file 3).
Evaluation
After the primary objectives/outcomes of the assessment were achieved, the secondary objectives/outcomes were evaluated using a Kirkpatrick model (Fig. 3).
-
•
The usable educational product was passed through a multistep reviewing process before it was added to the online platform (Q-bank and video playlist).
-
•
A triangular approach with multiple data collection tools (questionnaires/focus groups/scores) was adopted.
-
•
Evaluation of the students took place via the assessment.
-
•
Evaluation of the staff was carried out using questionnaires.
-
•
Evaluation of the product involved a multistep approach, including instructor review, peer review by three students, instructor evaluation of the review, and final editing and reviewing before publication on the Q-bank and educational video playlist.
-
•
Finally, item analysis was conducted using Q-bank software (after validation), which included difficulty and discrimination indices. This was achieved by arranging an online quiz competition wherein the cases undergoing review were featured.
Fig. 3.
Kirkpatrick evaluation model.
For questionnaire validation, the questions were delivered to a sample of students (n = 6); then, a focus group was performed for evaluation and the students' comments were considered based on the recommendations resulting from the discussion.
The results of the evaluation were tabulated and used for corrective actions (Fig. 1).
Fig. 1.
Flowchart depicting the process of integrating intended learning outcomes (ILOs) and teaching methodology with assessment and evaluation.
Experiences associated with the e-integrated, team-based assignments were disseminated. These assignments were planned as a part of the integrated module for student training during the post–COVID-19 era as well. Broadening the scope of the objectives in a spiral approach to create interwoven products to be used by the increasing number of the students enrolled was recommended.
Results
Results of the assessment
The students created case scenarios to provide a clinical context. All the students (100%) uploaded their project components (case scenarios and videos) before the final deadline. This method of assessment was used as a pass-or-fail method. One hundred eighty-seven of 203 students had passed the assessment based on fulfilling 60% of the rubrics.
Results of student feedback on the assignment
Forty percent of the students responded to the questionnaire. Eight questions were provided that required responses on a 5-point Likert scale; a final open-ended question was also included (Table 1, Fig. 4a–c).
Table 1.
Results of student feedback on the integrated assignment.
| Question | Average rating on the Likert scale |
|---|---|
|
4 |
|
4 |
|
4 |
|
∼3.2 |
|
∼3.2 |
|
∼3.7 |
|
∼3.2 |
|
∼3.8 |
Fig. 4.
(a–c) Results of the questionnaire.
As shown, the overall student feedback was positive. Among the specific points to be considered further, the students asked for more effective mentorship. Some of the students identified technical difficulties when attempting to create a video. The following is a sample of some of the student comments:
“I would like to mention that I loved the feeling of being able to create a full case scenario.”
“It was a very nice experience, and I would like to participate in it again.”
“I hope our faculty will use this method instead of the regular old assignments.”
“I think the students will be more passionate if they believe that these cases will be used.”
“We need more effective mentorship and more flexibility and freedom regarding the template provided.”
“Grading is subjective.”
“Some notes were provided to some students but not to the others (i.e., pointers that include a mention of “all of the above” and “none of the above” are considered flaws). This is not fair to all students.”
“We need more reference material for some subjects.”
Evaluation of the product
We were very impressed with the unique, well-structured, problem-based case scenarios prepared by some of the students; these could be added to the Problem based learning (PBL) bank (Supplementary file 3).
The video presentations revealed very creative informative case discussions using virtual patient methodology, animation, and traditional student presentations. We noted from the virtual patient simulations that some of our students were not fully proficient at taking a patient history; this drew our attention toward corrective measures that might be implemented in the near future. Some of the students had modified their project, notably the parts associated with the video presentation. These modifications were approved by the tutors.
Thirty percent of the case items created was found to be usable and ready for the multistep review process that will take place during the students' summer project. About 25% of the presented videos were professionally created and represented a usable product, which was uploaded to the learning management system as educational resources.12
Discussion
The third-year medical students were able to begin work on their item creation project immediately at the start of the lockdown because the rubrics had already been clarified, and the required mentorship was provided. The instructor performed regular monitoring of the uploaded material to ensure that the students were following the specified rubrics.
An annual summer elective project in which interested students had participated in scientific activities, including case scenario writing workshops, provided the experience to conduct this innovative assessment strategy as an alternative to the traditional summative assessment, which was not able to be conducted during the post–COVID-19 era.3
Appropriate assessment represents the core of the learning process as it leads to important learning outcomes through student motivation and direction. The implementation of integrated assessment methodologies is regarded as an integral part of the current process of medical education.3 Several strategies can be applied toward the effort to select appropriate assessment tools that will facilitate the testing of multiple competencies despite social distancing. The main purpose of creating and applying this mini-case project assessment tool was that it not only covered almost all the competencies required by the National Academic Reference Standards but also was very suitable for online use. This not only will be a critical feature during quarantine periods but also might be a suitable learning modality, given the often unexpected increase in the number of students at our school each year.
The utilization of interactive platforms, including Zoom and Microsoft Teams (Microsoft Inc. 2017), in addition to online objective structured clinical examination served as an alternative solution to traditional examinations. It minimized the need for physical interactions between the student and the examiner. Our assessment tool was quite innovative in comparison with traditional modalities, which typically assessed only a limited number of competencies.
Our goal was to provide a reliable and valid method for the final summative examination during the COVID-19 pandemic. The positive feedback received from most of the students suggested the more general applicability of the case and creation items as tools for student assessment. The fact that many students had previous experience with the same tools during their formative assessments played a significant role in their use in the final summative assessment.
Continuous modifications can be applied to the already existing assessment tools; as such, they can be tailored and refined to facilitate the evaluation of additional competencies and/or can help students acquire new skills.13 Innovations and presentation of alternative tools for assessment as part of the process of adapting to instability is termed an authentic assessment. Such creative methods stimulate student performance as they are asked to present knowledge in ways that simulate situations that they will ultimately face in reality.14 In our situation, third-year medical students were assigned an item creation project and asked to prepare a module featuring an objective-based mini-case. This project was used as an alternative assessment tool and replaced the final examination that would ordinarily take place at the end of the module.
As previously stated by Hart,15 assessment planning should include a blended combination of performance assessment, portfolios, and systematic observations. In this study, students were asked to present prepared cases via a video as an integral part of their project. Videos were created using the virtual patient approach and were guided by the Skillgym tool.16
We would also like to mention that for the preceding two months (March and April 2020), students were preparing, presenting, and attending online seminars under the supervision of faculty members as part of the adaptation to the lockdown necessitated by the COVID-19 pandemic. Through these seminars, students had presented their case studies using the virtual patient tool. These formative assessments implemented throughout the lockdown were the main drivers promoting the successful implementation of these tools for summative assessments.17
It was perhaps no surprise that after a team brainstorming session, some students suggested simulating the virtual patient videos and creating the patient–doctor dialog by themselves. Their suggestions were accepted as part of our acknowledgment of their creative thinking for this project. As this project has become a cornerstone in the educational process, the learner must also be an active contributor capable of interacting enthusiastically with teachers in a changing environment.18
The design of clear, specific rubrics for the assessment project was an indispensable part of this process. Rubrics ensure that our assessments were consistent and had provided an established guide for students who were using this new assessment tool. The students responded with enthusiasm to the project and began working immediately. The students divided themselves into teams; each team received its assigned objectives with respect to case scenarios and required video presentations. With mentorship from the faculty and staff, each team prepared case scenarios and video presentations that included virtual patient simulations, animations, or case discussions that were presented using the Zoom application. A feedback survey was designed using Google Forms; all students were invited to give their opinion about this assessment tool.
In conclusion, our need to respond to the COVID-19 quarantine provided us with a golden opportunity to assess the feasibility of changes in methods of student assessment and to promote the use of integrated and interactive tools.19 In the current COVID-19 era that has national and international impacts, there is an ever-increasing need for more validated evaluation tools.20 We recommend this method to be used as a modular work assessment tool and not to replace the final examinations during conventional circumstances. The adopted collaborative mega project, which involves assessment tasks carried out over several modules with an optional extension through summer electives, may alleviate the mental stress associated with traditional assessment methods.21
Conclusion
-
•
CBL is a customizable tool; CBL item creation imparts relevance for students as it ties theory to clinical practice.
-
•
CBL item creation is an innovative method that is both practical and efficient as a mode of assessment for adult learners.
-
•
Students' insights into the project promote value with respect to the learning process.
-
•
Students' contributions to planning for the assigned project promote engagement, enthusiasm, and self-esteem and enhance both learning and professionalism.
-
•
Mentorship with constructive feedback drives student learning and facilitates acquisition of new skills. However, tutors should be trained with regard to modes of constructive feedback.
-
•
Clear instructions and explicit rubrics for grading are essential.
Disclosure of competing interest
The authors have none to declare.
Acknowledgments
The authors would like to acknowledge the Egyptian Knowledge Bank for providing the Q-bank and the Skillgym tool to all Egyptian Universities. The authors acknowledge their department staff members who participated in the evaluation of the submitted assignments.
Footnotes
Supplementary data to this article can be found online at https://doi.org/10.1016/j.mjafi.2021.05.018.
Appendix A. Supplementary data
The following are the Supplementary data to this article:
References
- 1.Sahu P. Closure of universities due to coronavirus disease 2019 (COVID-19): impact on education and mental health of students and academic staff. Cureus. 2020;12(4):e754. doi: 10.7759/cureus.7541. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Williams G., Lau A. Reform of undergraduate medical teaching in the United Kingdom: a triumph of evangelism over common sense. Bmj. 2004;329(7457):92–94. doi: 10.1136/bmj.329.7457.92. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Amin H.A., Shehata M.H.K., Ahmed S.A. Step-by-step guide to create competency-based assignments as an alternative for traditional summative assessment. MedEdPublish. 2020;9(1) doi: 10.15694/mep.2020.000120.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Kaup S., Kaup S., Jain R., Shivalli S., Pandey S. Sustaining academics during COVID-19 pandemic: the role of online teaching-learning. Indian J Ophthalmol. 2020;68(6):1220. doi: 10.4103/ijo.IJO_1241_20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.McLean S.F. Case-based learning and its application in medical and health-care fields: a review of worldwide literature. J Med Educ Curric Dev. 2016;3 doi: 10.4137/jmecd.s20377. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Akers A., Blough C., Iyer M.S. COVID-19 implications on clinical clerkships and the residency application process for medical students. Cureus. 2020;12(4) doi: 10.7759/cureus.7800. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Yardley S., Dornan T. Kirkpatrick's levels and education 'evidence'. Med Educ. 2012 Jan;46(1):97–106. doi: 10.1111/j.1365-2923.2011.04076.x. PMID: 22150201. [DOI] [PubMed] [Google Scholar]
- 8.National Academic Reference Standards (NARS) Medicine. 2nd ed. 2017. [Google Scholar]
- 9.Passi V., Southgate L. Revisiting ‘Performance rating scale for peer and self-assessment’. Med Educ. 2016;50(3):267–270. doi: 10.1111/medu.12798. [DOI] [PubMed] [Google Scholar]
- 10.Bloxham S., West A. Understanding the rules of the game: marking peer assessment as a medium for developing students' conceptions of assessment. Assess Eval High Educ. 2004;29(6):721–733. [Google Scholar]
- 11.Harden R. The integration ladder: a tool for curriculum planning and evaluation. Med Educ. 2000;34(7):551–557. doi: 10.1046/j.1365-2923.2000.00697.x. [DOI] [PubMed] [Google Scholar]
- 12.Herron E.K., Powers K., Mullen L., Burkhart B. Effect of case study versus video simulation on nursing students' satisfaction, self-confidence, and knowledge: a quasi-experimental study. Nurse Educ Today. 2019;79:129–134. doi: 10.1016/j.nedt.2019.05.015. [DOI] [PubMed] [Google Scholar]
- 13.Brisbourne M.A., Chin S.S.-L., Melnyk E., Begg D.A. Using web-based animations to teach histology. Anat Rec. 2002;269(1):11–19. doi: 10.1002/ar.10054. [DOI] [PubMed] [Google Scholar]
- 14.Brouwers M., Custers J., Bazelmans E., Weel C.V., Laan R., Weel-Baumgarten E.V. Assessment of medical students’ integrated clinical communication skills: development of a tailor-made assessment tool. BMC Med Educ. 2019;19(1) doi: 10.1186/s12909-019-1557-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Hart D. Addison-Wesley; Menlo Park, CA: 1995. Authentic Assessment: A Handbook for Educators. [Google Scholar]
- 16.Berman N.B., Durning S.J., Fischer M.R., Huwendiek S., Triola M.M. The role for virtual patients in the future of medical education. Acad Med. 2016;91(9):1217–1222. doi: 10.1097/ACM.0000000000001146. [DOI] [PubMed] [Google Scholar]
- 17.Preston R., Gratani M., Owens K., Roche P., Zimanyi M., Malau-Aduli B. Exploring the impact of assessment on medical students' learning. Assess Eval High Educ. 2019;45(1):109–124. [Google Scholar]
- 18.Kaufman D.M. Understanding Medical Education. 2018. Teaching and learning in medical education; pp. 37–69. [Google Scholar]
- 19.Abu Farha R.J., Zein M.H., Al Kawas S. Introducing integrated case-based learning to clinical nutrition training and evaluating students' learning performance. J Taibah Univ Med. Sci. 2021 doi: 10.1016/j.jtumed.2021.03.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Ferris H.A., O' Flynn D. Assessment in medical education; what are we trying to achieve? Int J High Educ. 2015;4(2):139–144. [Google Scholar]
- 21.Goda O.G., Mahdy D.T., El Shahed M.M., Amin H.A., Ahmed S.A. Students' awareness as an underlying factor for satisfaction and compliance. J Microsc Ultrastruct. 2020;8:198–204. doi: 10.4103/2213-879X.310729. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.