MOOC Learning Assessment in Clinical Settings: Analysis from Quality Dimensions

Abstract

Online learning is becoming a fundamental modality of learning in medical education, and can be of great help during global crisis like the current COVID-19 pandemic. The MOOC (massive open online course) mode of e-learning is increasing its penetration worldwide, as a valid teaching approach to reach large populations. A major challenge in clinical education is the assessment of medical students and residents in clinical settings, and there is substantial evidence that the current situation requires improvement. The goal of this study was to evaluate the quality dimensions of a MOOC titled “Learning assessment in clinical settings,” developed by three Mexican universities in the Coursera platform. A mixed-method study design was used to assess the quality dimensions of the MOOC in two phases: pilot and implementation. The best-rated aspects were learning resources and pedagogical perspective, and those with opportunity for improvement were collaboration and time management. Assessment might be learned through a MOOC format, especially for participants willing to engage with educational technology and self-direction.

Background

The COVID-19 pandemic declared by the World Health Organization (WHO) has slowed down the entire world, changing multiple economic and social sectors, including higher education. Schools and universities have been forced to transition to an online format to protect the academic community. Education is being abruptly transformed into distance and self-directed learning. In consequence, medical educators around the globe are changing their courses, practices, and other activities to provide their teaching from home. The most innovative educational institutions have already implemented distance education as part of academic programs, faculty development, and continuing medical education. They are moving forward to guarantee the programs’ continuity, and those reluctant to change are now struggling to maintain educational services. Because of the crisis, the short-term future of every aspect of life is uncertain, but distance learning is the safest modality that will prevail.

Well-planned online experiences are meaningfully different from courses offered to respond to a crisis [1]; MOOCs (massive open online courses) offer education with high-quality standards to a large number of participants around the world who have internet access at little or no cost [2]. According to Daniel, Vázquez, and Gisbert [3], these courses have the potential to offer better educational experiences within learning communities where students and teachers from multiple institutions and geographic locations participate remotely. According to these authors, MOOC can be understood as an essential approach for learning in higher education, not only in countries where they are already offered but also in less economically developed countries.

MOOC programs are becoming an alternative to reduce the cost compared to traditional training. Therefore, they may increase the perceived value for participants. MOOCs intend to expose students to new disciplines, stimulate new interests, learn ideas, and create a learning community [4]. In medical education, several MOOCs are now delivered as independent education or as a complement to classroom courses [5]. During COVID-19, MOOCs become an opportunity for distance learning about topics of interest to users.

There is some skepticism that MOOCs can provide a satisfactory learning experience, no matter what technologies are used [6]. Online learning carries a stigma of being of lower quality than face-to-face learning, despite research showing otherwise [1]. Another criticism of MOOCs is the difficulty of teaching psychomotor skills, although the possibility of integrating virtual patients in MOOCs for clinical reasoning competencies is being explored [7]. The emergent move to online teaching during COVID-19 could accelerate the decolonization of curricula, and increase the perception that this type of learning has several advantages [8]. Then, medical educators may understand through their own mandatory experience, how distance learning may be an option for teaching and learning.

MOOCs for Medical Educators

Regarding faculty development, Coursera MOOCs provide completion certificates useful as evidence of micro-credentials for participants. However, there is currently a shortage of educational interventions available for medical educators, targeting the global community of Spanish-speaking health professionals. The National Board of Medical Examiners (NBME) created the Latin American Grants Program for projects related to assessment in medical education (https://www.nbme.org/about-nbme/contributions-opportunities/latin-american-grants). Faculty in health sciences require practical training in educational evaluation. Teacher training should encourage a comprehensive, patient-centered education, emphasizing the assessment of clinical competence [9]. Additionally, a MOOC is an alternative for teacher development since it is delivered on a flexible platform with a personal learning environment designed for autonomous learning [10]. The current pandemic crisis is an opportunity for educators to prepare themselves on important pedagogical competences as student assessment.

The present document describes the quality dimensions results of the MOOC “Learning assessment in Clinical settings” which three universities in Mexico designed as part of a project funded by the NBME. The online format also requires a quality model for the design, production, maintenance, and evaluation of a MOOC as part of the educational strategy.

MOOCs’ Dimensions of Quality

Several authors have proposed dimensions to assess the quality of a MOOC. Daniel et al. [3] consider these aspects as follows: teaching model, monetization, certification, and adaptive learning. The teaching model focuses on the student acquiring useful skills and knowledge that can be applied in real-world contexts. Monetization refers to making MOOC courses financially viable using different approaches that can reduce the costs of other online learning models while offering comparable or even higher learning quality. Certification has two aspects of MOOCs. The first is that it opens the door to revenue from course fees; secondly are the issues of how learning is assessed, authenticated, and valued by employers. Adaptive learning refers to making MOOC courses more personalized by collecting all the data for analysis and interpretation to improve the educational activities, the courses delivered, the learning experience, and the investments of exclusive educational offers. Zhu, Bonk, and Sari [11] list three categories: resources, pedagogy, and logistics. The first category refers to the affordance of MOOC platforms, support from the host institution and the platform, and the available intellectual and hardware resources. The pedagogical category focuses on the learning objectives, assessment methods, course length, course content, flexibility, and collaborative learning support. The logistics category refers to the amount of time that instructors spent designing the MOOC (planning, assessment, and teaching). DeBoer, Ho, Stump, and Breslow [12] consider the following as important variables: enrollment, participation, curriculum, and achievement. Enrollment refers to know the user commitment to particular pathways for certification, auditing, etc., in order to allow a better description and analysis of the users. Participation refers to the differentiated pathways created by the individual goals or objectives of the students, such as attendance, number of clicks, total number of hours spent in the course, homework time, and partial and final exams. Curriculum refers to the number of resources in the course, which users can choose according to individual preferences, for which there is no correct, prescribed way to proceed. Finally, achievement is defined by the individual goals of the user, rather than criteria set by instructors. Other authors suggest tips for MOOCs associated with preparation, design, implementation, and improvement (Table 1).

Table 1.

Twelve tips for MOOCs in medical education

Pickering, Henningsohn, DeRuiter, de Jong, and Reinders [13]	Kellogg [6]
Preparation
Enroll on a MOOC to fully appreciate the online environment   Learn from other MOOC enthusiasts   Develop a MOOC topic that you are passionate about	Practice the lecture before taping
Design
Recruit a committed and enthusiastic team to support the MOOC’s development and delivery   Develop a curriculum map   Create a clear project plan	Reach out to experts in curriculum development and technology for guidance Develop a list of learning goals Prepare background materials for non-scientists.
Implementation
Create video content based on good educational practice Construct an appropriate assessment profile   Promote your course with a clear and enticing message   Provide a supportive environment for your learners to interact with	Try out videos, online quizzes and animated models on a small cohort first. Make course expectations regarding accessing homework and quizzes from the beginning Encourage student-to-student interaction in online forums
Improvement
Devise a broad research and evaluation strategy for post-course reflection and improvement   Share your experience to enhance MOOC pedagogy	Hire an assistant to help in monitoring forums and distributing communications. Collect data on student learning to improve the next iteration of the course.

Alemán, Sancho, and Gómez [14] propose a model of four dimensions to assess a MOOC quality: Pedagogical perspective, functional features, technological features, and time management. Pedagogical perspective relates to content, learning resources, pedagogical approach, and evaluation. Functional features evaluate autonomy and user control, user friendliness, and functionality. Technological features evaluate visual environment, design, versatility, navigation, interaction, and dialogues. Time management addresses the agenda and timing for each activity.

Pedagogical Perspective

The pedagogical perspective should be the most important quality standard to be considered by the design team. Many faculty members believe that MOOC cannot replace a teacher because of the lack of interaction with the instructor. However, some studies have shown that collaboration and peer assessment promote critical thinking and an effective way to create a free, open, and highly interactive global community [15, 16]. To create an engaging environment, it is vital to understand learners’ motivations to start a MOOC, their expectations, and their preferred learning process [11]. Quality Matters [17] offers a list of requirements for online learning.

The MOOC “Learning evaluation in Clinical settings” has the objective of disseminating how teaching develops in various clinical settings, from the point of view of the student and the teacher. The program starts with an overview of traditional medical education and its evolution over time with an emphasis on assessment. The participant learns to analyze what elements are important in the assessment of learning, to improve decisions about what to assess and why. Each assessment tool is described considering Patient-Centered Learning principles, validity and quality measures, and feedback practices to improve student performance.

The MOOC “Learning evaluation in Clinical settings” was organized on four modules:

1. Medical education and evaluation: Participants learn the evolution of the role of the clinical educator in the Latin American context. The content contrasts the required competencies of the clinical educator and performance assessment practices. Activities promote reflection on the development opportunities for medical educators.

   • 1.1
     The clinical educator and his/her roles in the Latin American reality.
   • 1.2
     Evolution of clinical teaching.
   • 1.3
     Patient-Centered Learning as a proposal for clinical evaluation.
   • 1.4
     Current concepts in clinical medical education (clinical competence).
   • 1.5
     Educational assessment in response to the challenges of clinical teaching.
2. Patient-centered assessment of clinical competence: Participants understand the main changes through the history of medicine and medical education considering several teaching environments. They recognize that the common denominator in clinical teaching throughout history continues to be patient-centered care, regardless of changes in medical schools. The content includes the relationship between clinical cases and learning objectives.

   • 2.1
     Direct observation (360°, portfolios, MiniCEX, work-based assessment).
   • 2.2
     Structured assessment (OSCE, clinical reasoning, standardized patients).
   • 2.3
     Use of cutting-edge technology (Simulation, ICTs, augmented reality).
   • 2.4
     Clinical cases (clinical records, morbidity and mortality sessions, journal clubs).
   • 2.5
     Oral and written assessment (use of graphic media and exams).
3. Quality of Evaluation in clinical settings: The purpose is to teach the meaning of Patient-Centered Learning and each of its perspectives: human, biomedical, management, and entrepreneurship. Participants identify aspects of the clinical encounter that are associated with each of the patient perspectives and recognize the clinical scenarios in which the clinical evaluation can be done.

   • 3.1
     Patient safety and quality of assessment (accreditation and certification).
   • 3.2
     Basic concepts of educational assessment in the clinic.
   • 3.3
     Use of evidence in assessment (the Rational Clinical Examination).
   • 3.4
     Clinical assessment scenarios.
   • 3.5
     Using the perspectives of patient-centered learning.
4. The role of feedback in the clinical evaluation: This module has the purpose of orienting participants to recognize the importance of learning and teaching in various clinical settings. There are several competences to manage: diagnosis, treatment, prognosis, and prevention of health and disease. Faculty members should define and identify on how to improve clinical competence from students’ performance.

   • 4.1
     Socio-affective aspects of the clinical competence assessment (emotional intelligence).
   • 4.2
     Feedback strategies.
   • 4.3
     The feedback process and the difficult student.
   • 4.4
     Mentoring.
   • 4.5
     Assessment as a mechanism for continuous improvement.

On the first blueprint of the program, there were five modules to be distributed to each university participating in the project. Still, after the first design session, the topics were integrated and mixed in four modules. The course has the purpose of developing teachers, students, and health professionals on concepts and applications for evaluation in clinical settings based on the principles of Patient-Centered Learning [18]. The collaboration process to define the structure of the program was assessed by experts from the NBME and consensed after several face-to-face and online sessions.

The content has a variety of resources as videos, infographics, power point presentations, and reading materials. There are included interactive forums, self-assessments, and projects aimed to implement the learning into the practice. Similar to other MOOCs, because of the large number of participants, activities are tested by automated quizzes with multiple-choice answers, and assignments are reviewed by peers also active on the course [19].

Functional Features

The functional features are commonly called user friendly. Navigation inside the platform should be clear and easy to understand for every user. MOOC participants may have diverse backgrounds and levels of knowledge, so the links should take participants into activities of each module in a very intuitive direction [11]. References should be presented appropriately to invite users to learn deeper the topics of their interest.

The MOOC “Learning evaluation in Clinical settings” is presented with sequential topics and activities which might be accessed on both computer and mobile devices. The videos were limited to 6 with a length range from 5 to 8 min in order to enhance the participant experience. Contents may be downloaded to be accessed without a Wi-Fi connection.

Technological Features

Technological features relate to the beauty of materials considering colors, fonts, graphical design, and multimedia design. According to Graham [19], in a MOOC, the course content is delivered via video lectures. Darby [20] does not recommend posting a video providing a standard speech for a classroom; instead, the recording should capture the personality and passion of the instructor in ways different from what he might do in person, yet authentic. Quality Matters [17] recommends recording short multimedia videos with easy access to additional software and plugins. Besides the speaker, video editing provides opportunities to complement materials with other pictures, movement, data, and sound that expands the learning experience.

An important feature from the course is the interactive forums and peer-review activities where participants interact with each other to promote learning. An earlier study showed that one important success factor from students that completed the MOOCs was the multiple times posts on discussion forums [4]. According to Hoy [21], students that participate in online discussion forums get clarification, share thoughts, and build a sense of community.

The MOOC “Learning evaluation in Clinical settings” includes some videos that were recorded with professional equipment provided and edited by an audiovisual production company. A team of digital designers enriched the materials developed by faculty members and provided programming into the platform to enhance the user experience.

Time Management

Time management is crucial for medical educators. MOOCs provide learners with the ability to work at their own pace [19]. It is considered a primary advantage of a MOOC the flexibility of time and place for learners, allowing them to take the course when and where they prefer as most convenient. Zhu et al. [11] describe that MOOCs benefit participants to develop time management skills. Medical educators usually have tight schedules, and sometimes they even lack the interest to teach [22].

Purpose

The purpose of the study was to evaluate the quality dimensions of the MOOC “Learning assessment in Clinical settings” from the participants’ perspectives.

Method

A mixed, descriptive, and transactional method was applied considering two phases: Pilot (qualitative) and massive implementation (quantitative). During the pilot, a total of 57 faculty members from three universities were invited to register as participants on the course. As a qualitative phase, three focus groups were performed with 21 faculty members out of the 33 that completed the pilot course: 10 from Tecnologico de Monterrey, 8 from UNAM, and 3 from Westhill University. Faculty feedback in a health sciences’ MOOC was also applied by Sitzman, Jensen, and Chan [16]. Those faculty members provided feedback related to how conclusions resonated with their understandings and experiences in a MOOC. The focus groups proposed by Savin-Baden and Howell Major [23] aim to provide information to the researcher about the opinions of a group on a topic, to document ideas, and to highlight inconsistencies of opinion among members of a community. The process to guide each focus group session was with questions related to the quality dimensions proposed by Aleman et al. [14]. Each topic was reviewed, discussed, and analyzed by participants. Researchers stratified the excerpts by emergent coding [24]. Participants signed consent forms to participate in the study and approved the recording of the sessions. Results were applied to improve design before go-live implementation.

For the massive implementation, a quantitative approach was selected. There was designed a survey based on Alemán et al.’s [14] survey to assess quality dimensions considering the emergent coding from the pilot phase. The questionnaire has 12 items and Cronbach’ alpha of 0.97, to assess quality during implementation phase (Table 2).

Table 2.

Quality dimensions and assessment of MOOC courses

Groups	Quality dimensions	Items
Academic content	Pedagogical perspective	The activities considered the knowledge, skills, interests, and needs of the participants Activities and exercises develop skills to strengthen teaching practice
	Collaboration	Means of communication are offered among the participants (forums, press releases, etc.) Forums provide adequate help to resolve doubts or questions
	Learning resources	The activities offer resources that facilitate understanding of the topics The activities suggest the use of complementary documentation (readings, open resources, blogs, wikis)
Digital platform	Time management	The time indicated to carry out the activities is sufficient Compared to a face-to-face course, a massive and open online course allows you to do different activities at the same time
	Digital aesthetics	The technical and aesthetic quality of the multimedia materials is adequate The graphic design of the pages is attractive, and the resources are dynamic
	Functional features	Navigation on the pages is structured, simple, and ergonomic The instructions for the exercises are clear and easy to understand

In October, the course accumulated 4712 total learners who started the program, 2701 active learners with completed checkpoints, and 325 with certificate. Regarding participants’ profile, most of them have 25–34 years old (43.4%) followed by 18–24 years old range (26.2%). They are employed full time (39.8%) followed by unemployed (16.5%). For the present study, the survey only considered participants who have already completed the course to assess the MOOC’s quality effectively. A total of 190/322 participants completed the study, which represents a 59% response rate.

Results

Qualitative Phase from Pilot

Qualitative results from pilot are presented in Table 3. Some quote examples are included by code. The codes are organized, starting with the ones with best perceived value from participants, and lastly with codes related to topics with more opportunities for improvement.

Table 3.

Quotes from focus groups

Codes	Examples of quotes
Pedagogical perspective	The program has enough support for practice with your students. I did see it as useful to have tips for managing my group better. At some point, Patient Centered Learning seems repetitive for me. I enjoyed Patient Centered Learning content to introduce it in the cases. It seems to me that promotes self-management, which is also the value of the online course. There is feedback from the self-assessments, but sometimes it did not give you the feedback as you had answered Activities develop critical thinking. There may be other activities to improve it. There are challenging activities that allowed us to review what we previously knew and identify what was needed. Some sections of the activities do not invite reflection. A part of the course does.
Collaboration	There are activities in which a colleague will feed you back, which allows analyzing the colleague’s point of view. I like the platform, but face-to-face is also essential due to the interaction with my colleagues. I do not feel comfortable with people I do not know. Discussion in forums depends on others’ commitment. We contribute to the debate, but we do not go back to see what they said. The forums did not promote discussion. I Sometimes the participation is so extensive that it is difficult to read what they were writing.
Learning resources	The additional links are okay. They are not overloaded and arouse curiosity. There are additional links to deepen the topics. There are summaries, tables, maps, and other supports. It was well thought out what resources to use. It motivated me to investigate. The readings are interesting
Time management	The schedule helps to manage the time. It includes the hours for each topic. The time of the units is enough, and you just have to manage the times well. We did not have time to go deeper, and we went more to the multiple-choice. There was no time. I wanted to land in my quick work. I consider online courses faster than face-to-face. The time indicated limited the work that must be sent for feedback. The times were not enough for me.
Digital aesthetics	Very graphic, handy charts Visual quality is adequate. It does not hurt your eyes. The colors are beautiful and warm. The page design is attractive, and the resources are very dynamic. I found the presentation of the images very very good. I like the visual, and I thought that how it was developed was good.
Functional features	It was easy to access the content, it is clear. It is easy for me to use the phone, you can download the reading without the internet. Activity instructions are clear. The infographic summarizes everything that was seen on the subject. Activity instructions are clear. Browsing in the course is structured and fast. There was difficulty getting to some places where you wanted to work.

The resultant quality dimensions from Table 3 were classified into two groups, Academic content and Digital platform (Fig. 1). Academic content involves the process of learning competencies, both generic and disciplinary. Generic competences promoted and enhanced with MOOCs are creativity, critical thinking, self-direction, collaboration, and inquiry through learning resources. Disciplinary competencies from the MOOC “Learning assessment in clinical settings” are medical teaching and assessment. Participants valued content, exercises, and resources to assess Patient-Centered Learning with cases. However, there were contrasts in the answers. While some participants considered some material repetitive, others appreciated the inclusion of this module on the course. They also referred to how they have been introducing new practices in their clinical settings not considered before the training.

Fig. 1.

Adapted quality dimensions for MOOC programs

Digital platform refers to media production and preparation to enhance the learning experience, with an online well-designed and friendly environment, customized to the participant’s schedule and context. Quality Matters [17] recommends clear instructions and expectations, a sense of community and interaction, and a variety of resources for emergency remote learning. Similarly, Darby [20] offers tips for faculty to become a better online teacher, including clear expectations, content organization, and visual appeal.

Quantitative Analysis from Massive Implementation

Certainly, Academic content is the priority in online courses, but a friendly Digital platform prevents users’ frustration. The current situation with COVID-19 has forced emergency remote learning, which lacks adequate planning and multimedia production with high-quality standards. However, Academic content should be addressed to guarantee the continuity of education. This crisis also represents an opportunity for faculty development. Figure 2 shows the trends by month of the number of participants since the MOOC go-live in January 2020. It is evident that the lockdown in March caused an increasing interest in the program.

Fig. 2.

2020 monthly accumulated participants

Quality dimensions results for the MOOC “Learning assessment in clinical settings” are presented in Fig. 3. Scores are highly consistent with pilot perceptions of participants. The best-rated aspects were learning resources and pedagogical perspective. The dimensions with more opportunity for improvement were collaboration and time management.

Fig. 3.

Quality dimensions results for the MOOC “Learning assessment in clinical settings”

Conclusions

The use of digital platforms and assessment methods for education is exponentially growing since the COVID-19 pandemic. The future of medical education is uncertain and volatile, but MOOCs are becoming an excellent opportunity to educate faculty members and students with no instructor dependence. Right now, plenty of medical educators are already attending COVID-19 patients or they are waiting to be called to the clinical trenches. It is highly probable that some medical teachers will have little availability for their students.

Learning assessment in clinical settings in a MOOC format could be challenging for some medical educators, especially older ones. Participants had the opportunity to critically reflect on their prior knowledge and regarding evaluation to self-improve their practices considering Patient-Centered Learning principles. This program involves educational technology both in the content and the delivery. Several tools described in the course refer to the use of technology to assess students during clinical practices, which are also taught in the MOOC through the digital platform.

It is evident that students’ interactions with technology can change their motivation to advance in the course. Since MOOCs are open to diverse profiles, a friendly environment is required for every age and level of digital skills. This profile diversity in participants should be promoted as an opportunity to create a global interactive community of learners, developing self-guided individuals. Today more than ever it is imperative to collaborate to improve health care and education, to become better professionals, and to survive the crisis.

Funding

Financial support was provided by the National Board of Medical Examiners Latin American Grants Program.

Data Availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they have no conflict of interest.

Code Availability

Not applicable.