Skip to main content
PMC home page
JMIR Medical Education logoLink to JMIR Medical Education
. 2017 May 10;3(1):e10. doi: 10.2196/mededu.5434

Development and Assessment of an E-learning Course on Pediatric Cardiology Basics

Ana Cristina Oliveira 1,✉,#, Sandra Mattos 2,#, Miguel Coimbra 3,#
Editor: Carlos Luis Parra-Calderón
Reviewed by: Anthony Levinson, Poon Woei Bing
PMCID: PMC5443913  PMID: 28490416

Abstract

Background

Early detection of congenital heart disease is a worldwide problem. This is more critical in developing countries, where shortage of professional specialists and structural health care problems are a constant. E-learning has the potential to improve capacity, by overcoming distance barriers and by its ability to adapt to the reduced time of health professionals.

Objective

The study aimed to develop an e-learning pediatric cardiology basics course and evaluate its pedagogical impact and user satisfaction.

Methods

The sample consisted of 62 health professionals, including doctors, nurses, and medical students, from 20 hospitals linked via a telemedicine network in Northeast Brazil. The course was developed using Moodle (Modular Object Oriented Dynamic Learning Environment; Moodle Pty Ltd, Perth, Australia) and contents adapted from a book on this topic. Pedagogical impact evaluation used a pre and posttest approach. User satisfaction was evaluated using Wang’s questionnaire.

Results

Pedagogical impact results revealed differences in knowledge assessment before and after the course (Z=−4.788; P<.001). Questionnaire results indicated high satisfaction values (Mean=87%; SD=12%; minimum=67%; maximum=100%). Course adherence was high (79%); however, the withdrawal exhibited a value of 39%, with the highest rate in the early chapters. Knowledge gain revealed significant differences according to the profession (X22=8.6; P=.01) and specialty (X22=8.4; P=.04). Time dedication to the course was significantly different between specialties (X22=8.2; P=.04).

Conclusions

The main contributions of this study are the creation of an asynchronous e-learning course on Moodle and the evaluation of its impact, confirming that e-learning is a viable tool to improve training in neonatal congenital heart diseases.

Keywords: distance learning, continuing medical education, pediatrics, cardiology, congenital heart defects

Introduction

Background

Managing congenital heart diseases is a worldwide problem [1]. Early detection through newborn screening can potentially improve the outcome of these diseases [2]. In newborns, congenital heart diseases can be detected by auscultation, pulse oximetry, radiography, catheterization, although transthoracic echocardiogram, a specialized form of ultrasound, is the elected exam for diagnosis [3,4]. Pediatric cardiologists usually perform this examination, but in developing countries there is shortage of professional specialists, which are often concentrated in larger urban centers, hindering the widespread population screening and causing a need for constant transferal of patients from the isolated regions to reference health centers [5].

Context

In response to these challenges, the Health Secretary of Paraíba, in Brazil established a pediatric cardiology network [6]—Rede de Cardiologia Pediátrica (RCP)—in partnership with Círculo do Coração , a civil nonprofit organization from Recife, in order to create a neonatal screening program for the whole state of Paraíba and a hospital facility designed to manage patients. Of the 20 maternity centers covered by this network, only 7 had neonatal ICU beds. The remaining are level-1 district centers. In only one center, in the capital city of Joao Pessoa, there is a pediatric cardiologist available for echo and clinical diagnosis. No center performs cardiac surgery. This network provides perinatal and neonatal care in remote areas supervised by telemedicine; it was created due to the need to train local physicians and involve local professionals on screening, diagnosis, therapeutic treatment and management of congenital heart diseases in fetuses, newborns, and children of the public health system.

Echocardiography is used to diagnose congenital heart disease when either there is an abnormal clinical examination or an abnormal pulse oximetry. Abnormal pulse oximetry results are automatically noted on a database of the network, allowing the network to contact the clinic and request that they follow up any babies with abnormal test results. These active search protocols track the discharged neonates and ensure that abnormal findings are acted on.

Echo image acquisition and its interpretation is a combined process developed within this Web-based telemedicine network where trainees are constantly supervised by cardiologists. Not only the whole process is documented in the Web database system to track the apprentice's development, but also whenever needed, live interaction is called to adjust practical aspects. Initially, all echocardiograms needed to be performed with Web-based supervision by the pediatric cardiologist, as part of the neonatologists’ training, but with time cardiologists would only be requested for direct Web-based supervision when pathological findings were suspected. As there are always new neonatologists being trained, this learning process and interaction between teams is a continuous cycle [7].

E-Learning for Health Care

With the increasing use of Internet information and communication technologies, e-learning has emerged as a widely accepted modality in medical education [8]. For its convenience and its potential for cost savings, it has become popular among the medical education community [9]. E-learning is known to offer learning opportunities where there is limited access to teaching in a specific field, either because of a lack of qualified or geographically distant teaching institution [10]. Therefore, e-learning can be a powerful tool to increase the capacity of health professionals in constricted contexts for neonatal echocardiography congenital heart disease screening. There is evidence in the literature that e-learning is a useful tool for overcoming barriers to health professionals training [10]. A review carried out by Frehywot et al on e-learning in medical education in resource-constrained LMICs suggests that e-learning may be effective for increasing capacity in rural settings, although evidence is still limited.

The convenience, the self-paced and learner-centered learning, and the creation of a global learning community, are some significant benefits of e-learning that have been discussed in many articles [8,10-24].

In medical contexts, e-learning program results such as efficiency and costeffectiveness have typically gone unreported [25]. In nonmedical contexts, there is evidence that e-learning can result in cost savings of up to 50% over traditional learning programs, due to reduced instructor training time, travel and labor costs, institutional infrastructure, and the possibility of expanding programs with new technologies [8,9,26].

Nowadays, there is an increasing demand for e-learning courses [27]. Many software platforms and learning management systems (LMS) are being used to support Web-based courses for online continuing medical education. Among the various LMS, there is Moodle (Modular Object Oriented Dynamic Learning Environment; Moodle Pty Ltd, Perth, Australia), a well-known platform that is considered one of the best open-source LMS, in what concerns user-friendliness and adaptivity [28]. Moodle allows the integration of a broad range of educational resources, activity modules, such as Forums, Wikis, and Databases, that build a rich collaborative community of learning around a subject matter depending on the learning goal. Moodle can also be used to deliver content to students (such as standard SCORM packages) and for learning assessment; it provides assignments or quizzes. As another advantage, its interface allows surfing through the contents intuitively [28,29]. The reason we decided to use Moodle in this study is because we had access to it in the Faculty of Medicine of the University of Porto that could host the Moodle course, and given its utility, we did not need to address the costly development of a new platform.

Objectives of the Study

In this context, as primary outcomes we aim to (1) develop an e-learning Pediatric Cardiology Basics Moodle course for nonspecialists and (2) evaluate its pedagogical impact and user satisfaction.

As secondary outcomes, we want to understand whether there are significant differences between different types of professionals undergoing the course and to measure the adherence to the course in this specific scenario.

Methods

The E-Learning Course Description

We chose to build a new course, as the few actual existing options are in English, and also associated with other institutions. The course was in Brazilian Portuguese, implemented on Moodle, hosted on the server at the Faculty of Medicine of the University of Porto, Portugal. The contents of the course were mainly based on the “Cardiologia para o Pediatra” book [30], which was used in this network to teach their professionals, and was written by a specialist Dra Sandra Mattos, stakeholder of the RCP network. This book follows a didactic approach, where the basic concepts for neonatal screening of congenital heart diseases are addressed, such as the current standard clinical protocols and guidelines to apply [31,32]; proposed by the American Association of Cardiology and the Association for European Paediatric and Congenital Cardiology, concepts of cardiac neonatal anatomy and physiology, how to obtain the ultrasound images, and for each anatomic window, which echocardiographic findings to expect and look for. The course was structured into 3 modules, with a total of 8 chapters (Table 1).

Table 1.

Description of the main contents contained in the intervention e-learning course.

Modules Chapters Contents
Foreknowledge Cardiac neonatal anatomy
Physiology of neonatal circulation
Physical ultrasound principles		 Internal configuration of the heart
Fetal circulation
Neonatal circulatory changes
The ultrasound properties
Transducers
Echocardiography

Echocardiogram screening How to obtain the ultrasound images
Different echocardiographic modalities		 4-chamber image
Left ventricle outflow track image
Right ventricle outflow track image
Pathologies to exclude in each image
Bidimensional mode
M mode
Doppler mode
Pathologies Identified pathologies in the 4-chamber image
Identified pathologies in the outflow tracts image
Difficult to diagnose pathologies images in the neonatal period		 Interventricular communication
Defect of atrioventricular septum
Tricuspid atresia
Ductus arteriosus
Aortic stenosis
Pulmonary stenosis
Tetralogy of Fallot
Transposition of the great vessels
Truncus arteriosus
Interatrial communication
Coarctation of the aorta
Total anomalous pulmonary venous drainage
Open in a new tab

The content was presented using text, images, and videos. Diagnostic images and videos were collected directly from the RCP network at the Real Hospital Português in Recife, Brazil. A total of 22 images and 36 videos were used in this course, which were anonymized and illustrated examples of planes and imaging cuts, without any association with identifiable data from patients. Pediatric cardiology specialists from the cardiology and fetal medicine unit at Real Hospital Português reviewed all contents. Screenshots are presented in Figure 1.

Figure 1.

Figure 1

Open in a new tab

Course flow diagram.

Recruitment and Implementation Strategy

Participants were recruited via an email disclosure within the RCP network platform with a brief description of the study, the link to the course, a tutorial for Moodle log, and the password for enrolment in the course. To access it, the learners should register on the Moodle website according to their personal information. Thus, website security was guaranteed through an authentication mechanism with username and password. After that, we had access to the participant’s personal email given at the registry, and then we could do the follow-up of each participant by email.

The course was created and revised between April 2014 and July 2014. We contacted the participants in September 2014, and those who registered to do the course first did the pretest and then had access to the asynchronous course, which was given until November 2014.

The course was built in a unidirectional way (Figure 2). First, the participants registered on the Moodle website. Then, a pretest of 16 questions was available during 20 min. After the answers submission, the first chapter was available. At the end of each chapter, a summary with the main key-points and a formative test of two multiple-choice questions for self-assessment was presented. These intermediate tests allowed immediate knowledge self-assessment. The next chapter would only be available if the user had given the correct answers. In case of error, the learner would be directed to a summary of the lecture, and then it was possible to go back to the previous lecture or repeat the assessment. At the end of these 8 chapters, a final summative assessment of 16 questions corresponding to the posttest was available during 20 min. After the final approval, a certificate was sent to the learners.

Figure 2.

Figure 2

Open in a new tab

Screenshot of one of the echocardiographic views and schemes (left), and of one of the self-assessment tests (right) included in the e-learning course.

The learning activities chosen were lessons that corresponded to each chapter. We organized the content, images, and videos in different pages as it corresponds to different topics. The learner could control the lecture flow by pressing control buttons located at the end of that interface, moving forward or backwards, allowing the lecture to flow from the beginning to the end, page by page, managing his own learning process.

As the course was performed asynchronously, the learner could observe his evolution learning at any time during the course through a progress status bar that updates dynamically in order to manage his self-learning.

Evaluation Strategy

An important method of assessing educational training is a framework developed by Kirkpatrick [33,34], which focuses on 4 levels: reaction, learning, behavior, and results.

Numerous studies [16,21,35-42] have been carried out to assess the benefits of e-learning in medical education. Very often, evaluative education studies [38,43] rely solely on the reaction level and learning over the behavior and results levels [16,44,45].

Pre- and Postintervention Test

Concerning the learning assessment, the most common method described in literature is the pre- and posttesting self-controlled method, with multiple-choice test scores.

In this study, the participants were asked to do a pretest before taking the course and a posttest after completing the course. We used the same questions for pre and posttesting, so that we can guarantee the same level of difficulty and comparable results [46]. The test was structured with 16 different multiple-choice questions. They were all single-select questions and each question had 2-5 response options. The assessment questions were text-based to test knowledge based on the key ideas, learning outcomes, and objectives established for the course. In this test, a total of 2 questions were related to each of the course’s 8 modules (Table 1), which were also the same end-of-chapter practice questions. The participants could only proceed to the next chapter after answering these end-of-chapter practice questions correctly. An expert on pediatric cardiology revised the questions making sure they were appropriate to the course content. To improve the authenticity of the answers by discouraging access to support materials, participants had only 20 min to complete the 16-question test, so a time controller and a progress bar were available. After that time, if they had not submitted, the answers would be automatically saved. At the end, the participants could see the test result, but not the correction, and they could not repeat the test. The technology we used to provide this evaluation tool was the Moodle lesson questions, a free Web-based office suite and data storage service.

User’s Satisfaction Questionnaire

The reaction assessment is mostly done by questionnaires. One of the most cited questionnaires for assessing user satisfaction was developed by Wang [47]. Wang created an e-learning satisfaction model that consists of 26 items related to 4 qualities: content quality, learning interface quality, personalization quality, and learning community quality. However, the last 2 questions refer to global measurement in the context of end-user satisfaction, first developed by Doll in 1988 [48], and related to overall satisfaction and overall success. The measurement scale used was a 7-point Likert-type scale, with anchors ranging from “strongly disagree” to “strongly agree.” Globally, this questionnaire was shown to have a reliability (Cronbach alpha) of .95 [47].

For our study, we used the questions related to content quality, learning interface quality, and personalization quality from Wang’s questionnaire [47]. We did not use questions related to learning interaction quality, because the developed e-learning course was asynchronous. We used a Portuguese translated existing version from Wang’s questionnaire [49]. The technology used to apply this evaluation instrument was the Moodle survey module. The satisfaction questionnaire became available to the participants who had taken the course after they did the postcourse.

Statistical Analysis

The purpose of the data analysis was to determine whether there was a significant difference between the test score before and after the course. We considered learning improvement as the ratio of the difference between scores and the preintervention score. The learning efficiency is the learning improvement per hour of the course.

Data analysis was performed by descriptive and inferential statistics, using the IBM SPSS Statistics software version 22.0.

According to the fulfillment of the criteria necessary to perform parametric hypothesis testing, it was concluded that the sample did not follow a normal distribution. Thus, we used the following nonparametric tests: Wilcoxon Signed-Rank test, Mann-Whitney U Test and Kruskall-Wallis H test. For all this statistical analysis, we considered a significance level of 5% [50].

Results

Summary

The target population in this study was the health care professionals—neonatologists, pediatricians, obstetricians, nurses, and internship medical students—who work in the RCP network. This convenience universe potentially includes a total of around 80 people. We obtained 78% (62/80) registrations. Concerning adherence (Figure 3), 79% (49/62) started the course. At the end, 61% of the participants (30/49) managed to complete the course and 39% (19/49) dropped out (Figure 4). See the percentage of participants who dropped by chapter (Figure 5). From the participants who had taken the course, 67% (20/30) responded to the satisfaction questionnaire.

Figure 3.

Figure 3

Open in a new tab

Screenshot of one of the self-assessment tests included in the e-learning course.

Figure 4.

Figure 4

Open in a new tab

Participant flow diagram showing the enrolled sample and respective dropouts.

Figure 5.

Figure 5

Open in a new tab

Percentage of participants who dropped by chapter.

Sample Description

The total sample consisted of 62 registered health care practitioners, including 49 female (79%) and 13 male elements (21%); 49 individuals started the course, including 39 female (80%) and 10 male elements (20%); and 30 participants completed the course, including 22 female (73%) and 8 male elements (27%; Table 2).

Table 2.

Frequency and percentage of participants by gender.

Participants Female, n (%) Male, n (%)
Total sample (n=62) 49 (79) 13 (21)
Started the course (n=49) 39 (80) 10 (20)
Completed the course (n=30) 22 (73) 8(27)
Open in a new tab

With regard to the registered participants, most are doctors (n=33; 53%) and mostly are of Neonatology (n=24; 39%) and Pediatrics (n=23; 37%) departments. This ratio remains the same for those participants who initiated the course (Table 3).

Table 3.

Frequency and percentage of participants by profession and specialty.

Participants Assigned
(n=62), n (%)		 Initiated
(n=49), n (%)		 Concluded
(n=30), n (%)
Profession




Doctors 33 (53) 27 (55) 16 (53)

Internship medical students 10 (16) 9 (18) 8 (27)

Nurses 19 (31) 13 (27) 6 (20)
Speciality




Neonatology 24 (39) 19 (39) 18 (60)

Obstetrics 6 (10) 5 (10) 4 (13)

Pediatric Cardiology 9 (14) 8 (16) 3 (10)

Pediatrics 23 (37) 17 (35) 5 (17)
Open in a new tab

Regarding the state where they practice, 45 (73%) works in Paraíba and the other 17 (27%) in Pernambuco.

Pedagogical Impact

Regarding the pretest, 67% (n=20) passed (test score≥50%) and 33% (n=10) of participants failed (test score<50%). The test scores ranged between 0 and 100%. However, none of the participants failed to complete all the test items within the time limit. With respect to the posttest, 100% (n=30) of the participants passed.

The differences between the test scores before and after the course were all positive. There were no negative differences or equal scores before and after the course. A Wilcoxon signed-ranks test indicated that the median [Mdn (P25-P75)=94 (81-100)] posttest ranks were statistically significantly higher than the median [Mdn (P25-P75)=56 (38-69)] pretest ranks (Z=-4.788; P<.001).

Doing an intent-to-treat analysis by comparing the mean ranks (Mann-Whitney–U Test) of the score from the participants who did not do the course [Mdn (P25-P75)=50 (29-65); n=19] with those who did the course [Mdn (P25-P75)=94 (81-100); n=30], we found that test scores in those who did the course were statistically significantly higher than those who did not (U=21.500; P<.001).

Globally, the difference between the final and the initial scores can indicate the learning impact of the course [Mdn (P25-P75)=34 (19-50); n=30]. Moreover, the improvement in the results was related to what the participants already knew before the course [Mdn (P25-P75)=61 (36-114); n=30]. The efficiency was how much they improve per hour dedicated to the course [Mdn (P25-P75)=31 (12-80); n=30] (Figure 6). The maximum time dedicated to the course was 09 h 58 min and the minimum 25 min [Mdn (P25-P75)=01:47 (01:08-03:01); n=30].

Figure 6.

Figure 6

Open in a new tab

Improvement (left) and learning efficiency (right) stratified by professional status.

We found the initial scores were statistically significantly different (X22=13.8 P=.001) between professions (Table 4). Although nurses hadn’t the highest final score, they were the ones who exhibited the highest difference between the scores, more than doubling it, but with less efficiency, as they also dedicated more time to the course. However, internship medical students had the highest learning efficiency, improving their knowledge per hour, as they dedicated much less time to the course than doctors and nurses. Predictably, doctors have lesser benefits from such an e-learning course. There were significant statistical differences in the improvement between professions (X22=8.6; P=.01).

Table 4.

Median and percentile Tukey’s hinges (P25-P75) of the scores in study by profession.

Indicators Doctors (n=16) Internship medical students (n=8) Nurses (n=6) P valuea
Initial score 63 (53-78) 50 (41-59) 40 (31-50) .001
Final score 99 (88-100) 94 (81-94) 88 (81-100) .24
Differenceb 27 (13-44) 34 (28-44) 51 (31-63) .13
Improvement (%)c 43 (16-89) 63 (55-91) 128 (63-200) .09
Efficiency (%)/hd 15 (8-42) 76 (53-153) 26 (16-80) .01
Dedication (hh:mm) 02:01 (01:32-03:06) 00:50 (00:37-01:06) 03:07 (02:09-03:54) .45
Open in a new tab

aKruskal-Wallis H test.

bFinal score - initial score.

c([final score - initial score]/ initial score)*100.

d([final Score - Initial score]/ initial score)*100)/hour.

Concerning specialty (Table 5), the obstetricians were the ones who benefited the most, with the highest difference between the scores, the best improvement and efficiency (Figure 7), and were the most dedicated to the course. There were statistically significant differences in the improvement (X22=8.4; P=.04) and in the course dedication (X22=8.2; P=.04) between the different specialties.

Table 5.

Median and percentile Tukey’s hinges (P25-P75) of the scores in study by specialty.

Indicators Neonatology (N=18) Obstetrics (N=4) Pediatric Cardiology (N=3) Pediatrics (N=5) P valuea
Initial score 56 (44-63) 37 (25-46) 87 (68-87) 69 (69-81) .06
Final score 91 (81-100) 88 (81-87) 100 98 (94-100) .15
Differenceb 38 (25-44) 60 (45-63) 13 (13-32) 13 (13-29) .06
Improvement (%)c 61 (43-100) 170 (101-267) 15 (15-58) 18 (15-42) .039
Efficiency (%)/hd 42 (20-78) 50 (18-117) 12 (10-50) 9 (7-16) .28
Dedication (hh:mm) 01:39
(00:54-03:01)		 03:12
(02:20-05:30)		 01:20
(01:14-1:36)		 01:35
(01:08-2:08)		 .04
Open in a new tab

aKruskal-Wallis H test.

bFinal score-initial score.

c([final score-initial score]/ initial score)*100.

d([final score-initial score]/ initial score)*100)/hour.

Figure 7.

Figure 7

Open in a new tab

Improvement (left) and learning efficiency (right) stratified by specialty.

User Satisfaction

Only 20 (67%), out of the 30 participants who completed the course answered the satisfaction questionnaire. This was disappointing, showing that it was simpler to motivate these professionals to gain knowledge, than to contribute to a research study.

Globally, the satisfaction with the e-learning course was positive (μ=87%; σ=12%; minimum=67%; maximum=100%), whereas 6 (30%) learners were totally satisfied with the e-learning course. Regarding global measures (Q14 and Q15), 17 (88%) were satisfied and considered that the e-learning course was successful (6 and 7 points in the 7-point Likert-type scale). Considering content quality (Q1-Q4), 17 (88%) learners think that this course fits their needs, is useful, sufficient, and up to date.

About the learning interface quality (Q5-Q9), 15 (76%) learners found the e-learning course user-friendly, stable, making it easy to find contents needed. Concerning personalization quality (Q10-Q13), 16 (84%) participants think the e-learning course enables them to control the learning progress, to learn the content needed, to choose what to learn and record their learning progress and performance.

Within these 20 participants, all claim the necessity of continuing medical education (CME) and 18 (90%) said that they would like to do CME by e-learning systems, whereas the other 2 (10%) said that maybe they will be interested in CME by e-learning.

Discussion

Principal Findings

The first main result of this study was the development of an e-learning course for the neonatal screening of congenital heart diseases. Using a free open-access tool such as Moodle and adapting the pedagogical contents of a well-established book for teaching this subject did not require advanced programming skills and led to an effective e-learning course. Given our successful results, it is expectable that future e-learning courses that specifically use best practices for multimedia learning should have an even stronger impact.

The second principal outcome was the statistically significant results obtained in the used metrics of pedagogical impact, with quite interesting proportions of learning with just a few hours of training. These results meet the general literature reviews that identified e-learning as superior to noneducation intervention [45], or as having similar effects to traditional learning [51], and more effective when combined with traditional learning [29] (b-learning). Although, in some contexts, e-learning may not be an alternative for the traditional face-to-face learning method, it can always offer a contribution and be a complement, and a useful adjunct to traditional education [51,52].

When comparing professions, we confirmed that the background knowledge about pediatric cardiology varies by professional group. By the end of the course nurses were the ones who learned more, although they did not score the highest final score. The heterogeneity of time spent in learning was also perceived. Internship medical students had the highest learning efficiency, dedicating much less time to the course than doctors and nurses. Predictably, doctors have lesser benefits from such an e-learning course.

Concerning specialty, the obstetricians were the ones who improved their knowledge the most, with the highest difference between the scores, the best improvement and efficiency, and were the most dedicated to the course. There were statistically significant differences in the improvement and in the course dedication between the different specialties, in this case confirming that more time dedicated to the course does translate into higher knowledge gain.

The third principal result was the encouraging assessment of the user satisfaction questionnaire. However, a third of the students who completed the course did not complete this questionnaire. This might be an important consideration for future studies, with regard to the interest and motivation of the participants to answer satisfaction questionnaires.

Regarding the adherence to this study, although the number of participants can appear low, we must understand the context in which this course was applied. From a total of around 80 people who are fully engaged in their daily health care routine and who voluntarily accepted to participate and complete an 8-lesson course in their free time, we consider that 30 is a successful result that reflects the need and enthusiasm generated by the proposed initiative.

Limitations

Relevant limitations include the small sample size that affects the generalization of the results, and the limited implementation time, as the learners have their own agenda and priorities.

For the analysis, we had to use nonparametric tests because some of the variables did not follow a normal distribution and mainly because of the sample size. It would be definitely more interesting to use parametric methods to prove that there were significant differences before and after the course and the interactions between professions and specialty in a more robust manner, but for that possibility we would have to have a greater number of participants.

In addition, we have the impact of the exposure to the pretest and the end-of-chapter practice questions, which were identical to the ones that were used in the posttest. Although there was an expert validation of the test questions used before and after the course, the test was not assessed for its reliability or validity metrics.

The validity of the satisfaction assessment should be carefully considered, because we used a nonvalidated translation to a Portuguese version of the user satisfaction questionnaire, as it was not possible to find in the literature a Portuguese validated one for e-learning systems.

Another limitation of this work is the fact that the socioprofessional questionnaire was applied only at the end of the intervention. This situation limits our information about the description of our sample and other factors that could influence the adherence and learning process.

Concerning the evaluation strategy, we also faced the risk of a slight bias because we did not control if learners resorted to external sources in order to provide correct answers to the tests.

Conclusions

Globally, this study highlights the importance of training neonatologists and other health care professionals in the neonatal care units to screen for congenital heart disease. We consider the high rate of participation an important aspect of our study (78%), which reflects the great interest shown by these professionals to promote their professional skills. They took advantage of this learning opportunity, which confirms that these health care professionals are committed to responding to new challenges and evolving paradigms.

This study contributes to the Brazilian continuing training programs, as we did not find any similar course related to neonatal screening of congenital heart disease. It would be interesting to conduct additional assessments to demonstrate effective consolidation of knowledge gain. For future work, we also intend to assess the remaining levels of the Kirkpatrick framework, “behavior” and “results,” with respect to change in neonatal screening behavior and improved congenital heart disease detection. To do so, we plan to measure the number of telemedicine consultations conducted by the participants after the course and the number of congenital heart disease detected by them.

Our global results show that e-learning can provide statistically relevant knowledge gains in health care professionals in a neonatal screening context. We believe that this study underlines the importance of e-learning as a viable technology for training, especially in impoverished contexts. E-learning should be considered for continuing medical education in low- and middle-income countries, not only due to budget constraints, but also due to resource-constrained environments.

Acknowledgments

All the authors have made a significant contribution to this manuscript and have approved the final paper, thus meeting the criteria for authorship. All those entitled to authorship are listed as authors.

The authors thank Real Hospital Português, UCMF, for providing and approving images for the course, and the Faculty of Medicine of the University of Porto for hosting the Moodle course.

Abbreviations

CME

continuing medical education

LMS

learning management system

Moodle

Modular Object Oriented Dynamic Learning Environment

Footnotes

Conflicts of Interest: None declared.

References

  • 1.Bernier P, Stefanescu A, Samoukovic G, Tchervenkov CI. The challenge of congenital heart disease worldwide: epidemiologic and demographic facts. Semin Thorac Cardiovasc Surg Pediatr Card Surg Annu. 2010;13(1):26–34. doi: 10.1053/j.pcsu.2010.02.005. [DOI] [PubMed] [Google Scholar]
  • 2.Knowles R, Griebsch I, Dezateux C, Brown J, Bull C, Wren C. Newborn screening for congenital heart defects: a systematic review and cost-effectiveness analysis. Health Technol Assess. 2005 Nov;9(44):1–152, iii. doi: 10.3310/hta9440. http://www.journalslibrary.nihr.ac.uk/hta/volume-9/issue-44. [DOI] [PubMed] [Google Scholar]
  • 3.Huber J, Peres VC, Santos TJ, Beltrão L, Baumont AC, Cañedo AD, Schaan BD, Pellanda LC. Congenital heart diseases in a reference service: clinical evolution and associated illnesses. Arq Bras Cardiol. 2010 Mar;94(3):313–8, 333. doi: 10.1590/s0066-782x2010000300009. http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0066-782X2010000300009&lng=en&nrm=iso&tlng=en. [DOI] [PubMed] [Google Scholar]
  • 4.Tubman TR, Shields MD, Craig BG, Mulholland HC, Nevin NC. Congenital heart disease in Down's syndrome: two year prospective early screening study. BMJ. 1991 Jun 15;302(6790):1425–7. doi: 10.1136/bmj.302.6790.1425. http://europepmc.org/abstract/MED/1829969. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Fixler DE, Nembhard WN, Xu P, Ethen MK, Canfield MA. Effect of acculturation and distance from cardiac center on congenital heart disease mortality. Pediatrics. 2012 Jun;129(6):1118–24. doi: 10.1542/peds.2011-3114. http://pediatrics.aappublications.org/cgi/pmidlookup?view=long&pmid=22566422. [DOI] [PubMed] [Google Scholar]
  • 6.Mattos S, Hazin SM, Regis CT, de Araújo JS, Albuquerque FC, Moser LR, Hatem T, de Freitas C, Mourato FA, Tavares TR, Gomes R, Severi R, Santos CR, da Silva F, Rezende JL, Vieira PC, de Lima FJ. A telemedicine network for remote paediatric cardiology services in north-east Brazil. Bull World Health Organ. 2015 Dec 01;93(12):881–7. doi: 10.2471/BLT.14.148874. http://www.scielosp.org/scielo.php?script=sci_arttext&pid=BLT.14.148874&lng=en&nrm=iso&tlng=en. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Mattos S, Hazin SM, Regis CT, Soares de Araújo J, Albuquerque F, Moser LR, Hatem T, Gomes de Freitas CP, Mourato FA, Tavares TR, Gomes RG, Severi R, Santos CR, Ferreira da Silva J, Rezende JL, Vieira PC, de Lima FJ. A telemedicine network for remote paediatric cardiology services in north-east Brazil. Bull World Health Organ. 2015 Dec 01;93(12):881–7. doi: 10.2471/BLT.14.148874. http://www.scielosp.org/scielo.php?script=sci_arttext&pid=BLT.14.148874&lng=en&nrm=iso&tlng=en. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Ruiz JG, Mintzer MJ, Leipzig RM. The impact of E-learning in medical education. Acad Med. 2006 Mar;81(3):207–12. doi: 10.1097/00001888-200603000-00002. [DOI] [PubMed] [Google Scholar]
  • 9.Sandars J. Cost Effectiveness in Medical Education. United Kingdom: Radcliffe Publishing; 2010. [Google Scholar]
  • 10.Frehywot S, Vovides Y, Talib Z, Mikhail N, Ross H, Wohltjen H, Bedada S, Korhumel K, Koumare AK, Scott J. E-learning in medical education in resource constrained low- and middle-income countries. Hum Resour Health. 2013 Feb 04;11:4. doi: 10.1186/1478-4491-11-4. https://human-resources-health.biomedcentral.com/articles/10.1186/1478-4491-11-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Bouhnik D, Marcus T. Interaction in distance-learning courses. J Am Soc Inf Sci Technol. 2006;57:299–305. [Google Scholar]
  • 12.Liaw S, Huang H, Chen G. Surveying instructor and learner attitudes toward e-learning. Computers & Education. 2007;49:1066–80. doi: 10.1016/j.compedu.2006.01.001. [DOI] [Google Scholar]
  • 13.Raab R, Ellis W, Abdon B. Multisectoral partnerships in e-learning. The Internet and Higher Education. 2001 Jan;4(3-4):217–29. doi: 10.1016/S1096-7516(01)00067-7. [DOI] [Google Scholar]
  • 14.Liaw S. Investigating students’ perceived satisfaction, behavioral intention, and effectiveness of e-learning: A case study of the Blackboard system. Computers & Education. 2008 Sep;51(2):864–73. doi: 10.1016/j.compedu.2007.09.005. [DOI] [Google Scholar]
  • 15.McKimm J, Jollie C, Cantillon P. ABC of learning and teaching: web based learning. BMJ. 2003 Apr 19;326(7394):870–3. doi: 10.1136/bmj.326.7394.870. http://europepmc.org/abstract/MED/12702625. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Wutoh R, Boren SA, Balas EA. eLearning: a review of internet-based continuing medical education. J Contin Educ Health Prof. 2004;24(1):20–30. doi: 10.1002/chp.1340240105. [DOI] [PubMed] [Google Scholar]
  • 17.Abdellatief M. A technique for quality evaluation of e-learning from developers perspective. American Journal of Economics and Business Administration. 2011 Jan 01;3(1):157–64. doi: 10.3844/ajebasp.2011.157.164. [DOI] [Google Scholar]
  • 18.Zhang D, Zhao J, Zhou L, Nunamaker J. Can e-learning replace classroom learning? Commun ACM. 2004 May 01;47(5):75–9. doi: 10.1145/986213.986216. [DOI] [Google Scholar]
  • 19.Capper J. IST. 2001. E-learning growth and promise for the developing world http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.600.3600&rep=rep1&type=pdf.
  • 20.Abdullah H. Queen’s University. 2012. [2017-05-03]. The development of an introductory e-learning module on echocardiography https://qspace.library.queensu.ca/bitstream/handle/1974/7161/Abdullah_Hanin_201202_MSC.pdf;jsessionid=96A797355F0A45AB86144672136706BB?sequence=1 .
  • 21.Wong G, Greenhalgh T, Pawson R. Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Med Educ. 2010;10:12. doi: 10.1186/1472-6920-10-12. http://www.biomedcentral.com/1472-6920/10/12. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Omar A, Kalulu D, Alijani G. Management of innovative e-learning environments. AELJ. 2011;15:37–64. [Google Scholar]
  • 23.Harun M. Integrating e-learning into the workplace. Internet High Educ. 2001 Jan;4(3-4):301–10. doi: 10.1016/S1096-7516(01)00073-2. [DOI] [Google Scholar]
  • 24.Nisar T. E-Learning in public organizations. Public Personnel Management. 2004 Mar;33(1):79–88. doi: 10.1177/009102600403300106. [DOI] [Google Scholar]
  • 25.Walsh K. E-learning in medical education: problems with research. J Contemp Med Edu. 2016;4(3):132–4. doi: 10.5455/jcme.20161007103941. [DOI] [Google Scholar]
  • 26.Gibbons AS, Fairweather PG. Computer-based Instruction: Design and Development. Eaglewood Cliffs, New Jersey: Educational Technology Publications; 1998. [Google Scholar]
  • 27.Loftus J, Stavraky T, Urquhart B. Design it yourself (DIY): in-house instructional design for online pharmacology. Adv Health Sci Educ Theory Pract. 2014 Dec;19(5):645–59. doi: 10.1007/s10459-013-9492-2. [DOI] [PubMed] [Google Scholar]
  • 28.Reyes NR, Candeas PV, Galan SG, Viciana R, Canadas F, Reche PJ. Comparing open-source e-learning platforms from adaptivity point of view. EAEEIE Annual Conference; June 22-24, 2009; Valencia, Spain. IEEE; 2009. [Google Scholar]
  • 29.Núñez JC, Cerezo R, Bernardo A, Rosário P, Valle A, Fernández E, Suárez N. Implementation of training programs in self-regulated learning strategies in Moodle format: results of a experience in higher education. Psicothema. 2011 Apr;23(2):274–81. [PubMed] [Google Scholar]
  • 30.Mattos SS. Cardiologia para o Pediatra - Manual Pratico para o Reconhecimento e Manuseio das Cardiopatias na Infancia. Brasil: Caduceus; 2004. [Google Scholar]
  • 31.Lai WW, Geva T, Shirali GS, Frommelt PC, Humes RA, Brook MM, Pignatelli RH, Rychik J, Task Force of the Pediatric Council of the American Society of Echocardiography. Pediatric Council of the American Society of Echocardiography Guidelines and standards for performance of a pediatric echocardiogram: a report from the Task Force of the Pediatric Council of the American Society of Echocardiography. J Am Soc Echocardiogr. 2006 Dec;19(12):1413–30. doi: 10.1016/j.echo.2006.09.001. [DOI] [PubMed] [Google Scholar]
  • 32.Mertens L, Helbing W, Sieverding L, Daniels O, Working Group on Cardiac Imaging of the Association for European Paediatric Cardiology Guidelines from the Association for European Paediatric Cardiology: standards for training in paediatric echocardiography. Cardiol Young. 2005 Aug;15(4):441–2. doi: 10.1017/S1047951105000958. [DOI] [PubMed] [Google Scholar]
  • 33.Galloway D. Evaluating distance delivery and e-learning is kirkpatrick's model relevant? Nonprofit Management Leadership. 2005 Apr;44(4):21–7. doi: 10.1002/pfi.4140440407. [DOI] [Google Scholar]
  • 34.Alliger GM, Janak EA. Kirkpatrick's levels of training criteria: thirty years later. Pers Psychol. 1989 Jun;42(2):331–42. doi: 10.1111/j.1744-6570.1989.tb00661.x. [DOI] [Google Scholar]
  • 35.Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-based learning in the health professions: a meta-analysis. JAMA. 2008 Sep 10;300(10):1181–96. doi: 10.1001/jama.300.10.1181. [DOI] [PubMed] [Google Scholar]
  • 36.Chumley-Jones HS, Dobbie A, Alford CL. Web-based learning: sound educational method or hype? a review of the evaluation literature. Acad Med. 2002 Oct;77(10 Suppl):S86–93. doi: 10.1097/00001888-200210001-00028. [DOI] [PubMed] [Google Scholar]
  • 37.Cook DA, Garside S, Levinson AJ, Dupras DM, Montori VM. What do we mean by web-based learning? a systematic review of the variability of interventions. Med Educ. 2010 Aug;44(8):765–74. doi: 10.1111/j.1365-2923.2010.03723.x. [DOI] [PubMed] [Google Scholar]
  • 38.Curran VR, Fleet L. A review of evaluation outcomes of web-based continuing medical education. Med Educ. 2005 Jun;39(6):561–7. doi: 10.1111/j.1365-2929.2005.02173.x. [DOI] [PubMed] [Google Scholar]
  • 39.Cobb SC. Internet continuing education for health care professionals: an integrative review. J Contin Educ Health Prof. 2004;24(3):171–80. doi: 10.1002/chp.1340240308. [DOI] [PubMed] [Google Scholar]
  • 40.Lahti M, Hätönen H, Välimäki M. Impact of e-learning on nurses' and student nurses knowledge, skills, and satisfaction: a systematic review and meta-analysis. Int J Nurs Stud. 2014 Jan;51(1):136–49. doi: 10.1016/j.ijnurstu.2012.12.017. [DOI] [PubMed] [Google Scholar]
  • 41.Schittek M, Mattheos N, Lyon HC, Attström R. Computer assisted learning. A review. Eur J Dent Educ. 2001 Aug;5(3):93–100. doi: 10.1034/j.1600-0579.2001.050301.x. [DOI] [PubMed] [Google Scholar]
  • 42.Feng J, Chang Y, Chang H, Erdley WS, Lin C, Chang Y. Systematic review of effectiveness of situated e-learning on medical and nursing education. Worldviews Evid Based Nurs. 2013 Aug;10(3):174–83. doi: 10.1111/wvn.12005. [DOI] [PubMed] [Google Scholar]
  • 43.Curran VR, Fleet LJ, Kirby F. A comparative evaluation of the effect of internet-based CME delivery format on satisfaction, knowledge and confidence. BMC Med Educ. 2010 Jan 29;10:10. doi: 10.1186/1472-6920-10-10. http://europepmc.org/abstract/MED/20113493. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Maloney S, Chamberlain M, Morrison S, Kotsanas G, Keating JL, Ilic D. Health professional learner attitudes and use of digital learning resources. J Med Internet Res. 2013;15(1):e7. doi: 10.2196/jmir.2094. http://www.jmir.org/2013/1/e7/ [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Stark CM, Graham-Kiefer ML, Devine CM, Dollahite JS, Olson CM. Online course increases nutrition professionals' knowledge, skills, and self-efficacy in using an ecological approach to prevent childhood obesity. J Nutr Educ Behav. 2011;43(5):316–22. doi: 10.1016/j.jneb.2011.01.010. [DOI] [PubMed] [Google Scholar]
  • 46.Maki WS. Assessment and Education Research. Texas Tech University, Lubbock, TX: AAAS; 2004. Evaluating Outcomes; pp. 27–32. [Google Scholar]
  • 47.Wang Y. Assessment of learner satisfaction with asynchronous electronic learning systems. Information & Management. 2003 Oct;41(1):75–86. doi: 10.1016/S0378-7206(03)00028-4. [DOI] [Google Scholar]
  • 48.Xiao L, Dasgupta S. Measurement of user satisfaction with web-based information systems: An empirical study. Eighth Americas Conference on Information Systems; December 2002; AIS Electronic Library (AISeL). 2002. pp. 1149–55. [Google Scholar]
  • 49.Moreira IC, Ventura SR, Ramos I, Rodrigues PP. Development and assessment of an e-learning course on breast imaging for radiographers: a stratified randomized controlled trial. J Med Internet Res. 2015 Jan 05;17(1):e3. doi: 10.2196/jmir.3344. http://www.jmir.org/2015/1/e3/ [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50.Marôco J. Análise Estatística com o SPSS Statistics. Lisboa: ISPA; 2014. [Google Scholar]
  • 51.Nkenke E, Vairaktaris E, Bauersachs A, Eitner S, Budach A, Knipfer C, Stelzle F. Acceptance of technology-enhanced learning for a theoretical radiological science course: a randomized controlled trial. BMC Med Educ. 2012 Mar 30;12:18. doi: 10.1186/1472-6920-12-18. https://bmcmededuc.biomedcentral.com/articles/10.1186/1472-6920-12-18. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 52.Mahnken AH, Baumann M, Meister M, Schmitt V, Fischer MR. Blended learning in radiology: is self-determined learning really more effective? Eur J Radiol. 2011 Jun;78(3):384–7. doi: 10.1016/j.ejrad.2010.12.059. [DOI] [PubMed] [Google Scholar]

Articles from JMIR Medical Education are provided here courtesy of JMIR Publications Inc.

RESOURCES