Integrating Public Health–Oriented E-Learning Into Graduate Medical Education

Abstract

Objectives. In fall 2008, the New York City Department of Health and Mental Hygiene collaborated with Albert Einstein College of Medicine residency program directors to assess the effectiveness of an e-learning course on accurate death certificate completion among resident physicians.

Methods. We invited postgraduate year 1 and 2 (PGY1 and PGY2) residents (n = 227) to participate and administered a pretest, e-learning module, posttest, and course evaluation to PGY1 residents; PGY2 residents completed a pretest and survey only.

Results. In all, 142 residents (63%) participated. The average pretest scores for PGY2 residents (61%) and PGY1 residents (59%) were not significantly different. The PGY1 residents’ average test score increased significantly after taking the e-learning module (59% vs 72%; P < .01). The participants rated course length, delivery method, and utility highly.

Conclusions. Results suggest that e-learning can effectively integrate public health–oriented training into clinical residency programs.

The epidemic of preventable chronic diseases, the ongoing gap in health disparities, and the continued emergence of infectious diseases are among the 21st century’s leading public health challenges. To adequately respond, physicians require not only clinical skills, but also a basic understanding of population health.¹ In the United States, requirements for certification in the primary care specialties include competence in selected public health–oriented topics such as disease prevention, epidemiology, and systems-based practice.^2,3 In addition, all licensed physicians are expected to understand the role of the public health system and to comply with requirements for disease reporting and for certifying vital events. However, because integrating population health concepts into hospital- and clinic-based residency training can be challenging, physician knowledge of public health is often inadequate. In 2006 through 2008, 34% of graduating medical students believed that their medical school curricula did not devote sufficient attention to key public health topics.⁴

Completing death certificates correctly is a task for which residents receive little or no formal training.^5–7 Instead, they often learn on the job, with informal guidance provided by more senior colleagues or administrative staff.⁵ A small pilot study found that 66% of the medical students (n = 68) studied received no training in death certificate completion, and 91% of medical residents (n = 21) first encountered the process during their residencies.⁵ Without training, physicians are unlikely to accurately record the causes of death or to understand the broad implications of poorly completed death certificates. Cause of death data are critical to health policy development, resource allocation, and population-level assessments of disease burden, treatment, and prevention.

A random review of death certificates from January through June 2003 in New York City suggested that the city’s unexpectedly high rate of death from coronary heart disease may be attributable to misreporting on death certificates.⁶ In response, the New York City Department of Health and Mental Hygiene (Health Department) developed an e-learning module to teach correct completion of death certificates.⁸ In fall 2008, the Health Department collaborated with Albert Einstein College of Medicine residency program directors to evaluate the effectiveness and acceptability of this course among resident physicians.

METHODS

We invited postgraduate year 1 (PGY1) internal medicine and general surgery residents (n = 114) and postgraduate year 2 (PGY2) internal medicine, emergency medicine, and general surgery residents (n = 113) via e-mail to participate. We asked PGY1 residents to complete a pretest, e-learning course, posttest, and course evaluation and PGY2 residents to complete the same pretest administered to PGY1 residents and a brief survey describing their experience completing death certificates. The PGY2 residents did not take the e-learning module. We selected PGY1-level residents as the target group for the e-learning module because, though they have limited experience completing death certificates, they are expected to complete them accurately. Residents had 1 month to complete all components. Residency directors and Health Department staff sent e-mail reminders at 2 weeks and then again at 1 week before the end of the study to residents who had not completed the assigned tools. Participation was voluntary; incentives were not offered to those who participated. This activity represented the evaluation of an existing public health program and was deemed a nonresearch public health activity not subject to institutional review board review.

The e-learning module covered the importance of cause of death reporting, how to complete the cause of death section of the certificate, when to refer a case to the medical examiner, and special instructions for completing the death certificate in emergency situations, all topics identified by the Health Department’s Bureau of Vital Statistics as poorly understood.⁶ The module also included case-based exercises where course participants could practice completing death certificates. The test consisted of 10 multiple-choice questions drawn from key topics covered in the module. Forty percent of the test questions were case-based, requiring participants to write a cause of death statement or determine the sequence to be recorded on the death certificate based on a case history. E-learning module authors and Health Department public health preventive medicine residents developed and reviewed the test questions; a Health Department nosologist, a specialist in the classification of diseases, reviewed the final questions and answers for accuracy. The e-learning course and tests were deployed through a learning management system, which was used to score the tests. A course evaluation, also deployed online, gathered learner feedback on course length, content, and structure, and the usefulness of e-learning as a teaching method. A brief survey solicited information from both PGY1 and PGY2 residents on their experience completing death certificates.

We used a t-test comparison to determine significant differences between the average PGY1 pretest score and average PGY2 test score and between the average PGY1 pre- and posttest scores. We calculated χ² statistics to determine differences between PGY1 and PGY2 residents for the number of death certificates completed and knowledge in completing death certificates.

RESULTS

Of the 227 residents invited, 142 (63%) completed at least 1 component of the evaluation. Of the 68 PGY1 residents who participated, 59 (52% of all invited PGY1 residents) completed all 4 components of the evaluation; 74 PGY2 residents completed the pretest, and 55 (49% of all invited PGY2 residents) completed both the pretest and the experience survey.

The PGY1 and PGY2 average pretest scores were comparable (59% and 61%, respectively). The average PGY1 posttest score was significantly higher than the average PGY1 pretest score (72% vs 59%, respectively; P < .01; df = 118; Figure 1) and the average PGY2 pretest score (72% vs 61%, respectively; P < .001; df = 132). For 6 of the 10 test questions, more PGY1 residents answered each question correctly in the posttest than in the pretest by at least 10%, demonstrating improvement in knowledge in all key topics addressed in the e-learning module.

FIGURE 1—

Average test scores of residents at baseline and at posttest after taking the e-learning course on accurate death certificate completion: New York City, 2008.

Note. PGY = postgraduate year.

Before taking the course, PGY1 residents more frequently rated themselves as beginners or only slightly knowledgeable in completing death certificates than did PGY2 residents (71% and 55%, respectively). Fewer PGY1 residents had completed 3 or more death certificates than had PGY2 residents (32% vs 64%; P < .01; Figure 2). After completing the course, only 14% of PGY1 residents rated themselves as beginners or only slightly knowledgeable in completing death certificates (Figure 3).

FIGURE 2—

Number of death certificates completed by residency year at time of evaluation: New York City, 2008.

Note. PGY = postgraduate year.

FIGURE 3—

Postgraduate year 1 and 2 residents’ perceived expertise in completing death certificates: New York City, 2008.

Note. PGY = postgraduate year.

Among PGY1 residents, 91% rated the usefulness of the course as excellent, very good, or good; 95% indicated that it provided the right amount of information; 81% completed the course in 45 minutes or less time; 58% agreed that it was easy to find time to complete the course; 98% described it as easy to navigate; and 80% indicated that an e-learning module as opposed to an in-person class or a combination of e-learning and face-to-face instruction was the best way to deliver this training. Eighty-six percent of PGY1 and 87% of PGY2 residents thought that the course would be most appropriate during the first year of residency.

DISCUSSION

The Health Department’s Improving Cause of Death Reporting e-learning course is an effective means of training resident physicians in accurate completion of death certificates. Although other studies have evaluated the impact of interactive workshops, printed material, theoretical seminars, and videos,⁹ this is the first to our knowledge to examine the use of e-learning as a tool for training physicians on this topic. Furthermore, we found studies on the use of e-learning in clinical residency programs to be limited, and this appears to be the first study that has looked at e-learning to teach a public health–oriented skill.

The PGY1 residents’ average posttest score was significantly higher than their average pretest score, indicating increased knowledge after taking the course. Increased knowledge was demonstrated in all key topics covered by the module. Interestingly, despite their reported greater experience in completing death certificates, PGY2 residents’ average test score was not significantly different from the PGY1 average pretest score, suggesting that an additional year of residency training alone does not adequately train residents to accurately record cause of death. Other studies have also found that more experience and medical knowledge alone do not improve accuracy in death certificate completion.¹⁰ The PGY2 residents also had greater confidence in their abilities to complete the cause of death section of death certificates, suggesting poor awareness of their knowledge deficits.

Resident satisfaction with the course was high: most described the course as useful and easy to navigate, and nearly 60% stated that it was easy to find time to complete the course. However, despite multiple reminders, including from the residency program leadership, the course completion rate was not high, suggesting that additional strategies are needed to effectively deliver public health–oriented training through e-learning. Designing a series of short modules on key topics in public health that also fulfill existing competency requirements might increase the incentive for program directors to adopt them as a required component of the curriculum.

Among the limitations to this evaluation is the possibility that the test did not accurately measure knowledge of death certificate completion. Second, the majority of residents in this study were internal medicine residents; findings may not be generalizable to residents in other fields. The e-learning course on completing death certificates is applicable to residents’ clinical work; acceptability of a course may depend on its obvious relevance to residents’ existing duties. Finally, this evaluation was not designed to determine whether the increase in knowledge was sustained. Although the higher average posttest score is encouraging, we should consider retesting participants 6 to 12 months after the course to assess retention.

Demonstration that this e-learning course was an effective and acceptable tool for training busy clinical resident physicians lent strong support to the Health Department’s proposal that the course be mandatory for all providers who report deaths through the New York City electronic death registration system. A modification of the New York City Health Code based on this proposal, which became effective in 2010, imposes this training requirement on the majority of individuals completing death certificates in New York City.¹¹

The Health Department continues to assess e-learning as a viable and alternative means of integrating public health–oriented training into environments where resources and time to complete training are limited. The Health Department has begun developing an e-learning module on tobacco cessation primarily for a resident audience. As resources become increasingly scarce and residents’ time to complete training becomes more limited, graduate medical educators should give serious consideration to the use of e-learning as a method to integrate public health–oriented training into clinical residency programs. E-learning will afford residents the flexibility of completing training on their own time, and at their own pace.