Abstract
Objective:
Sleep disorders are highly prevalent across all age groups but often remain undiagnosed and untreated, resulting in significant health consequences. To overcome an inadequacy of available curricula and learner and instructor time constraints, this study sought to determine if an online sleep medicine curriculum would achieve equivalent learner outcomes when compared with traditional, classroom-based, face-to-face instruction at equivalent costs.
Method:
Medical students rotating on a required clinical clerkship received instruction in 4 core clinical sleep-medicine competency domains in 1 of 2 delivery formats: a single 2.5-hour face-to-face workshop or 4 asynchronous e-learning modules. Immediate learning outcomes were assessed in a subsequent clerkship using a multiple-choice examination and standardized patient station, with long-term outcomes assessed through analysis of students' patient write-ups for inclusion of sleep complaints and diagnoses before and after the intervention. Instructional costs by delivery format were tracked. Descriptive and inferential statistical analyses compared learning outcomes and costs by instructional delivery method (face-to-face versus e-learning).
Results:
Face-to-face learners, compared with online learners, were more satisfied with instruction. Learning outcomes (i.e., multiple-choice examination, standardized patient encounter, patient write-up), as measured by short-term and long-term assessments, were roughly equivalent. Design, delivery, and learner-assessment costs by format were equivalent at the end of 1 year, due to higher ongoing teaching costs associated with face-to-face learning offsetting online development and delivery costs.
Conclusions:
Because short-term and long-term learner performance outcomes were roughly equivalent, based on delivery method, the cost effectiveness of online learning is an economically and educationally viable instruction platform for clinical clerkships.
Citation:
Bandla H; Franco RA; Simpson D; Brennan K; McKanry J; Bragg D. Assessing learning outcomes and cost effectiveness of an online sleep curriculum for medical students. J Clin Sleep Med 2012;8(4):439-443.
Keywords: Learning outcomes, cost effectiveness, on-line sleep curriculum
Sleep disorders are highly prevalent in the general population across all age groups1,2 and can result in significant multiorgan dysfunction, with serious health consequences.3,4 If diagnosed early and appropriately, most sleep disorders are treatable.5 However, despite the prevalence of sleep disorders and the relationship of these disorders to underlying medical conditions, physician performance on sleep-related knowledge or skills assessments, from medical students through practicing primary care physicians, is poor at best.6,7 As a result, many sleep disorders are undiagnosed in children and in adults.8–10 Underdiagnosis of sleep disorders has been historically attributed to the limited inclusion of sleep medicine in medical school curricula. A 1978 survey by the American Sleep Disorders Association showed that 46% of medical schools provided no education in sleep medicine.11 Approximately 10 years later, a survey by the National Commission on Sleep Disorders Research reported that, despite some improvement in time spent teaching about sleep, at least 37% of medical schools continued to provide minimal sleep medicine education.12 Ongoing concerns of government agencies and other professional groups about the paucity of sleep disorders-related medical education resulted in several strategies to increase sleep medicine education, including formation of a special task force of American Sleep Disorders Association on medical education in sleep and sleep disorders13 and the Sleep Academic Award Program of the National Commission on Sleep Disorders Research.14 Although the Sleep Academic Awardees have developed curricula models and have successfully implemented sleep education in their respective institutions, incorporating sleep medicine into the curricula of medical schools remains difficult, due to the inadequacy of curriculum, and time constraints of faculty and learners, and the limited availability of qualified physician-sleep educators.
BRIEF SUMMARY
Current Knowledge/Study Rationale: Medical students receive limited education in the rapidly expanding science of sleep medicine sleep due to curricular and faculty time constraints and physician-teacher knowledge gaps. The aim of the study is to determine the cost and educational effectiveness of on-line sleep medicine curriculum.
Study Impact: An online sleep medicine curriculum is educationally effective. In the long-term on-line instruction addresses curriculum time and sleep medicine expert availability constraints providing a cost-effective solution to advance medical student sleep medicine education.
Delivery of instruction over the Internet, including the use of online courses and learning-management systems, has been used to enhance access to content designed by experts in particular topic areas and to address inadequacies in the curriculum and time limitations of the teacher and learner. The results of a comprehensive meta-analysis on the effectiveness of Internet-based learning in the health professions found that the effects of Internet delivery of instruction were equivalent to those using traditional instruction methods.15 However, the authors of this review reported that few studies that compared Internet-based with traditional instruction also assessed the learners' skills (< 16%) or behaviors and patient effects (< 8%); the review did not look at the cost of Internet-based compared to traditional instruction. Thus, the literature provides limited guidance to assist specialty educators whose clinical content is important but must decide if it is cost effective to deliver instruction online.
Therefore this study sought to address the issues related to curriculum content, learner and instructor time constraints, and the availability of experts by developing and evaluating an online sleep medicine curriculum. The evaluation included the short- and long-term performance-based learner outcomes and the costs associated with developing and implementing the online sleep medicine curriculum, as well as comparing these results with those of a traditional face-to-face workshop.
METHODS
Curriculum Competencies and Design
Four competencies previously identified by Strohl et al.16 as core knowledge and skills for medical education in sleep medicine served as the common curriculum content for third-year medical students and focused on the students' ability to (1) articulate the putative nature of sleep medicine; (2) identify common sleep disorders with associated signs and symptoms; (3) perform a sleep history; and (4) manage common sleep disorders to improve sleep, reduce sleepiness, or both. Four instructional modules were developed with the competencies threaded across each: (1) sleep physiology and chronobiology, (2) sleep disordered breathing, (3) hypersomnias (excessive daytime sleepiness), and (4) parasomnias.
Specific objectives for each module were outlined. The face-to-face and e-learning modules were then developed in parallel to ensure concordance between formats. The learning modules were then pilot tested and revised based on feedback from fourth-year medical students who were completing an ambulatory-medicine rotation; the modules were modified as needed to enhance clarity and education impact. The final versions of both formats were then implemented. A PowerPoint presentation (Office 2003, Version 11, Microsoft, Redmond, WA) was the primary instruction method for both delivery formats, and each learning point was explicitly linked to the objectives. Content was drawn from a variety of resources, including sleep medicine textbooks, journal review articles, and existing electronic learning materials. The key objectives were further enhanced through incorporation of case scenarios and video clips. The online modules were finalized using Microsoft Producer (Producer 2003Microsoft) to support incorporation of video clips, images, and oral narration.17 The modules were posted on the Medical College of Wisconsin education website, ANGEL, (A New Global Environment for Learning) e-learning platform for courses and course evaluation for medical students.
Study Design
All third-year medical students rotating on the required pediatrics clerkship from July1, 2005, to June 30, 2006, participated. The students were alternately assigned to the face-to-face or e-learning delivery format, with a balanced number participating in each format through the year. Students, per institutional review board-approved protocol, were informed that their participation and test results would not be included in their clerkship evaluation.
Prior to completion of the instruction, all students completed the online multiple-choice examination. At the end of the rotation, students also participated in a post-instruction standardized patient interaction that was focused on sleep and incorporated into an objective structured clinical examination (OSCE).
Students assigned to the face-to-face workshop met on a single occasion during the rotation for 2.5 hours. Two physician-sleep medicine educators served as the workshop instructors, providing the live narrative for the PowerPoint slides and interactive discussion throughout the workshop. No enduring materials were distributed.
The e-learning group was given access to the sleep-education modules beginning on the first day of the rotation after completing the multiple-choice pre-intervention test. They were encouraged to complete the modules by the end of the rotation. Faculty then audited the students' write-ups of results obtained through the patient history and physical examination. Audits also included sleep content for those students who then participated in an ambulatory medicine rotation after they completed the pediatrics rotation.
Learning Outcome Instruments and Data Collection
The Kirkpatrick 4-stage training-evaluation model18 was used to frame instrumentation decisions at 3 levels: (1) satisfaction or reaction related to instruction, (2) learning, and (3) application or transfer of behavior to practice. Learner satisfaction (Level 1) was assessed for both e-learning and face-to-face formats, with evaluation survey items focused on clarity of objectives, organization, effectiveness of instruction, and impact on learning. The survey rating scales and the number of questions were not identical (due to different delivery formats), but similar items were grouped into Effective Instruction and Apply Knowledge, with the response data transformed to create equivalent scales (1 = positive).
Forty-seven multiple-choice questions were developed to assess key objectives (Level 2) per a test blueprint composed of core instruction domains. These questions were tested with 134 fourth-year students who had no prior exposure to a sleep medicine curriculum. Using the item statistics, the most effective questions were retained to appropriately sample the content blueprint, resulting in a 20-item multiple-choice test that was administered before and after the instructional intervention. All students completed the online multiple-choice test on the first day of the rotation (pre-intervention test) and again within 2 weeks of completing the education and before they completed the clerkship (post-intervention test). The multiple-choice post-intervention test was locked, only appearing once the learner had viewed all 4 modules (e-learning group) or after the workshop (face-to-face group). The post-intervention test could be opened only once and was timed to lock out at 30 minutes.
A standardized patient case was developed to highlight students' sleep-specific history-gathering skills and was included in the required end-of-clerkship OSCE (Levels 2-3). The patient scenario involved a grade-school–aged child with school-performance issues. The child was accompanied by a parent, who was trained to ask the medical student whether the child's sleep disturbances, which the parent had observed, could be the cause of the child's inattention at school. The medical student's performance was rated using a checklist focused on sleep history—including sleep disordered breathing, sleepiness, and sleep quality and quantity—along checklist items for a focused physical examination.
Assessing the students' learning retention and transfer and application of sleep medicine knowledge to patient care (Level 3) required history-and-physical write-ups for approximately 25% of the pediatric clerkship students who went on to complete a required ambulatory medicine rotation during the study period (2-12 months after completion of the sleep medicine education). A history-and-physical sleep-audit coding sheet, designed to assess students' long-term knowledge retention and transfer of knowledge to patient care, was developed using validated screening history mnemonics such as BEARS19 (bedtime issues, excessive daytime sleepiness, awakenings during the night, regularity and duration of sleep, snoring) and ISNORED6 (insomnia, snoring and sleep quality, not breathing, older or obese, restorative or refreshing sleep, excessive day time sleepiness, drugs).
As part of her or his ambulatory-medicine rotation, each student was required to complete 4 write-ups. These de-identified patient write-ups were audited for content regarding sleep disorders and sleep history. An audit form was used to score the presence or inclusion of sleep-related complaints and incorporation of a sleep-related assessment and care plan. The audit form—which used a checklist structure to record the presence or absence of sleep-related details in the history of present illness, review of systems, or past medical history (1 point)—and the presence of sleep-specific details in the assessment and plan (1 point) resulted in a maximum of 2 points per write-up and a total maximum score of 8 points per student (4 write-ups × 2 points).
Audit-coding consistency was assessed by 2 physicians, who independently coded a common sample of the history and physical examination to determine concordance (Pearson correlation, 0.6). The coded data from the audit sheets were transferred to an Excel spreadsheet (Microsoft) for analysis, with notations made regarding face-to-face or e-learning instruction format, to enable comparison of findings by delivery method.
Development and Implementation Time and Cost Data
Time logs submitted by physician authors and educators from project commencement to project completion provided longitudinal time records associated with curriculum development and revisions, including supplementary materials, production of the face-to-face and on-line education models, and face-to-face teaching time. Faculty costs, based on hours invested throughout the project, were calculated using the most recent Association of America Medical Colleges Faculty Database Report,20 so that data would be generalizable to other medical schools. Staff costs were calculated based on actual salaries, including fringe benefits; the college's Office of Human Resources established these salaries through a comparative analysis with similar positions nationally. Supplies and expenses were tracked using the college's internal budgeting and purchase-order system (e.g., CD-ROMs, duplication or materials, video production) during the study period by method.
Statistical Analysis
Student performance on the pre-intervention and post-intervention multiple-choice tests were compared based on the learning methods (e-learning vs face-to-face) using a repeated-measures analysis of variance (SPSS for Windows, version IBM, Chicago, IL). Audit data sets from the standardized-patient OSCE and patient notes were analyzed by the student t-test comparing face-to-face and e-learning groups.
RESULTS
Results are reported by Kirkpatrick evaluation model level, each with a specific sample size, because sample size varied by instrument due to the longitudinal nature of the data collection process (see Figure 1).
Figure 1. Diagram outlining the number (n) of students enrolled in Face to Face vs Online and the results of three levels of learning outcomes.
Level 1: Satisfaction with Instructional Format
Evaluations were submitted by 173 students prior to the end of the pediatric clerkship, with slightly more students completing their instruction via online learning (n = 111) versus the face-to-face format (n = 62) due to the students' rotation schedules. The face-to-face learners rated instruction effectiveness (mean ± SD) (1.73 ± 0.80; p < 0.01) and application of sleep knowledge (1.77 ± 0.89; p < 0.05) more positively, compared with the online learners (instruction effectiveness: 2.12 ± 0.55; application of sleep knowledge: 2.04 ± 0.54) using a 5-point rating scale, with 1 being the most positive.
Level 2: Knowledge of Sleep Medicine via Multiple-Choice Examination
Data from the pre-intervention and post-intervention multiple-choice sleep medicine examination was available for 207 students: 100 face-to-face learners and 107 online learners. Overall, student performance on the multiple-choice examination revealed a significant pre-intervention-to-post-intervention increase from 53% to 73%. Performance increased significantly by group: face-to-face learners' performance increased by 16% (from 52% pre-intervention to 68% post-intervention) and on-line learners' performance increased by 22% (from 55% pre-intervention to 77% post-intervention) (p < 0.001, effect size [partial eta2] = 0.59).
Level 3: Transfer or Application of Learning Assessed via Performance in Relationship to the Standardized-Patient and Patient Write-ups
Students' performance data from the results of the patient history and physical examination were available from the pediatric end-of-rotation OSCEs for 190 students (93 face-to-face, 97 online) and were reported as the number of checklist items completed (maximum 35). No significant differences in OSCE performance were obtained by group: face-to-face learners 23.3 (SD 3.3), on-line learners 23.9 (SD 3.1).
Fifty-eight students who completed the pediatric sleep medicine education enrolled in the ambulatory-medicine rotation during the study period; write-ups were available for audits from all of these students. No significant difference was found using t-test analysis (p = 0.083) by learner group: the mean write-up score of the 32 face-to-face learners was 0.89 (SD = 1.28), the mean score of the 26 on-line learners (n = 26) was 0.92 (SD = 1.02), and the standard error of the mean was 0.2 for both groups.
Cost
A comparison of the costs of curriculum design and delivery, based upon delivery method, revealed that the development costs for physician authors were roughly equivalent (see Table 1). However, the educator-related costs associated with on-line curriculum design added an additional 160 hours to the costs during the first year. Total first-year development and instruction-delivery costs were equivalent: additional physician hours were required for the repeated delivery of the face-to-face curriculum ($21,640) and additional educator costs were required for support of on-going e-learning ($21,752).
Table 1.
First-year development and delivery costs of a sleep medicine curriculum for third-year medical students
CONCLUSIONS
The primary objective of this study was to determine the cost and learning effectiveness of online education compared with traditional face-to-face instruction, which is repeated with each clerkship rotation. This study demonstrates that, although online delivery requires a start-up cost comparable to that of traditional face-to-face instruction during the initial implementation year, these costs are offset when the curriculum is incorporated in a required clinical rotation because of the increased costs associated with the need to repeatedly present the same material throughout the year with the face-to-face instruction. Costs associated with online instruction after initial development are minimal, whereas physician delivery of face-to-face instruction is ongoing.
Learner satisfaction for the students who completed the face-to-face instruction was higher, but learner performance was equivalent between groups or was slightly better for online learners. Student satisfaction may be the result of the “absent instructor” phenomenon common with the non-moderated online format. Students in the face-to-face group were able to directly interact, to ask questions specific to their learning needs, and to have personal contact with sleep medicine physicians.
Study conclusions may be limited by several factors. (1) Student exposure to sleep medicine-related patients during the clinical portions of their clerkships (per clinical experience log data recorded by students through their third-year clerkships) shows minimal sleep-related patient exposure. (2) The number of students who completed write-ups of the results of their patients' histories and physical examinations was limited. This was due to the time lag between the focused sleep education and the collection of the write-ups during the ambulatory-medicine rotation. Unfortunately, this could not be controlled by the authors; because there were no significant differences between face-to-face and on-line learners, further analysis was not preformed. (3) An inherent deficiency is associated with the use of the log-based medical records of clinical encounters as a valid measure of transfer of knowledge. (4) The study was performed in a private medical school that is located in the upper Midwest and is attended by students who are demographically representative of all US students; hence, additional studies may be needed to determine the generalizability of our findings to students from medical schools in other locations and those in public institutions.
Cognizant of these limitations, this study does demonstrate the cost effectiveness of online learning as an economically and educationally viable instruction platform for clinical clerkships.
DISCLOSURE STATEMENT
This was not an industry supported study. The authors have indicated no financial conflicts of interest.
ACKNOWLEDGMENTS
The authors gratefully acknowledge Paola Palma Sisto, M.D., Clerkship Director, Department of Pediatrics, Medical College of Wisconsin, and Ann Maguire, M.D., Clerkship Director, Department of Internal Medicine, Medical College of Wisconsin, for their cooperation and support. We also thank all the students for their dedicated participation in this study.
This project was partially supported by an education research award from American Sleep Medicine Foundation (Grant #25-ER-04).
REFERENCES
- 1.National Sleep Foundation. 2008 Sleep in America Poll: Sleep, performance and the work place. 2008. Mar 3, [Accessed on: April 6, 2011]. Available at http://www.sleepfoundation.org/article/sleep-america-polls/2008-sleep-performance-and-the-workplace.
- 2.Young T, Paulta M, Dempsey J, Weber S, Badr S. The occurrence of sleep-disordered breathing among middle-aged adults. N Engl J Med. 1993;328:1230–35. doi: 10.1056/NEJM199304293281704. [DOI] [PubMed] [Google Scholar]
- 3.Punjabi NM, Caffo BS, Goodwin JL, et al. Sleep-disordered breathing and mortality: a prospective cohort study. PLoS Med. 2009;6:e1000132. doi: 10.1371/journal.pmed.1000132. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Gozal D. Sleep, sleep disorders and inflammation in children. Sleep Med. 2009;10:S12–6. doi: 10.1016/j.sleep.2009.07.003. [DOI] [PubMed] [Google Scholar]
- 5.Wei JL, Bond J, Mayo MS, Smith HJ, Reese M, Weatherly RA. Improved behavior and sleep after adenotonsillectomy in children with sleep-disordered breathing: long-term follow-up. Arch Otolaryngol Head Neck Surg. 2009;135:642–6. doi: 10.1001/archoto.2009.74. [DOI] [PubMed] [Google Scholar]
- 6.Haponik EF, Frye AW, Richards B, et al. Sleep history is neglected diagnostic information. Challenges for primary care physicians. J Gen Intern Med. 1996;11:759–61. doi: 10.1007/BF02598994. [DOI] [PubMed] [Google Scholar]
- 7.Rosen RC, Zozula R, Jahn EG, Carson JL, Raymond R. Low rates of recognition of sleep disorders in primary care: comparison of a community-based versus clinical academic setting. Sleep Med. 2001;2:47–55. doi: 10.1016/s1389-9457(00)00043-5. [DOI] [PubMed] [Google Scholar]
- 8.Silverberg DS, Oksenberg A, Iaina A. Sleep related breathing disorders are common contributing factors to the production of essential hypertension but are neglected, underdiagnosed, and undertreated. Am J Hypertens. 1997;10:1319–25. doi: 10.1016/s0895-7061(97)00322-1. [DOI] [PubMed] [Google Scholar]
- 9.Ram S, Seirawan H, Kumar SK, Clark GT. Prevalence and impact of sleep disorders and sleep habits in the United States. Sleep Breath. 2010;14:63–70. doi: 10.1007/s11325-009-0281-3. [DOI] [PubMed] [Google Scholar]
- 10.Mindell JA, Moline ML, Zendell S, Brown L, Fry J. Pediatricians and sleep disorders: training and practice. Pediatrics. 1994;94:194–200. [PubMed] [Google Scholar]
- 11.Orr WC, Stahl ML, Dement WC, Reddington D. Physician education in sleep disorders. J Med Educ. 1980;55:367–9. doi: 10.1097/00001888-198004000-00009. [DOI] [PubMed] [Google Scholar]
- 12.Rosen RC, Rosekind M, Rosevear C, Cole WE, Dement WC. Physician education in sleep and sleep disorders: a national survey of US medical schools. Sleep. 1993;16:249–54. doi: 10.1093/sleep/16.3.249. [DOI] [PubMed] [Google Scholar]
- 13.Rosen R, Mahowald M, Chesson A, et al. The Taskforce 2000 survey on medical education in sleep and sleep disorders. Sleep. 1998;21:235–8. doi: 10.1093/sleep/21.3.235. [DOI] [PubMed] [Google Scholar]
- 14.Owens J. Introduction to special section: NIH Sleep Academic Award program. Sleep Med. 2005;6:45–6. doi: 10.1016/j.sleep.2004.11.001. [DOI] [PubMed] [Google Scholar]
- 15.Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-based learning in the health professions. JAMA. 2008;300:1181–96. doi: 10.1001/jama.300.10.1181. [DOI] [PubMed] [Google Scholar]
- 16.Strohl KP, Veasey S, Harding S, et al. Competency-based goals for sleep and chronobiology in undergraduate medical education. Sleep. 2003;26:333–6. doi: 10.1093/sleep/26.3.333. [DOI] [PubMed] [Google Scholar]
- 17.Mattheos N, Nattestad A, Attstrom R. Local CD-ROM in interaction with HTML documents over the Internet. Eur J Dent Educ. 2000;4:124–7. doi: 10.1034/j.1600-0579.2000.040306.x. [DOI] [PubMed] [Google Scholar]
- 18.Kirkpatrick D. San Francisco, CA: Berrett-Koehler Publishers; 1998. Evaluating training programs: the four levels. [Google Scholar]
- 19.Owens JA, Dalzell V. Use of the ‘BEARS’ sleep screening tool in a pediatric residents' continuity clinic: a pilot study. Sleep Med. 2005;6:63–9. doi: 10.1016/j.sleep.2004.07.015. [DOI] [PubMed] [Google Scholar]
- 20.Association of American Medical Colleges. Washington, DC: AAMC; 2003. Report on Medical School Faculty Salaries: 2001-2002. [Google Scholar]