Medical student case presentation performance and perception when using mobile learning technology in the emergency department

Abstract

Background

Hand-held mobile learning technology provides opportunities for clinically relevant self-instructional modules to augment traditional bedside teaching. Using this technology as a teaching tool has not been well studied. We sought to evaluate medical students’ case presentation performance and perception when viewing short, just-in-time mobile learning videos using the iPod touch prior to patient encounters.

Methods

Twenty-two fourth-year medical students were randomized to receive or not to receive instruction by video, using the iPod Touch, prior to patient encounters. After seeing a patient, they presented the case to their faculty, who completed a standard data collection sheet. Students were surveyed on their perceived confidence and effectiveness after using these videos.

Results

Twenty-two students completed a total of 67 patient encounters. There was a statistically significant improvement in presentations when the videos were viewed for the first time (p=0.032). There was no difference when the presentations were summed for the entire rotation (p=0.671). The reliable (alpha=0.97) survey indicated that the videos were a useful teaching tool and gave students more confidence in their presentations.

Conclusions

Medical student patient presentations were improved with the use of mobile instructional videos following first time use, suggesting mobile learning videos may be useful in medical student education. If direct bedside teaching is unavailable, just-in-time iPod touch videos can be an alternative instructional strategy to improve first-time patient presentations by medical students.

Learning in the clinical setting has traditionally been centered around the bedside. William Osler championed this approach more than 100 years ago (1). However, in a busy academic emergency department (ED), the amount of bedside teaching faculty can provide may be limited by increasing clinical demands, including higher patient volumes, decreased time with patients and constant interruptions (2). Academic faculty have the additional challenge of interacting with and educating multiple learners at different levels of training. The challenge for emergency medicine (EM) clinical educators is to develop ways to educate learners effectively at the bedside while keeping up with clinical demands.

The use of just-in-time learning strategies allows students and educators to choose the timing, location and amount of learning needed for each scenario or patient encounter (3). This type of learning typically relies on computer-based instruction or online learning modules, and is limited by the availability and location of a computer (4, 5). As technology has advanced, portable handheld devices are now available that allow an individual the ability to review information at the point of need. These devices are already being used for patient care, and their uses have been described previously (6–9).

One of the several key challenges for medical students during their education is learning how to perform a history and physical exam. While students are universally taught a standard approach to the history and physical exam during their early years in medical school, they need further instruction in how to perform a focused assessment of an undifferentiated, acutely ill patient with a specific chief complaint in the ED. They then need to quickly assimilate the data they have gathered and present the pertinent case information from the patient encounter. This is a challenging but crucial aspect of their education.

At our institution we developed short instructional videos for use by our medical students in the ED, focusing on the approach to the undifferentiated patient. With the arrival of iTunes U at our institution and the distribution of the iPod touch to every student, we decided to use the iPod touch as the platform to educate our students and evaluate its effect. At the time this project began, there was essentially no information in the literature about the effectiveness of the iPod touch as a teaching tool using educational videos. We thus developed a pilot investigation evaluating the effectiveness of just-in-time mobile educational videos using the iPod touch to teach medical students case presentation skills in our busy clinical setting.

Methods

Study design

This was a prospective pilot study designed to evaluate the effect of just-in-time mobile learning videos on medical student case presentations in our ED. Three common chief complaints were used: chest pain, difficulty breathing and abdominal pain. For each of these complaints we created separate five-minute videos that focused on the initial approach to these patients in the ED (Fig. 1). The students used the iPod touch as the platform on which to view the three videos. At the end of the rotation, students were surveyed about their perceptions of the effectiveness of using the videos and mobile technology at the bedside. This project was determined to be exempt from review by the Medical College of Wisconsin Institutional Review Board.

Fig. 1.

Video format for: The initial approach and stabilization of the patient with chest pain, difficulty breathing or abdominal pain.

Setting and study population

The study was conducted from July to October 2009 at Froedtert Hospital Emergency Department, a level one trauma center with over 60,000 annual visits. This ED functions as the main teaching hospital for the Medical College of Wisconsin. A convenience sample of 22 fourth-year medical students rotating in the ED during this period participated in the study. Eight EM faculty served as clinical evaluators.

Study protocol

The study involved gathering data in the clinical setting during real-time patient care. Students were randomized to watch or not watch one of the videos prior to patient encounters involving one of the three chief complaints. They then clinically evaluated the patient and afterwards presented the case to the EM faculty. Patient encounters with chief complaints other than chest pain, difficulty breathing or abdominal pain were not included in the study. The eight faculty who served as evaluators were trained in the study protocol and each evaluated the student's case presentation by completing a structured data collection sheet (Fig. 2). None of the study presentations or surveys was graded or used for the student's final rotation grade, and student participation was voluntary.

Fig. 2.

Clinical data collection sheet.

Randomization

During the rotation orientation at the beginning of each month, students were informed about the study and given the option to participate. Participants were assigned a study number, which was linked to their case presentations. The use of the video was assigned by block randomization. Pre-sealed envelopes that contained the faculty data collection sheet and the video randomization instructions were located in the clinical arena. Students evaluated no more than two patients for each chief complaint as part of the study. Thus each student either watched the video prior to their first patient interaction for each chief complaint and not for the second, or vice versa. Both presentations were evaluated by faculty, who were blinded to the student's use of the video.

Clinical data collection

When a patient with one of the three chief complaints presented to the ED, students retrieved the appropriate pre-sealed envelope. The faculty then gave the student the study iPod touch, and directed the student to a location out of the view of the faculty. If the students were randomized to watch the video they would do this prior to entering the patient's room. After seeing the patient, the students exited the room and returned the iPod touch to the faculty.

Following the student's case presentation, the faculty filled out the data collection sheet. They were instructed to ask only one question at the end of the presentation, which was to have the student list the life-threatening diagnoses of the chief complaint, if not already done. Once the student had finished the presentation and the data collection sheet was completed by the faculty, the faculty could then ask further questions and provide feedback to the student.

The videos

Each of the three videos followed the same format, which was adapted from our rotation textbook (10), and based on common practice in the ED. Each began with a short introduction followed by sections describing what to look for, think about and do before entering the room, while in the room and after leaving the room, as well as instruction on beginning early interventions (Fig. 2). These videos were created by the authors as original pieces of work developed for use by students in the ED, with the goal of helping them approach and present patients to the EM faculty.

Data analysis

Faculty assessments of medical student presentations were analyzed to determine if there was a difference in mean ratings between those using the iPod touch videos and those not using them. All items were scored using a seven-point Likert scale and converted to percentage scores. The differences between the percentage scores of those using the iPod touch and those who did not are reported. Data were further analyzed to determine if there was a difference between first-time video use versus several uses during the rotation, and whether there was a difference for those who participated in the study early versus late in the rotation, as defined by before or after the 15th of the month, respectively.

The data for the student presentations were analyzed comparatively using Mann-Whitney U-tests and independent t-tests. Cohen's d was used as an effect size to indicate the standardized difference between two means. Descriptive statistics for the end of rotation survey are reported as medians/inter-quartile ranges (IR) for Likert-scale items. The internal consistency of data collected for survey items was analyzed with Cronbach alpha. All analyses were generated with SPSS 15.0.

Results

During the four-month study period, 30 fourth-year medical students rotated in the ED and were eligible for participation. Twenty-two students participated in the study by giving clinical presentations during this time, completing a total of 67 case presentations: 29 for abdominal pain, 26 for chest pain and 12 for difficulty breathing. The numbers of students who used and did not use the video for each chief complaint are listed in Table 1. Twenty-two students completed the survey at the end of the rotation.

Table 1.

Use of the video for each chief complaint

	Sample size (N.
Chief complain	Used vide	Did not use vide
Chest pain	12	14
Abdominal pain	13	16
Difficulty breathing	6	6

When data from the three chief complaints were combined, there was a statistically significant difference in presentations when the students used the video for the first time during the rotation (p=.032). When first-time use was reviewed by chief complaint, the scores were higher for each, but the differences were not statistically significant (Table 2). However, some of the effect sizes for the individual chief complaints, such as difficulty breathing (Cohen's d=1.40), were quite high, even if the mean differences were not statistically significant due to small sample sizes. The effect size for the chest pain first-time encounters was also high (Cohen's d=0.96). Forty-eight of the overall encounters were first encounters, and there were 19 second encounters, or students who used a second envelope for a chief complaint.

Table 2.

Faculty subcategory evaluation comparing iPod touch video use or not

		Mean differences (Δ) and Cohen'sd effect sizes of iPod touch video use or not Statistical significance (p)
		Abdominal pai			Chest pai			Difficulty breathin			Overal
Encounter	N	Δ	p	Cohen's d	Δ	p	Cohen's d	Δ	p	Cohen's d	Δ	p	Cohen's d
First encounter	48	–3.7	.552	.29	–10.2	.055	.96	–19.6	.084	1.40	–8.4	.032*	.65
Early versus late encounters	67	–6.2	.242	.50	8.8	.159	.56	–15.9	.083	1.08	–1.6	.671	.11

^*Mean difference is statistically significant of p<.050

Data were also analyzed by comparing the total encounters for those who used versus those who did not use the iPod touch videos in the first half of the rotation (early) versus the second half of the rotation (late) as defined by encounters that occurred before or after the 15th of each month (Table 2).Although the video scores were higher for the first half of the month, there was no statistically significant difference when comparing students who completed study encounters early in the rotation versus late in the rotation. Again, the effect size for the early versus late difficulty breathing encounters (Cohen's d=1.08) was quite high, but the mean difference was not statistically significant due to the smaller sample size.

Cumulative data were analyzed from all 67 presentations done with and without the videos. While the scores were higher when students used the videos, there was no statistically significant difference in presentations for each individual chief complaint. When student scores from all three chief complaints were summed, there were higher scores when students used the videos, but the difference was not statistically significant (p=.226).

Survey

Twenty-two students participated in the post-study survey. Data collected from the survey had a high level of internal consistency (alpha=0.97). Overall, 80% of students preferred to use quick video reviews of topics rather than reading to review a topic prior to seeing a patient; 97% would prefer to use the iPod touch technology instead of their current personal digital assistant (PDA). The survey demonstrated that when using the videos the students felt more confident in their ability to perform and present a focused history, physical exam and patient care plan (Table 3). The students also thought that when using the videos, they gave a better presentation to the faculty. Students considered the videos well done and presented content pertinent to their education and the ED environment (Table 3). Additionally, the students reported that the iPod touch technology was useful as an educational teaching tool (Table 3). Median scores for all items in these three tables were ≥5.0.

Table 3.

End of rotation survey

Survey item	Median (IR)
When using the iPod touch videos, I felt more confident...
Approaching the patient	5.0 (1.0)
Developing a differential diagnosis	5.0 (1.0)
In my presentations	5.5 (1.3)
In my ability to perform a focused history, focused physical exam and specific patient care plan	5.0 (1.2)
In my ability to present a focused history, focused physical exam and specific patient care plan	5.0 (1.2)
Overall, when using the iPod touch prior to seeing the patient I felt that I gave a better patient presentation	5.0 (1.1)
The iPod touch videos...
Were well done and professional appearing	6.0 (1.1)
Presented content pertinent to the ED's clinical setting	6.5 (1.1)
Presented content pertinent to my education	6.5 (1.1)
Were of sufficient length to provide a quick content review before seeing the patient	6.0 (1.1)
I felt the iPod touch...
Was easy to use	6.5 (0.8)
Was useful as a teaching tool	7.0 (0.8)
Would be useful for quick procedural reviews (ex. I&D of an abscess)	7.0 (0.8)
Would be useful for further quick content reviews (ex. headache, neuro exam)	7.0 (1.1)

Responses were scored on a scale from 1 (strongly disagree) to 7 (strongly agree).

Discussion

The potential benefits of handheld technology have been discussed in the literature (7–9, 11–20). To date, there is little evidence demonstrating improved educational outcomes or learning using this technology (6, 21, 22). In our study we set out to determine if using just-in-time videos with the iPod touch improved case presentations for students rotating in our ED. While this was a small pilot study, it is the first that we know of to evaluate educational outcomes using iPod touch and educational ‘just-in-time’ videos as a teaching tool for medical students in the clinical setting.

‘Just-in-time’ learning has been described in the medical literature and used for the education of physicians (23–26). The term refers to learning opportunities that are structured to be delivered exactly when and where the learner needs them (24). The goal of just-in-time learning is to provide an educational resource that is available for the learner's immediate needs, and that gives the necessary information to get a task correct on the first attempt (13). Videos are often the platform used to distribute information, such as on medical journal websites (e.g. www.nejm.org). We chose to deliver our educational information ‘just in time’ in video format with the iPod touch to the students before they entered a patient's room. In this way, they were not tied to a computer and could view the videos anywhere in our ED. We then evaluated the educational impact using this technology and mode of delivery on student case presentations.

During our study we found that the students liked using the videos, and felt they were relevant and of professional quality, and contained information that was pertinent to their education and to the clinical environment of the ED. The majority of students preferred using videos to review information instead of reading it, which was not surprising given that current medical students are of the ‘YouTube generation’ who regularly get information from short video clips online and on handheld devices (27). Instead of being a new concept to them, the students likely felt comfortable with the platform and the fact that the information was distilled into what they needed to know for a specific encounter. Despite this, some students cited a concern that patients would think they are just playing video games or listening to music instead of reviewing educational material if they were seen using the device in the clinical setting. While most institutions have policies regarding use of these handheld devices, their use in the clinical setting is not new and students could explain what they are doing to their patients. We suspect most patients would likely be interested to learn about the students’ educational experiences with this technology. Even though this was the first use of the iPod touch for a number of students, we were not able to determine whether the students liked the device for its novelty to them or its interface and video capabilities. With regard to ease of use, students commented that they felt the iPod touch was easy to use and had the advantage of easy access to the videos during their shifts. Overall, students preferred using the iPod touch over their current PDAs and they recommended its continued use for their education.

The main goal of our pilot study was to determine if the students gave better presentations to our faculty using the just-in-time learning videos prior to patient encounters. We found a statistically significant difference in student presentations between all those who used the videos for the first time versus all those who did not, based on the faculty's impression of the medical student presentations. There was also a high effect size for the chest pain and difficulty breathing videos when used for the first time and a trend towards, yet not achieving, statistical significance, likely due to our sample size (Table 3). The same held true for the early versus late rotation encounters for difficulty breathing, which showed a high effect size and a trend towards statistical significance, but had a small sample size. This early versus late calculation was done to determine if there was a general effect of the timing of using the iPod touch, since students would have experience giving presentations by the second half of the month. However, we did not find such a difference for the early versus late encounters. Given the trend towards statistical significance and the high effect size for these videos used for the first time, we felt this gives promise to our results if we had been able to collect a larger sample size.

Subsequent use of the videos did show an overall improvement in the presentation scores for each chief complaint and overall, but this was not statistically significant. This suggests that initial learning occurred when using the videos for the first time, possibly bringing the students to the same level in what was expected of them in their presentations. This could reduce the benefit of multiple viewings, which is supported by the lack of a statistically significant difference when all encounters were combined. Students may also have benefited from giving repeated presentations outside of the study encounters, which may have improved their presentations throughout their rotation, thus decreasing the benefit of repeated use of the iPod touch and videos. We did not assess the students’ presentations based on any other method of information dissemination, such as a written copy or audio-only version of the information, and our results are based solely on the videos we created.

While this is a small pilot study, we think it demonstrates that using the iPod touch technology for just-in-time educational videos was feasible and was effective in our clinical setting, in addition to being well liked by students as an instructional method. While we do not feel we can make any definitive recommendations based on our results, we have chosen to continue to offer educational videos using the iPod touch as a teaching tool in our department. We think that clinical educators should consider whether, in an instance where live bedside or direct interactive teaching is unavailable, using just-in-time educational videos on a handheld device might be useful as a supplemental instructional strategy. With the rapid advancement of technology in medical education, we encourage medical educators to study the use of just-in-time videos on a handheld device as a clinical teaching modality and to evaluate outcomes in order to develop optimal bedside teaching methods for medical students using this technology.

Limitations

While this pilot study was randomized, it was designed as a convenience sample so as not to interrupt patient care in any way. We thus could not control for how many patients with a given chief complaint a student saw before or after the study cases, or when they saw a study case during the month. While collecting real-time data in a busy clinical setting is important to the design of the study and authenticity of the results, the faculty did not always have time to participate with the student which limited our sample size. It would be ideal to have a trained rater listen to every presentation that a medical student gave while the faculty were blinded to the study, so as to limit any potential bias.

The students we enrolled in the study were fourth-year medical students who already had one year of clinical experience. These students likely benefited from their experiences thus far which may have contributed to the lack of statistical significance for repeated viewings. In addition, we did not obtain a presentation skills baseline at the beginning of the rotation for the students who participated, and some may have already been good at patient case presentations. It would have been ideal to use novice learners, such as third-year medical students, as we suspect we would have likely seen more significant gains in their presentation ability. Unfortunately, we did not have third-year students rotating in the ED during these months.

As a pilot investigation we were pleased to obtain 67 encounters, but this is below the number needed to give appropriate statistical power for the study, as suggested by the improved scores we saw, but lack of statistical significance in certain categories. We did not feel that using one chief complaint video would have produced enough encounters, and we felt that using three common chief complaints would increase the potential encounters and allow us to teach and evaluate concepts given our limited time to collect data. We feel our study is a beginning to what could be a larger, multi-institutional study looking at the benefits of using the iPod touch technology, videos and just-in-time learning in the clinical setting for medical student education.

Conclusion

The use of technology in medical education is rapidly advancing and educators are attempting to keep up. Determining the best utilization is an ongoing and rapidly changing process. The latest technology has allowed just-in-time videos to be viewed on easy-to-use, high-resolution devices, such as the iPod touch. device technology as a platform for our educational videos, we found that our medical students not only preferred the method of dissemination, but also demonstrated an improvement in patient case presentations after watching short just-in-time educational videos for the first time. While the ideal use of the iPod touch or any handheld device in medical education is still evolving, the use of just-in-time video reviews prior to patient encounters should be considered for further study to determine its place in medical education and in the clinical setting.