Visual Abstract
Keywords: clinical nephrology, digital learning, digital videos, education, internship and residency, pencasts
Abstract
Background
In graduate medical education (GME), there are many barriers to achieving a personalized learning process with standardized learning outcomes. One way to support this is through mobile-friendly digital blackboard videos. We sought to measure the effect of a mobile-friendly video curriculum on resident satisfaction, knowledge, and clinical skills during a nephrology rotation.
Methods
This was a prospective, controlled, nonrandomized trial. The control group consisted of internal medicine residents who completed our inpatient nephrology consult rotation as usual. The classroom group had the same clinical experience, but also had access to a library (Nephrology Immersion Classroom) of mobile-friendly, nephrology-themed, digital blackboard videos. In a postrotation assessment, we measured resident satisfaction, clinical knowledge using 15 multiple-choice questions, and nephrology-specific clinical skills.
Results
Of the residents in the classroom group, 77% enrolled in the online classroom, and the majority reported using the classroom occasionally or frequently. A majority found it very easy to use (86%) and strongly recommended having similar videos for other rotations (77%). We observed improved report of rotation-specific clinical skills, but no difference in short-term knowledge between the two study groups.
Conclusions
A mobile-friendly, digital video curriculum for internal medicine residents on an inpatient consult rotation was well utilized, highly rated, and associated with improved nephrology-specific clinical skills. Continued evaluation and incremental improvement of such resources could enhance implementation of GME core curricula.
Introduction
“It’s time to change the way we educate doctors” (1). Historically, scheduled learning activities for internal medicine (IM) residents have consisted of large group lectures held during a noon conference or academic half day. The reliance on lectures has persisted due to the ease of scheduling, programmatic inertia, and faculty unease with implementing new teaching activities. However, varied schedules, competing clinical demands, and postcall duty-hour limits frequently interfere with conference attendance. Even with the provision of food in one example, resident noon conferences were attended, on average, by only 35% of the residents over 2 years (2). Lecture attendees frequently cycle between episodes of attention and inattention (3), disagree with the faculty on the ideal content (4,5), and their learning outcomes may not improve, especially when compared with active learning strategies (6,7). Taken together, resident learning activities should be reformed to better meet trainees’ needs, accommodate their fluid schedules, maintain their attention, and promote knowledge retention.
As medical education reformers have called for standardized learning outcomes along with an individualized learning process (8), digital videos have emerged as a promising and popular learning platform with the current generation of learners (9–11). One style of digital videos is called the digital blackboard video, or pencast. These short, visually stimulating, and dynamic tutorials can help explain difficult concepts, organize existing knowledge, and teach clinical reasoning (9,10). At Duke University School of Medicine, we successfully implemented a digital blackboard curriculum for medical students, for whom the addition of a library of pencast videos to usual course materials was associated with a significant improvement in performance on the physiology exam (12). Therefore, in light of questionable efficacy, inattention, and poor attendance at large group lectures, we sought to create and study the implementation of a mobile-friendly pencast video curriculum for IM residents.
Within IM training programs, residents perceive the care of patients with kidney disease as being too complicated or difficult, and a quarter of those surveyed would have considered nephrology if it had been “taught well” (13). A recent empirical study of patient complexity found that nephrologists do indeed care for the most complex patients, according to a variety of complexity markers (14). Because of the recent and substantial decline in IM residents applying to nephrology fellowships (15), improving the quality of our educational program for IM residents is paramount. Therefore, we sought to understand the feasibility and effect of the Nephrology Immersion Classroom, a mobile-friendly library of digital pencast videos, for IM residents on a nephrology consult rotation.
Materials and Methods
This was a prospective, nonrandomized study that included second- and third-year IM residents completing the nephrology consult rotation at either Duke University Hospital or the Durham Veterans Affairs Medical Center. Residents were paired with a consult team comprised of an attending and fellow. Rotation length varied from 1 to 4 weeks, depending on residency program scheduling needs. During each week on the consult rotation, residents attend weekly educational conferences offered by the nephrology division, including journal club and renal grand rounds.
The control group comprised all IM residents who completed the consult rotation between July 1, 2017 and June 30, 2018. No changes were made to the structure, organization, or composition of the nephrology consult rotation during this time. The classroom group consisted of all IM residents who completed the consult rotation between July 1, 2018 and June 30, 2019. The classroom group differed only in having private access to the Nephrology Immersion Classroom: the mobile-friendly library of 32 blackboard-style, narrated videos and associated review articles. At our institution, the nephrology rotation is not required for all residents, and a majority of residents do not select it as a top preference.
The videos were authored by a single faculty member (J.K.R.) and constructed using digital-art software (Autodesk Sketchbook), screen-capture software (Screencast-o-Matic), USB tablet and pen (Wacom Bamboo Tablet), and a USB microphone (Blue Snowball iCE Condenser Microphone). The content for the entire curriculum was chosen after surveying IM residents (N=5) and nephrology faculty and fellows (N=21), who were asked to prioritize clinical nephrology topics taken from the American Board of Internal Medicine Certification Blueprint (16). This process resulted in a list of core topics chosen for the video curriculum (Table 1). We reorganized the broad core topics into subtopics for a series of 6- to 15-minute videos. We restricted the videos to this length based on the available empirical evidence for maximizing student engagement with online videos (17,18). Each major topic was covered by one to six videos, resulting in a total of 32 videos. For the classroom cohort, all residents were given invitations to join the Nephrology Immersion Classroom on the first day of the rotation. The classroom was hosted using Google Classroom, a learning-management system that hosted private links to all of the videos in a mobile-friendly format with search capabilities. The classroom also included access to high-yield review articles related to some of the core topics. All videos were hosted on YouTube using unlisted links, so the videos were only accessible through the classroom. Table 1 includes links to each video by clicking the video title.
Table 1.
Clinical nephrology topics and corresponding videos in the Nephrology Immersion Classroom
Figure 1 shows a representative screenshot of the Nephrology Immersion Classroom. To access Google Classroom, the participants needed a Google account, and if they did not have one, we asked them to create one for the purposes of the study. Classroom participants were also encouraged to download the free Google Classroom smartphone application. Classroom cohort residents were allowed continued access to Google Classroom for the duration of their residency training.
Figure 1.
Nephrology immersion classroom visualization. Links to the mobile-friendly pencast videos and high-yield review articles are bundled together, arranged by topic, and searchable.
To assess perceptions of the Nephrology Immersion Classroom, residents in the classroom cohort were given a postrotation survey with questions related to the use and quality of the online classroom, including frequency of use, ease of use, when and where it was used, and recommendations for other rotations. We also included a section for comments in response to the following questions “what went well with the online nephrology classroom?” and “what are some things that would make the classroom better meet your needs?”
To assess self-reported skills, we collected responses from the rotation evaluation form administered by the IM residency training program. This evaluation form asks residents to rate their ability (on a five-point Likert scale) to perform certain rotation-specific skills. All residents completing the rotation in both cohorts were invited to complete the rotation evaluation at the direction of the residency training program.
To assess nephrology knowledge, residents in both cohorts were invited to participate in the knowledge pre- and post-test. Residents were invited to take the pretest on the first day of the nephrology rotation and complete the post-test within 1–4 weeks after completing the rotation. Nephrology medical knowledge was assessed using a set of 15 multiple-choice questions (MCQs) in the pretest, and a separate set of 15 MCQs in the post-test. These case-based questions were modified from validated examination questions used in the Kidney Self-Assessment Program. MCQs were chosen and directly linked to core content, as reflected in the video curriculum and the current practice of nephrology. A $5 Starbucks gift card was given to the participant as compensation for each completed survey. The gift card incentive was implemented after the first academic quarter of the control year, due to initially low response rates. Assuming an α of 0.05 and SD of 20, to detect a mean difference of 15% on the knowledge assessment with 80% power, we anticipated a sample size of 28 residents would be needed in each group. In addition, we collected and examined performance on nephrology questions from the IM In-Training Examination, a formative knowledge assessment completed by IM residents. For categorical residents, the exam was taken in the second year of training. For combined program residents (medicine-pediatrics and medicine-psychiatry residents), we used scores taken from the third year of training. We also examined responses related to acquisition of rotation-specific skills based on the anonymous rotation evaluation administered and collected by the IM training program. This study was reviewed and approved by the Duke University Institutional Review Board (project number 00087609).
We conducted all quantitative analyses in STATA version 14.0 (Statacorp, College Station, TX). Normally distributed data are reported as mean±SD; non-normally distributed data are reported as median and interquartile range. The analysis of knowledge assessment was restricted to residents who completed both pre- and post-tests. For these analyses, we used paired t tests to compare the means within groups, and unpaired t tests to compare means between the groups. Responses on the rotation evaluation regarding self-reported skills were anonymous, and we used unpaired t tests to compare the mean score between the groups for each item. Two-tailed P values <5% were considered statistically significant.
Results
During the control year, 39 of 80 eligible IM residents completed the nephrology consult rotation. All received invitations to take pre- and post-tests and 13 completed both (33% response rate). The majority of pretests were completed on the first day of the rotation, and post-tests were completed a median of 26 days (range, 1–117 days) after the end of the rotation. During the classroom year, 41 of 86 eligible IM residents were assigned to the nephrology consult service. Two residents were excluded from the study because they also participated in the control group. Out of the final cohort of 39 classroom residents, 24 (61% response rate) completed both pre- and post-tests. The majority of pretests were completed on the first day of the rotation, and post-tests were completed a median of 5 days (range, 1–84 days) after the end of rotation. A greater proportion of respondents in the classroom cohort were postgraduate year 2 compared with the control year (64% versus 54%, P=0.42). In terms of when participating residents completed the nephrology rotation, survey respondents were equally distributed throughout the academic calendar, with the exception of the first quarter of the control year, which had fewer respondents compared with the other academic quarters.
Knowledge Assessment
Table 2 shows performance on the pre- and postrotation knowledge assessments for residents who successfully completed both tests (33% response rate in the control cohort; 61% response rate in the classroom cohort). In both cohorts, we observed significantly higher exam scores on post-tests compared with pretests, but no significant differences in the mean post-test exam scores or change in scores between the control and classroom groups. The results were no different when restricting the analysis to residents in the classroom cohort who enrolled in the Google Classroom, or to those who reported more than occasional use of the classroom (data not shown).
Table 2.
Performance on pre and postrotation knowledge assessments
| Group | Control Group | Classroom Group | ||
| Pretest | Post-test | Pretest | Post-test | |
| N | 13 | 24 | ||
| Median time to complete test (min) | 30 | 33 | 38 | 40 |
| Mean test score, % correct (SD) | 57 (10) | 68 (15) | 59 (11) | 64 (16) |
| P value (within group) | 0.04 | 0.05 | ||
| P value (between group) | 0.45 | |||
Table 3 shows performance for both groups on nephrology questions from the IM In-Training Examination. Mean scores (percent correct) were similar between the control group and the classroom group. We also calculated mean scores among classroom group residents who reported using the digital video curriculum (any resident who reported using it occasionally, frequently, or very frequently, including one resident who did not complete a post-test survey, but notified the authors that they used the curriculum very frequently). In this restricted analysis, classroom users had higher mean scores on the In-Training Examination nephrology questions compared with the control group. This difference, however, was not statistically significant (P<0.26).
Table 3.
Resident performance for nephrology questions on the IM In-Training Examination
| Group | Control Group | Classroom Group (All Residents) | Classroom Group (Residents Who Used the Classroom) |
| N | 33 | 39 | 21 |
| Nephrology item score, % correct (SD) | 78 (13) | 78 (11) | 81 (10) |
Nephrology-Specific Clinical Skills
Using the end-of-rotation evaluation administered by the IM training program, we compared resident self-assessment of their ability to perform particular skills on a five-point scale (1, never; 2, infrequently; 3, sometimes; 4, most of the time; 5, always). Table 4 shows each rotation-specific skill and mean (SD) response for each cohort of residents. On average, residents in the classroom group (97% response rate) reported higher levels of readiness with all of the skills compared with residents from the control group (74% response rate). The difference in the following two skills were statistically greater in the classroom group: prioritizing a workup for acute kidney failure and recognizing the indications for a kidney biopsy.
Table 4.
Resident self-assessment as determined by ability to perform rotation-specific skill
| Rotation-Specific Skills | Mean Ability To Perform Skill (SD) | Comparison (P Value) | |
| Control Cohort (N=29) | Classroom Cohort (N=38) | ||
| I can generate a differential diagnosis for ARF | 4.5 (0.6) | 4.7 (0.5) | 0.07 |
| I can prioritize a workup for ARF | 4.5 (0.6) | 4.7 (0.4) | 0.04 |
| I can explain basic principles and indications of the three types of RRT: intermittent hemodialysis, continuous hemodialysis, and peritoneal dialysis | 4.3 (0.7) | 4.4 (0.5) | 0.58 |
| I recognize the indications for renal biopsy | 3.6 (1.0) | 4.1 (0.7) | 0.02 |
| I will be able to apply principles learned on this rotation to the care of patients with CKD in my outpatient practice | 4.3 (0.8) | 4.5 (0.6) | 0.38 |
| I can explain the causes and workup of disorders of sodium balance to an intern on my general medicine service | 4.3 (0.7) | 4.4 (0.5) | 0.41 |
| I can explain the causes and workup of disorders of water balance to an intern on my general medicine service | 4.3 (0.7) | 4.3 (0.5) | 0.54 |
| I can interpret a urinalysis and urine electrolyte studies | 4.4 (0.6) | 4.5 (0.5) | 0.46 |
The assessment was based on a five-point scale (1, never; 2, infrequently; 3, sometimes; 4, most of the time; 5, always).
Feasibility of the Nephrology Immersion Classroom
A total of 30 residents (77%) electively joined the online classroom, and 66% completed the postrotation survey. Table 5 shows usability metrics related to the web-based Nephrology Immersion Classroom. The majority of respondents reported using the classroom occasionally or frequently. Classroom users watched videos mostly at home or at work during down time. A majority of users (86%) found the classroom very easy to use, and most users (77%) recommended having video tutorials available for other clinical rotations. Table 6 lists representative comments found on the postrotation survey written in response to prompts about what went well and could have been improved regarding the nephrology classroom and rotation.
Table 5.
Usability and perceptions of the Nephrology Immersion Classroom
| Question | No. Responding per Category, n (%) | |||||||||||||||
| If you used the nephrology Google Classroom, how often did you use it? | Not at all | Rarely | Occasionally | Frequently | Very frequently | |||||||||||
| 5 (19) | 1 (4) | 14 (54) | 5 (19) | 1 (4) | ||||||||||||
| If you used the nephrology Google Classroom, when/where did you use it? (Mark all that apply) | At home | At work at the point of care | At work during down time | |||||||||||||
| 15 (71) | 1 (5) | 10 (48) | ||||||||||||||
| If you used the nephrology Google Classroom, how easy was it to use? | Very difficult | Moderately difficult | Slightly difficult | Neither easy nor difficult | Slightly easy | Moderately easy | Very easy | |||||||||
| 0 | 0 | 0 | 0 | 1 (5) | 2 (10) | 18 (86) | ||||||||||
| Would you recommend having video tutorials for other rotations? | Do not recommend | Slightly recommend | Moderately recommend | Strongly recommend | ||||||||||||
| 0 | 1 (5) | 4 (18) | 17 (77) | |||||||||||||
Table 6.
Narrative comments about the Nephrology Immersion Classroom
| Responses to “What Went Well with the Online Nephrology Classroom?” | Responses to “What Are Some Things that Would Make the Classroom Better Meet Your Needs?” |
| “The provided video curriculum was great.” | “N/A It was excellent!” |
| “Due to busy service, it was valuable to have online curriculum as supplemental education.” | “An associated question bank with a couple of questions to check your learning afterward could be helpful for solidifying concepts.” |
| “Just wanted to say thanks for putting that online curriculum for the nephrology rotation. It is great to have as a resource for this rotation and for residency going forward.” | “Sometimes the 15-minute video is too long and being able to skim a sheet would be helpful. That being said what was provided was wonderful when time allowed.” |
| “I just want to let you know I’ve been watching and rewatching your Nephrology Google Classroom videos and they are fantastic. Thank you so much for putting the time in to make these, I find them extremely helpful and wish we had something like this for every rotation.” | “Shorter clinical review articles that could be more point of care consumable. Perhaps diagnostic algorithm charts/diagrams could be included.” |
| “Thank you so much for this, just finished watching the videos and they were great. I’ve been trying to make a point of recent [sic] of going back and reviewing the basic biology and physiology we learned in med school so as to integrate this knowledge more intimately into my clinical practice, so this was really helpful. Thanks again for all your work and time here. It is much appreciated.” | “It would be great if the Classroom videos were accompanied by a transcript. Sometimes it’s difficult to watch a 20-minute video, but much faster and equally educational to read a transcript.” |
| “I really appreciated the teaching from fellows and attendings and felt like I learned a lot in the 2-week period. I appreciated having access to the online curriculum as well.” | “I think it is great. I just haven’t had a chance to go through the videos yet. Planning on it though! Love that we have access to them!” |
Discussion
In this study, we examined the usability and effects of an online video curriculum on IM residents’ experience during an inpatient nephrology consult rotation. Overall, enthusiasm and interest in the Nephrology Immersion Classroom was high. Despite the short rotation length and elective nature of our curriculum, the majority of IM residents enrolled in the online classroom. These residents reported modest to heavy use of the video curriculum, typically at home or at work during down time. Although we observed no significant improvement on MCQs from pre- to postrotation, the video curriculum was rated as very easy to use, and the majority of users strongly recommended having similar resources for other clinical rotations. Despite having a greater proportion of junior (postgraduate year 2) residents in the classroom cohort, we observed higher self-reported readiness to perform nephrology-specific skills, namely prioritizing a workup for ARF and recognizing the indications for a kidney biopsy. Residents in the classroom cohort reported high levels of satisfaction with the video curriculum and it appeared to have improved self-reported clinical readiness.
In contrast, reported use of the classroom was variable and our residents identified some opportunities for improvement. In the narrative comments, residents made statements suggesting there was not enough time to watch some of the longer videos (two were >15 minutes). Also, residents commented on the need for multiple modalities. For example, one resident recommended a transcript of the video, whereas others recommended a single summary sheet with high-yield algorithms or flowcharts. These comments suggest that additions like these may contribute to greater learning at the point of patient care. Another proposed opportunity for improving the classroom is the addition of knowledge self-assessments, which could help the learner check knowledge retention after watching some of the videos.
Historically, scheduled learning activities in graduate medical education (GME) have relied heavily on the delivery of large-group, didactic lectures. With evolving medical education research and concurrent advances in technology, GME programs are now embracing learning activities that better personalize the learning process, foster self-directed learning behaviors, and reserve the classroom for active learning. In this paradigm, the Nephrology Immersion Classroom represents successful implementation of a mobile-friendly video platform for self-directed learning while on a clinical rotation. The advantage of this is multifold. First, the current generation of learners are comfortable with using digital learning tools in place of traditional lectures (9–11). Second, short concept videos have been empirically associated with high levels of engagement, and mobile-friendly videos allow high level of user control (i.e., location, time, and viewing speed) (17,18). Third, access to a library of short concept videos allows customization of the learning process: users can self-select topics based on perceived need, interest, or relevance to their patients’ problems. Large-group, lecture-based curricula suffer from a “one-size-fits-all” approach and “just-in-case” learning, whereas a pencast video library facilitates a personalized approach with “just-in-time” learning.
Although the residents in our study reported high levels of satisfaction and modest use of the pencast curriculum, questions remain about how such a curriculum should best be implemented. The purpose of this study was to measure satisfaction and short-term knowledge from simply having elective access to a library of clinically focused pencast videos after starting an inpatient consult rotation. We observed no difference in short-term clinical nephrology knowledge in our classroom cohort compared with a control cohort of residents. Given the breadth and depth of the curriculum (along with competing clinical demands), GME trainees likely need more time to fully engage with such a curriculum. If implemented longitudinally, however, it is unknown whether a pencast curriculum would enhance long-term knowledge outcomes over the duration of GME training. Also, it matters how such a curriculum is structured within global GME learning activities. Some GME programs have reported success using short videos to help achieve procedural readiness (19,20), and others have successfully implemented flipped classroom activities using online videos (21,22). In the context of flipped classroom learning activities, a video pencast curriculum could easily serve as the preparatory material to be viewed before case-based active learning sessions. The effects of flipped classroom teaching on learning outcomes in GME has been mixed, despite high levels of resident satisfaction (21–24). Recent work by Graham et al. (25) describes successful implementation and improved long-term knowledge outcomes with a flipped classroom for IM residents on an ambulatory rotation. In an accompanying letter, Wortzman and Saddawi-Konefka (26) suggest that the success of this flipped classroom was due to IM residents having dedicated time in their weekly schedule to do the preparatory work, because prior studies in GME showed poor compliance with the prework (27,28). When surveyed, 66% of IM residents thought that the flipped classroom is not feasible on inpatient rotations, mainly due to the absence of protected time (25). Therefore, assigning pencast videos as preparatory work for flipped classroom activities probably works best during ambulatory rotations, where provision of study time is feasible. In summary, the Nephrology Immersion Classroom serves as a potential prototype for GME learning in the 21st century: it could serve the preparatory modules for flipped classroom experiences or as a structured repository of short, mobile-friendly videos for self-directed, practice-based learning and improvement.
Our study has a number of limitations. First, self-reporting in the postrotation survey is subject to recall bias. Second, we observed incomplete response rates for both pre- and post-test assessments in both cohorts, and the knowledge outcome analysis was restricted to paired samples. Despite an incentive to boost survey completion, these low response rates underpowered the analysis and could have also introduced a nonresponse bias to the other survey items. It is possible that digital videos improved resident knowledge, but the 15-item examination was unable to detect a difference between the two groups. Although major elements of the nephrology rotation were unchanged between the two rotation years (other than addition of the classroom), we were unable to control for potential residual confounding given the nonrandomized nature of the study. There is also at least one resident who reportedly watched all pencast videos, but never submitted the post-test knowledge assessment and survey, further contributing to nonresponse bias.
Hosting the Nephrology Immersion Classroom, a mobile-friendly pencast video curriculum, for IM residents on an inpatient rotation was associated with high levels of resident satisfaction and self-report clinical skill development. Adding this video curriculum did not improve short-term medical knowledge, as measured by case-based MCQs, when compared with the usual nephrology consult rotation. The classroom was rated as very easy to use and the majority of residents strongly recommend having similar video tutorials for other rotations. To improve the classroom, we will add images of diagnostic algorithms, flowcharts, and embedded quizzes for self-assessment. We found that a mobile-friendly video curriculum is well utilized, highly rated, and open to broad application elsewhere in GME curriculum.
Disclosures
All authors have nothing to disclose.
Funding
This study was supported by the American Society of Nephrology William and Sandra Bennett Clinical Scholar Award.
Acknowledgments
This research was previously presented in poster form at the American Society of Nephrology Kidney Week 2019 Scientific Session, Washington, DC.
Author Contributions
J. Roberts was responsible for resources, software, validation, and visualization; J. Roberts and M. Wolf wrote the original draft, provided supervision, and were responsible for data curation, formal analysis, funding acquisition, investigation, and methodology; and all authors conceptualized the study, were responsible for project administration, and reviewed and edited the manuscript.
References
- 1.Prober CG, Heath C: Lecture halls without lectures--a proposal for medical education. N Engl J Med 366: 1657–1659, 2012 [DOI] [PubMed] [Google Scholar]
- 2.Cosimini MJ, Mackintosh L, Chang TP: Number needed to eat: Pizza and resident conference attendance. Med Educ 50: 1204–1207, 2016 [DOI] [PubMed] [Google Scholar]
- 3.Bunce DM, Flens EA, Neiles KY: How long can students pay attention in class? A study of student attention decline using clickers. J Chem Educ 87: 1438–1443, 2010 [Google Scholar]
- 4.Sawatsky AP, Zickmund SL, Berlacher K, Lesky D, Granieri R: Understanding resident learning preferences within an internal medicine noon conference lecture series: A qualitative study. J Grad Med Educ 6: 32–38, 2014 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Sawatsky AP, Zickmund SL, Berlacher K, Lesky D, Granieri R: Understanding the challenges to facilitating active learning in the resident conferences: A qualitative study of internal medicine faculty and resident perspectives. Med Educ Online 20: 27289, 2015 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Meyer NB, Gaetke-Udager K, Shampain KL, Spencer A, Cohan RH, Davenport MS: (Lack of) measurable clinical or knowledge gains from resident participation in noon conference. Acad Radiol 25: 719–726, 2018 [DOI] [PubMed] [Google Scholar]
- 7.Freeman S, Eddy SL, McDonough M, Smith MK, Okoroafor N, Jordt H, Wenderoth MP: Active learning increases student performance in science, engineering, and mathematics. Proc Natl Acad Sci U S A 111: 8410–8415, 2014 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Irby DM, Cooke M, O’Brien BC: Calls for reform of medical education by the carnegie foundation for the advancement of teaching: 1910 and 2010. Acad Med 85: 220–227, 2010 [DOI] [PubMed] [Google Scholar]
- 9.Prober CG, Khan S: Medical education reimagined: A call to action. Acad Med 88: 1407–1410, 2013 [DOI] [PubMed] [Google Scholar]
- 10.Adam M, Chen SF, Amieva M, Deitz J, Jang H, Porwal A, Prober C: The use of short, animated, patient-centered springboard videos to underscore the clinical relevance of preclinical medical student education. Acad Med 92: 961–965, 2017 [DOI] [PubMed] [Google Scholar]
- 11.Doherty I, Sharma N, Harbutt D: Contemporary and future eLearning trends in medical education. Med Teach 37: 1–3, 2015 [DOI] [PubMed] [Google Scholar]
- 12.Roberts JK, Chudgar SM, Engle D, McClain EK, Jakoi E, Berkoben M, Lehrich RW: Digital chalk-talk videos improve knowledge and satisfaction in renal physiology. Adv Physiol Educ 42: 146–151, 2018 [DOI] [PubMed] [Google Scholar]
- 13.Jhaveri KD, Sparks MA, Shah HH, Khan S, Chawla A, Desai T, Iglesia E, Ferris M, Parker MG, Kohan DE: Why not nephrology? A survey of US internal medicine subspecialty fellows. Am J Kidney Dis 61: 540–546, 2013 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Tonelli M, Wiebe N, Manns BJ, Klarenbach SW, James MT, Ravani P, Pannu N, Himmelfarb J, Hemmelgarn BR: Comparison of the complexity of patients seen by different medical subspecialists in a universal health care system. JAMA Netw Open 1: e184852, 2018 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.National Residency Matching Program: Charting outcomes in the match: Specialties matching service, appointment year 2018. Available at: https://mk0nrmp3oyqui6wqfm.kinstacdn.com/wp-content/uploads/2018/10/2018-Charting-Outcomes-SMS.pdf. Accessed November 18, 2019
- 16.American Board of Internal Medicine: American Board of Internal Medicine Certification Blueprint. Available at: https://www.abim.org/%7E/media/ABIM%20Public/Files/pdf/exam-blueprints/certification/internal-medicine.pdf. Accessed July, 2017
- 17.Rana J, Besche H, Cockrill B: Twelve tips for the production of digital chalk-talk videos. Med Teach 39: 653–659, 2017 [DOI] [PubMed] [Google Scholar]
- 18.Guo PJ, Kim J, Rubin R: How video production affects student engagement: An empirical study of MOOC videos. Presented at the First ACM Conference on Learning at Scale, Atlanta, GA, March 4–5, 2014 [Google Scholar]
- 19.Mishra K, Mathai M, Della Rocca RC, Reddy HS: Improving resident performance in oculoplastic surgery: A new curriculum using surgical wet laboratory videos. J Surg Educ 74: 837–842, 2017 [DOI] [PubMed] [Google Scholar]
- 20.Kassam AF, Cortez AR, Diwan TS, Bailey A, Mirza A, Sussman JJ, Makley AT, Quillin RC 3rd: Return to the real world: A novel program to improve resident readiness for the transplant surgery rotation. J Surg Educ 77: 242–245, 2020 [DOI] [PubMed] [Google Scholar]
- 21.Rose E, Claudius I, Tabatabai R, Kearl L, Behar S, Jhun P: The flipped classroom in emergency medicine using online videos with interpolated questions. J Emerg Med 51: 284–291.e1, 2016 [DOI] [PubMed] [Google Scholar]
- 22.Blair RA, Caton JB, Hamnvik OR: A flipped classroom in graduate medical education. Clin Teach 17: 195–199, 2020 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Riddell J, Jhun P, Fung CC, Comes J, Sawtelle S, Tabatabai R, Joseph D, Shoenberger J, Chen E, Fee C, Swadron SP: Does the flipped classroom improve learning in graduate medical education? J Grad Med Educ 9: 491–496, 2017 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Allenbaugh J, Spagnoletti C, Berlacher K: Effects of a flipped classroom curriculum on inpatient cardiology resident education. J Grad Med Educ 11: 196–201, 2019 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Graham KL, Cohen A, Reynolds EE, Huang GC: Effect of a flipped classroom on knowledge acquisition and retention in an internal medicine residency program. J Grad Med Educ 11: 92–97, 2019 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Wortzman JR, Saddawi-Konefka D: Is providing a “study hall” the key to flipped classroom effectiveness? J Grad Med Educ 11: 355, 2019 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Cooper AZ, Hsieh G, Kiss JE, Huang GC: Flipping out: Does the flipped classroom learning model work for GME? J Grad Med Educ 9: 392–393, 2017 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Heitz C, Prusakowski M, Willis G, Franck C: Does the concept of the “flipped classroom” extend to the emergency medicine clinical clerkship? West J Emerg Med 16: 851–855, 2015 [DOI] [PMC free article] [PubMed] [Google Scholar]