Abstract
Objective. To describe what and how infectious diseases (ID) topics are taught in US schools of pharmacy and summarize pharmacy faculty members’ and students’ perceived successes and challenges in teaching and learning about ID.
Methods. A 23-item survey instrument was distributed electronically to ID faculty members at 137 US pharmacy schools. Data collected included curricular hours and format, topics covered, active-learning strategies, and curricular successes and concerns.
Results. Surveys were collected from 106 schools (77% response rate). Infectious diseases curricula were allotted a median of 60 (IQR=40) hours of classroom time. Respondents dedicated 33% of curriculum hours to ID fundamentals and 66% to disease states. Greater than 94% of schools taught all tier one ID topics from the 2016 American College of Clinical Pharmacy Pharmacotherapy Didactic Curriculum Toolkit. Curricula were primarily delivered through traditional lectures rather than active learning (75% vs 25% of classroom time, respectively). The median number of active-learning strategies used was four (IQR=3). The most common active-learning modalities used either consistently or frequently were patient case application (98%) and audience response systems (76%). The most common successes cited by faculty members were implementation of active learning, the “real-world” applicability of the ID topics, and the breadth of topics and topic exposure covered in the curriculum. The most common concerns were a lack of time to cover material and the amount of material covered.
Conclusion. Increased communication and collaboration between ID educators is warranted to increase consistency of ID education and distribution of educational innovations.
Keywords: education, curriculum, teaching, infectious diseases
INTRODUCTION
Inappropriate use of antimicrobials and antimicrobial resistance is a global threat.1 As pharmacists are the gatekeepers of prescription antimicrobials, all pharmacy students should receive enough infectious diseases (ID) education to be practical antimicrobial stewards.2 While most attention about antimicrobial stewardship has focused on the inpatient setting because of the recent Centers for Medicare and Medicaid Services (CMS) antimicrobial stewardship standard,3 pharmacists have the potential to be antimicrobial stewards in a variety of practice settings.4,5 Because of the shortage of postgraduate year two (PGY2) pharmacy residency programs in ID available,6 the primary source of ID knowledge will occur within the Doctor of Pharmacy (PharmD) program.
Education in pharmacotherapy for general IDs is required in the PharmD curriculum by the Accreditation Council for Pharmacy Education (ACPE),7 yet there is no specific criteria on the topics that must be covered or the educational methods that should be used. The 2016 American College of Clinical Pharmacy (ACCP) Pharmacotherapy Didactic Curriculum Toolkit provides a list of ID topics recommended to be included in the PharmD curriculum.8 The Infectious Disease Society of America (IDSA) Preclinical Curriculum Committee recommends the use of active learning to stimulate student participation, collaborative learning, and communication with the instructor.9
The objective of this study was to describe the topics delivered and the pedagogical strategies used in the teaching of ID curricula to PharmD students in US pharmacy schools. We also aimed to summarize the perceived successes and challenges of faculty members and students in the teaching and learning of ID, respectively.
METHODS
This was a cross-sectional, multicenter observational study of didactic ID curricula at US pharmacy schools. We took a concurrent mixed methods approach that used a qualitative in quantitative embedded design.10 This study was reviewed and approved by the Colorado Multiple Institutional Review Board at the University of Colorado Anschutz Medical Campus.
A 23-question survey instrument was inspired by the survey by Bonura and colleagues who developed a survey to identify factors associated with medical resident career choice.11 The authors further developed the survey and then pilot-tested it with the Infectious Diseases Educator Network (ID-EN) steering committee. A draft survey instrument was sent to 17 ID faculty experts from various pharmacy schools. Faculty members completed the survey and provided feedback about the accuracy of survey logic and question clarity. Faculty responses and feedback were used to improve question reliability and validity. Feedback from the committee was incorporated into the final version of the survey.
The final survey included quantitative and qualitative sections. This approach was chosen to illustrate quantitative findings with qualitative themes. The quantitative section inquired about curricular format, delivery of ID within the curriculum, number of hours dedicated to ID content, and frequencies and methods of active-learning employed. The 2016 American College of Clinical Pharmacy (ACCP) Pharmacotherapy Didactic Curriculum Toolkit was used to assess ID topics covered.8 The purpose of the Toolkit is to provide guidance on the incorporation of various pharmacotherapy topics to deliver within the pharmacy school curriculum. The Toolkit uses a three-tiered approach to recommend specific topics based on the level of importance for incorporation into the pharmacy school curriculum. Tier one topics include those that pharmacy schools should provide education and training on to the degree that students are practice ready upon graduation. Tier two topics should be taught, but additional knowledge or skills may be required after graduation (eg, residency training) prior to being practice ready. Tier three topics may not be covered in pharmacy school curriculum, and students are expected to obtain knowledge and skills on their own prior to being practice ready. The qualitative section of the survey asked respondents open-ended questions about successes, concerns, and changes they would like to see in in the delivery of ID topics. The full survey instrument is available upon request.
In September 2017, the survey instrument, which was created using Qualtrics (Qualtrics, Inc., Provo, UT) survey software, was electronically distributed to ID course directors and faculty members at 137 US pharmacy schools with ACPE accreditation or candidate status. Those PharmD programs with pre-candidate status (n=4) were excluded as faculty recruitment and curriculum development may have still been in progress. Contact information for ID faculty members was obtained via an internet search of each pharmacy school. The survey was distributed to the department chair or curriculum dean when an ID faculty member was not identifiable. Up to two follow-up emails were sent, and one telephone call was made to nonresponders during the 12 weeks in which the survey was open. The survey was completed by a single individual per school, except when a single school had two campuses that used different ID curriculum content and/or academic schedule.
All quantitative data were analyzed using SPSS Statistics, version 24 (IBM, Armonk, NY). Frequencies and percentages were used to summarize categorical variables. Descriptive statistics were used to summarize continuous variables. Medians and interquartile ranges were used to describe continuous variables. The Mann-Whitney test was used to compare demographic data and continuous variables. A Fisher exact test was used to compare demographic data and categorical variables. For all tests of significance, a two-tailed alpha value was set at .05. Free-text responses were analyzed using Dedoose (SocioCultural Research Consultants, Manhattan Beach, CA). Themes were extracted using a general inductive approach of open coding based on our research objectives.10,13 One author developed a coding scheme based on 33% of the responses in the sample. The author discussed these codes with two of the other authors to determine face validity. Once consensus was achieved, the authors blindly applied the same codes to the initial sample, and results were compared to maintain consistency. When new themes emerged, they were discussed between the three researchers and the responses were re-coded. Once inter-rater reliability was established, two of the authors independently coded the remaining 67% of the survey responses. The final codes were interpreted with our research questions to generate salient themes.
RESULTS
The completed survey response rate was 77%. The 137 completed survey instruments represented 106 schools. The survey was completed by ID faculty for 95 (90%) schools and completed by department chairs or curricular deans for 11 (10%) schools. Of the responding schools, 97 (92%) were accredited by ACPE, 9 (8%) were in candidate status, 56 (53%) were private institutions, 59 (56%) were greater than 20 years old, and 95 (90%) were four-year programs. Most schools (69, 65%) reported having a semester format. Class sizes were less than 50 students at two (2%) schools, 50 to 100 students at 52 (49%) schools, 101-200 at 41 (39%) schools, and greater than 200 students per class at 11 (10%) schools.
A median of 60 hours (IQR=40-80) of classroom time was allotted for teaching ID curricula. The percent of time allocated to teaching antimicrobial fundamentals was 33% (IQR=25-40) and 67% (IQR=60-75) for disease states. Active learning was employed during 25% (IQR=20%-42.5%) of classroom time, with lectures used for the remaining 75% (IQR=57.5%-80%). The placement of ID content in the curriculum schedule varied, but was primarily taught in the third year of the curriculum in four-year programs and in the second year in three-year programs (Table 1).
Table 1.
Greater than 94% of schools taught all tier one ID topics from the ACCP Pharmacotherapy Didactic Curriculum Toolkit (Table 2). The percentage of schools teaching tier two topics in required curricula varied by topic from 99% for human immunodeficiency virus infection to 33% for spirochetal diseases. Tier three topics were uncommonly covered, with the most commonly covered tier three topic being non-tuberculous mycobacterial infections (36% of schools). The most commonly covered topics in ID electives were the tier two topics of antimicrobial stewardship (14%), travel medicine (13%), and prevention of healthcare-acquired infections (10%). The topics that were most commonly allotted greater than two didactic classroom hours included human immunodeficiency virus infection (84%), antimicrobial regimen selection (72%), and immunization (64%).
Table 2.
All but two schools used at least one active-learning method either consistently or frequently. The average number of active-learning strategies used was four (IQR=3). The most common active-learning modalities reported as used consistently or frequently were patient case applications (98%) and audience response systems (76%) (Table 3).
Table 3.
Classroom hours allotted to the ID curriculum, percentage of classroom hours dedicated to antimicrobial fundamentals vs diseases states, and percentage of classroom hours devoted to lecture vs active-learning differed based on school characteristics (Table 4). Public schools dedicated more time to antimicrobial fundamentals than private schools (40 vs 30 hours, p=.02). Schools using a blocked curricular schedule allotted more classroom hours to ID content than schools using semester or quarter schedules (77.5 vs 60 vs 50 hours, p=.001). Schools established less than five years ago devoted more classroom hours to active learning than schools established five to 10, 11 to 20, and greater than 20 years ago (65 vs 25 vs 25 vs 20 hours, p=.01). Private schools were more likely to consistently or frequently use audience response systems (86% vs 68%, p=.04) and concept maps (27% vs 6%, p<.01) compared to public schools. Public schools were more likely to use debates (18% vs 2%, p<.01. Schools established 10 or less years ago were more likely to use concept maps (28% vs 12%, p=.05), games (38% vs 19%, p=.05), and simulations (41% vs 18%, p=.02). No significant differences in consistent or frequent use of an active-learning method occurred based on class size or curricular course format (semester vs quarter vs block).
Table 4.
As for qualitative findings, the coders observed that faculty members provided similar responses on items asking for their point of view and then what they believed students’ point of view on the topic was. Based on these results, the coders determined that eliciting student opinions on these topics would result in richer data compared to faculty perceptions of student opinions. Therefore, only responses to items that asked about the faculty’s point of view were coded. The coders identified 60 codes in total, comprised of seven parent codes and 53 child codes. Three major themes emerged from the codes, one theme for each open-ended question.
“Structure” emerged from descriptions about changing the way ID is taught. Structure refers to how sessions were organized, who taught the content, instructional methods, placement of content in the curriculum, and time allotted. Structure also described how and when ID content was taught in relation to other disciplines. For example, one respondent stated: “ID is such a broad topic. Very little real estate is allocated to stewardship. I think a good amount of the course material could be pushed into gen med and allow ID faculty to focus on ASP and advanced ID issues a little more.” Taken altogether, respondents mentioned several different structural changes they would like to make to their ID curricula.
“Comprehensive” was the major theme derived from responses about ID curricular achievements. Comprehensive describes the number of ID topics taught, instructional methods (eg, real-world application and active learning), and the expertise of the instructors. One respondent described the “added depth provided by a content expert (ie, teacher),” exemplified through application-based instruction.
Academic burden was the major theme identified in faculty concerns about their ID curriculum. Academic burden refers to the amount of time or amount of material taught within the ID curriculum. While some respondents identified the breadth of topics covered by their curriculum as a successful component, others indicated that exposure to more topics led to increased student stress.
DISCUSSION
Given the limited number of ID postgraduate training opportunities and the increased role of pharmacists in antimicrobial stewardship programs, most pharmacy graduates will contribute to antimicrobial stewardship, relying largely on the ID knowledge gained during pharmacy school.6,14,15 Thus, ID instruction within the PharmD curriculum needs attention. Based on the findings of this study, a typical US pharmacy school dedicates 40 to 80 classroom hours to delivering their ID curriculum, which is taught primarily through traditional lectures and delivered in the third professional year. A 2015 survey of 75 US pharmacy schools reported that ID had the highest number of contact hours (median=36, IQR=29-60) followed by cardiology (median=32, IQR=22.5-48) and hematology/oncology (median=22.5, IQR=12-32).16 The difference in reported classroom hours could be because of the lower response rate, timing of the survey, and survey completion performed by curriculum committee members.
Nearly all schools taught every ACCP Pharmacotherapy Didactic Curriculum Toolkit tier one topic and most of the tier two topics. Our qualitative analysis determined that comprehensive ID curricula are perceived as successful by faculty members. Surprisingly, antimicrobial stewardship is only a tier two topic, and was included in the curriculum of 75% of schools. Sixteen other tier two topics were more commonly taught, including diseases which the majority of pharmacists will rarely encounter (infections of immunocompromised hosts, tuberculosis, HIV). Kufel and colleagues found only 68% of pharmacy schools included antimicrobial stewardship in the didactic curriculum.17 They also found that schools that did not employ ID specialty pharmacists were less likely to offer experiential education opportunities that included antimicrobial stewardship principles. ID educators must continue to communicate with pharmacy organizations making recommendations about in curricula that antimicrobial stewardship is not a specialty, but instead a skillset in which all pharmacists should be proficient.
Variability in the ID topics covered and instruction methods delivered within the curriculum may contribute to ineffective learning of essential concepts. Both pharmacy and medical students’ knowledge and perceptions of antimicrobial agents suggest that deficiencies exist in pharmacy and medical schools’ curricula.18,19 A European study of 7,328 graduating medical students reported that 66% of the students wanted more education on the prudent use of antimicrobials, and students from countries with higher rates of antimicrobial resistance reported feeling less prepared to appropriately prescribe antimicrobials.20 Abbo and colleagues found 90% of fourth-year medical students from three schools wanted more education about appropriate use of antimicrobials.21 Only one third of respondents identified themselves as adequately prepared regarding the fundamental principles of antimicrobial use, and only 40% of respondents were familiar with the term antimicrobial stewardship. Justo and colleagues performed a nearly identical study on graduating pharmacy students from 12 US schools of pharmacy and found similar results, with nearly 90% of pharmacy students desiring more education on antimicrobial use.22 Despite rating their education around antimicrobial use more highly than medical students (84% vs 58% rating it useful or very useful) and receiving more didactic training in antimicrobial pharmacology, pharmacy students performed almost identically on the knowledge questions focused on antimicrobial use. Moreover, there was significant variability in knowledge scores between pharmacy schools. Some of that variability may be explained by our study, which found substantial variability between pharmacy schools in the time dedicated to ID instruction and the use of active-learning strategies associated with greater knowledge retention. Additionally, in our qualitative analysis, respondents voiced a desire to change the structure of their ID curriculum, which addressed both instructional methods and allocation of curricular time. However, the amount of time and material involved in ID education was also identified as a concern among respondents.
The current study found that approximately 25% of classroom time was dedicated to active-learning approaches, with the most commonly employed approaches being patient cases and audience response systems. Melber and colleagues surveyed 142 medical schools regarding the preclinical microbiology curriculum and reported that lecture, faculty-led small groups, and team-based learning were the primary teaching methods at the majority of schools.23 The primary teaching resources were faculty-written notes and textbooks. Active-learning methods were not specifically assessed in the survey but were a theme in the qualitative data. Increasing and/or inclusion of active-learning modalities was a theme in recent changes, innovations, and successes.
The ideal amount of classroom time allocated to active learning has yet to be determined. However, Bonura and colleagues found a relationship between the method of classroom teaching used and interest in a career in ID among graduating medical residents.11 Medical residents uninterested in a career in ID reported 78% of their ID education was delivered through traditional lectures. In contrast, residents who applied for an ID fellowship reported 48% of their education was delivered through traditional lecture and 40% through case-based discussion. The strongest predictors of a career in ID was the use of case-based learning and a lack of memorization-based pedagogy. In our study, the second most common active-learning strategy faculty members reported using was audience response systems. Commonly reported benefits of adding audience response questions to lectures are the creation of an active-learning environment, improvements in student engagement and attendance, and increased student comprehension and retention of material.24,25 A randomized study by Mains and colleagues assessed knowledge retention among students who watched a recorded lecture compared to students who watched a recorded lecture with three embedded audience response questions. They found that knowledge retention, both immediately after the lecture and two weeks later, was higher among students exposed to embedded questions.26
Based on our results, considerable variability exists in the delivery of the ID curriculum among US pharmacy schools. Increased educational efforts to standardize ID curricula delivery may facilitate more consistent knowledge among future pharmacists. The Infectious Diseases Educator Network (ID-EN), comprised of over 100 ID pharmacist educators nationwide and representing most of the schools in the current study, may be a resourceful, collaborative group of ID pharmacists to increase such consistency in ID education. The primary aims of the ID-EN are to share educational best practices and resources in ID pharmacotherapy and antimicrobial stewardship, and stimulate collaboration on educational research. The ID-EN provides a forum for ID pharmacy educators to share educational materials and to develop collaborative research initiatives. The group is poised to include additional healthcare disciplines in an effort to improve antimicrobial prescribing for all clinicians and may serve as a model for other specialty educators.
One of the strengths of our study was a survey response rate of 77% of all US pharmacy programs, which minimized the possibility of selection bias. However, several limitations should be considered. When an ID faculty member did not respond to the survey, a curricular dean or department chair was invited to complete the survey. This occurred for 10% of the respondents. Their assessment of ID curriculum may have been different from that of an ID faculty actively teaching the curriculum. Another limitation was the aggregation and estimation of classroom time allocated to lecture and active learning, as a single respondent at each school provided these estimations. For example, we found that schools with a block-type curricular structure reported significantly greater classroom time dedicated to ID than those using a traditional semester or quarter approach. Because block-type structures aggregate basic science knowledge into a disease-state focused block, respondents may have found it easier to include those hours in their estimates, whereas with longitudinally dispersed curricula, those hours may not have been counted. Use of active-learning strategies likely vary based on topic and teacher. Additionally, the ACCP Toolkit was used to identify the scope of topics covered in ID curriculum across pharmacy programs. There could be multiple other ID topics or concepts that were being taught but were not identified through this survey. Finally, because of the descriptive nature of this study, we cannot identify best teaching practices or make curricular recommendations regarding topics or methods. Studies evaluating the relationship between teaching practices and outcome measures, including ID pharmacotherapy knowledge, licensing board test scores, and student career choice and accomplishments, are warranted.
CONCLUSION
As the expectation for more healthcare providers to play a role in slowing the increase of antimicrobial resistance, didactic education about ID and antimicrobial stewardship in PharmD programs must evolve. To our knowledge, this is the first study to evaluate ID curricula within US pharmacy schools. We documented substantial variability across US pharmacy schools in educational time dedicated to ID and antimicrobial stewardship and in the use of high-yield teaching approaches. Communication and collaboration between ID educators may accelerate adoption of best practices and ideally increase teaching efficiency and effectiveness.
ACKNOWLEDGMENTS
The authors would like to thank all survey respondents for their contributions and acknowledge the members of the Infectious Diseases Educator Network for their support.
REFERENCES
- 1.Centers for Disease Control and Prevention, U.S. Department of Health and Human Services Antibiotic Resistance Threats in the United States 2013. https://www.cdc.gov/drugresistance/threat-report-2013/pdf/ar-threats-2013-508.pdf. Accessed September 26, 2019.
- 2.Gallagher JC, Justo JA, Chahine EB, et al. . Preventing the post-antibiotic era by training future pharmacists as antimicrobial stewards. Am J Pharm Educ. 2018;82(6):6770. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Centers for Medicare & Medicaid Services Medicare and Medicaid Programs; Hospital and Critical Access Hospital (CAH) Changes To Promote Innovation, Flexibility, and Improvement in Patient Care. Fed Reg. National Archives. Doc. 81 FR 39447. https://www.federalregister.gov/articles/2016/06/16/2016-13925/medicare-and-medicaid-programs-hospital-and-critical-access-hospital-cah-changes-to-promote. Published June 16, 2016. Accessed August 14, 2016.
- 4.Klepser ME, Dobson EL, Pogue JM, et al. . A call to action for outpatient antibiotic stewardship. JAPhA. 2003. 2017;57(4):457-463. [DOI] [PubMed] [Google Scholar]
- 5.Gubbins PO, Klepser ME, Dering-Anderson AM, et al. . Point-of-care testing for infectious diseases: opportunities, barriers, and considerations in community pharmacy. JAPhA. 2003. 2014;54(2):163-171. [DOI] [PubMed] [Google Scholar]
- 6.Gauthier TP, Worley M, Laboy V, et al. . Clinical infectious diseases pharmacists in the United States: a problem of both supply and demand. Clin Infect Dis. 2015;60(5):826-827. [DOI] [PubMed] [Google Scholar]
- 7.Accreditation Council for Pharmacy Education. Accreditation standards and key elements for the professional program in pharmacy leading to the Doctor of Pharmacy degree 2016. https://www.acpe-accredit.org/pdf/Standards2016FINAL.pdf. Accessed March 5, 2018.
- 8.Schwinghammer TL, Crannage AJ, Boyce EG, et al. . The 2016 ACCP pharmacotherapy didactic curriculum toolkit. Pharmaco. 2016;36(11):e189-e94. [DOI] [PubMed] [Google Scholar]
- 9.Southwick F, Katona P, Kauffman C, et al. . Commentary: IDSA guidelines for improving the teaching of preclinical medical microbiology and infectious diseases. Acad Med. 2010;85(1):19-22. [DOI] [PubMed] [Google Scholar]
- 10.Creswell J, Clark V. Designing and Conducting Mixed Methods Research. 2nd ed. Thousand Oaks, CA: SAGE Publications Inc; 2011. [Google Scholar]
- 11.Bonura EM, Lee ES, Ramsey K, Armstrong WS. Factors influencing internal medicine resident choice of infectious diseases or other specialties: a national cross-sectional Study. Clin Infect Dis. 2016;63(2):155-163. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Sullivan GM. A primer on the validity of assessment instruments. J Grad Med Educ . 2011;3(2):119-120. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Thomas D. General inductive approach for analyzing qualitative evaluation data. Am J Eval. 2006;27(2):237-246. [Google Scholar]
- 14.DiazGranados CA, Abd TT. Participation of clinical pharmacists without specialized infectious diseases training in antimicrobial stewardship. Am J Health Syst Pharm. 2011;68(18):1691-1692. [DOI] [PubMed] [Google Scholar]
- 15.Ernst EJ, Klepser ME, Bosso JA, et al. . Recommendations for training and certification for pharmacists practicing, mentoring, and educating in infectious diseases pharmacotherapy. Pharmaco. 2009;29(4):482-488. [DOI] [PubMed] [Google Scholar]
- 16.Woodruff A, Albanese N, Prescott W. Comparing Pharmacotherapy Instruction to the 2009 and 2016 ACCP Toolkit Recommendations. Am J Pharm Educ. 2018;82(10):Article 6771. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Kufel WD, Jeffres MN, MacDougall C, Cho JC, Marx AH, Williams DM. Antimicrobial stewardship education in US colleges and schools of pharmacy. J Antimicrob Chemother. May 2018. [DOI] [PubMed] [Google Scholar]
- 18.Minen MT, Duquaine D, Marx MA, Weiss D. A survey of knowledge, attitudes, and beliefs of medical students concerning antimicrobial use and resistance. Microb Drug Resist. 2010;16(4):285-289. [DOI] [PubMed] [Google Scholar]
- 19.Weier N, Thursky K, Zaidi STR. Antimicrobial knowledge and confidence amongst final year medical students in Australia. PLoS One. 2017;12(8):e0182460. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Dyar OJ, Nathwani D, Monnet DL, et al. . Do medical students feel prepared to prescribe antibiotics responsibly? Results from a cross-sectional survey in 29 European countries. J Antimicrob Chemother. 2018;73(8):2236-2242. [DOI] [PubMed] [Google Scholar]
- 21.Abbo LM, Cosgrove SE, Pottinger PS, et al. . Medical students' perceptions and knowledge about antimicrobial stewardship: how are we educating our future prescribers? Clin Infect Dis. 2013;57(5):631-638. [DOI] [PubMed] [Google Scholar]
- 22.Justo JA, Gauthier TP, Scheetz MH, et al. . Knowledge and attitudes of doctor of pharmacy students regarding the appropriate use of antimicrobials. Clin Infect Dis. Oct 2014;59 Suppl 3:S162-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Melber DJ, Teherani A, Schwartz BS. A Comprehensive survey of preclinical microbiology curricula among US medical schools. Clin Infect Dis. 2016;63(2):164-168. [DOI] [PubMed] [Google Scholar]
- 24.Pradhan A, Sparano D, Ananth CV. The influence of an audience response system on knowledge retention: an application to resident education. Am J Obstet Gynecol. 2005;193(5):1827-1830. [DOI] [PubMed] [Google Scholar]
- 25.Rubio EI, Bassignani MJ, White MA, Brant WE. Effect of an audience response system on resident learning and retention of lecture material. AJR Am J Roentgenol. 2008;190(6):W319-322. [DOI] [PubMed] [Google Scholar]
- 26.Mains TE, Cofrancesco J, Milner SM, Shah NG, Goldberg H. Do questions help? The impact of audience response systems on medical student learning: a randomised controlled trial. Postgrad Med J. 2015;91(1077):361-367. [DOI] [PubMed] [Google Scholar]