Comparison of Active Learning Techniques: Audience Response Questions Versus Small Group Discussion on Immediate‐ and Long‐term Knowledge Gain

Abstract

Objectives

Active learning techniques help with motivation, involvement, and retention during didactics. There are few studies comparing different active learning methods, and these have yielded mixed results. The objective of this study was to compare the effect of two active learning methods—small‐group discussion and audience response system (ARS)—on immediate‐ and long‐term knowledge gain.

Methods

This was a prospective experimental study of emergency medicine (EM) subinterns and residents. Participants were randomized into two groups, and baseline knowledge was assessed with a multiple‐choice pretest. Didactic sessions on salicylate toxicity and ocular trauma were given to both groups utilizing either small‐group discussion or ARS. A crossover design was utilized to ensure that both groups received instruction by each method. A multiple‐choice posttest was administered following the didactics and again 2 months later. Pre‐ and posttests were identical. All test items were written by an academic faculty member with advanced training in medical education and item writing and were based on the goals and objectives of the session. Test items were piloted with a reference group of learners. Didactic instructors were blinded to test items. Data were analyzed using a linear mixed‐effects model.

Results

Thirty‐eight subinterns and residents participated in the study. Both instructional methods showed immediate‐ and long‐term knowledge gain. The linear mixed‐effects model did not demonstrate any significant difference between instructional methods on immediate knowledge gain (mean difference = 0.18, p = 0.62, 95% confidence interval [CI] = −0.52 to 0.88) or long‐term knowledge gain (mean difference = −0.42, p = 0.36, 95% CI = −1.32 to 0.47).

Conclusion

In this small study, there was no significant difference between instructional methods on immediate‐ and long‐term knowledge gain in EM subinterns and residents.

The use of active learning techniques in medical education continues to grow. ¹ Active learning techniques, based on constructivist learning theory, allow learners to actively engage in material and build new knowledge and understanding by contextualizing content and connecting new information to past knowledge and experiences. ² , ³ Active learning urges trainees to complete higher‐level objectives on Bloom’s taxonomy rather than focusing on memorization. ⁴ There is a large body of literature demonstrating the benefits of active learning techniques on outcomes such as knowledge acquisition, retention of information, perceived competence, critical thinking, and clinical decision making. ⁵ , ⁶ , ⁷ , ⁸ , ⁹ , ¹⁰ , ¹¹ , ¹² , ¹³ , ¹⁴ These techniques vary and may include flipped classroom, audience response systems (ARS), games, debates, case‐based problems, small‐group discussions, team‐based activity, etc. ¹

Case‐based small‐group discussions and ARS are two examples of active learning techniques that have been employed extensively in the medical education setting with positive outcomes. ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ²⁸ , ²⁹ , ³⁰ , ³¹ , ³² Case‐based small‐group discussion has been shown to positively impact knowledge acquisition, critical thinking, communication skills, procedural competency, and academic performance. ¹⁵ , ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ ARS, which supports active learning and increases participation by allowing all learners to answer questions posed by the instructor simultaneously, has been shown to augment learning, increase attention levels, and increase learner engagement. ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ²⁸ , ³⁰ , ³¹ , ³²

Active learning techniques, including case‐based small‐group discussions and ARS, have been shown to be superior to traditional lectures for learning outcomes and have been positively viewed by learners. ⁷ , ⁹ , ¹⁰ , ¹² , ¹³ , ¹⁴ , ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ³³ However, there are limited data comparing various types of active learning techniques, and the few studies published have yielded mixed results. ¹⁴ , ³⁴ , ³⁵ The objective of this study was to compare the effect of two active learning methods—case‐based small‐group discussion and lectures incorporating an ARS—on immediate‐ and long‐term knowledge gain in emergency medicine (EM) subinterns and residents. We hypothesized that knowledge gain would be similar between instructional modalities.

METHODS

Study Setting and Participants

This study took place in the Department of Emergency Medicine at Harbor–UCLA, a large, urban, county medical center. All residents and subinterns attending the EM didactic conference were eligible to participate. There were no exclusion criteria for those meeting inclusion criteria. Data were collected between January and March 2017. This study was approved by the institutional review board of the Los Angeles Biomedical Research Institute at Harbor–UCLA Medical Center.

Study Design

This was a prospective randomized crossover trial. The study was explained to potential participants on the day of the study. Potential participants were allowed to ask questions prior to participation. All potential participants were required to participate in the didactic sessions as this was part of their normal education, but they were permitted to opt out of having their data included in the analysis for the purpose of the study. Didactic sessions consisted of 1) ARS lectures, which incorporated six ARS questions per lecture, and 2) small‐group discussion sessions that were case‐based and incorporated all stages of case management from pathophysiology and epidemiology to history and physical examination to diagnostic tests and therapeutic plans. Participants were randomized to either group A or group B as they arrived for didactic conference on the day of study initiation. Prior to instruction, participants completed a multiple‐choice pretest to assess baseline knowledge and demographic information. Participants were then provided with didactic education on salicylate toxicity and ocular trauma, utilizing the instructional methods determined by their group assignment. A crossover design was utilized to ensure that both groups received instruction by each method. Group A received didactic sessions on salicylate toxicity utilizing small‐group discussion and ocular trauma utilizing ARS lecture. Group B received didactic sessions on salicylate toxicity with ARS and ocular trauma with small‐group discussion. Didactic instructors were two postgraduate year (PGY)‐4 residents participating in the program’s education selective, a longitudinal scholarly track designed to give residents specialized training; experience; and mentorship in educational theory, instructional methods, and education scholarship. A single instructor was assigned to each group (A or B) and taught the didactic sessions according to their group assignment. A multiple‐choice posttest was administered immediately following the didactics to assess immediate knowledge gain and again 2 months later to assess long‐term knowledge gain. Each participant was assigned a unique study ID number, and only this ID number was present on the knowledge tests to track performance. Correct answers and explanations to the test questions were not provided. Instructors were blinded to test items.

Instrument Development

Pretests, immediate posttests, and delayed posttests were identical. To maximize content validity, all test items were written by an academic faculty member with advanced training in medical education and item writing and were based on the goals and objectives of the session. Test items were piloted with a reference group of learners that included various levels of residents as well as faculty to optimize response process validity. These learners were not part of the experimental sample. Scores of pilot testing increased with years of training, demonstrating known‐group validity. Based on feedback from pilot testing, one item regarding aspirin toxicity was deleted, yielding a final test of 29 items (14 covering the topic of aspirin toxicity and 15 covering ocular trauma). Final test items are available in Data Supplement S1 (available as supporting information in the online version of this paper, which is available at http://onlinelibrary.wiley.com/doi/10.1002/aet2.10464/full).

Data Analysis

Differences in the scores between groups were analyzed using a linear mixed‐effects model, which included fixed effects for topic, level of training, and intervention group as well as a random effect for each subject. Two linear mixed‐effects models were fitted: one to compare pre‐ and posttest scores and one to compare pretest scores and delayed posttest scores. Analyses were performed using the “nlme” package in R program for statistical computing (version 3.3.2).

A post hoc power calculation was performed to determine the sample size required to detect a significant difference in mean scores of 2 points, assuming a variance of 2 points and a type 1 error of 0.05. The calculation was performed using PASS 16 Power Analysis and Sample Size software (2018 NCSS, LLC).

RESULTS

All 38 residents and subinterns present on the date of administration participated in the didactic sessions and consented to have their data analyzed for the purpose of the study. Demographic data of participants are displayed in Table 1. Completion rates for aspirin toxicity pretests, immediate posttests, and delayed posttests were 38 of 38 (100%), 38 of 38 (100%), and 28 of 38 (74%), respectively (Table 2). Completion rates for ocular trauma pretests, immediate posttests, and delayed posttests were 35 of 38 (92%), 38 of 38 (100%), and 28 of 38 (74%), respectively (Table 2). Mean test scores by topic and instructional method are displayed in Table 2. Knowledge gain scores by instructional method are displayed in Table 3.

Table 1.

Participant Demographics

	Group A	Group B	p‐value
Male sex	9 (47)	15 (79)	0.09
Training level
MS IV	4 (21)	4 (21)
PGY‐1	1 (5)	2 (11)
PGY‐2	6 (32)	2 (11)	0.65
PGY‐3	3 (16)	5 (26)
PGY‐4	5 (26)	6 (32)
Total	19 (100)	19 (100)

Data are reported as n (%).

Table 2.

Test Scores by Topic and Instructional Method

	ARS	Small‐group Discussion
Pretest
Eye trauma	10.9 ± 1.7 (18)	10.5 ± 2.7 (17)
Aspirin toxicity	8.3 ± 2.2 (19)	8.7 ± 1.9 (19)
Immediate posttest
Eye trauma	12.9 ± 1.4 (19)	13.2 ± 1.8 (19)
Aspirin toxicity	12.2 ± 1.8 (19)	12.5 ± 1.3 (19)
Delayed posttest
Eye trauma	13.6 ± 1.1 (14)	12.9 ± 1.9 (14)
Aspirin toxicity	10.8 ± 2.3 (14)	10.4 ± 1.8 (14)
Difference pre–post tests
Eye trauma	2.0 ± 1.4 (18)	2.5 ± 2.0 (17)
Aspirin toxicity	3.9 ± 1.8 (19)	3.7 ± 1.6 (19)
Difference pretest–delayed posttests
Eye trauma	2.1 ± 1.3 (14)	1.8 ± 2.6 (13)
Aspirin toxicity	1.6 ± 2.1 (14)	1.2 ± 1.9 (14)

Data are reported as mean ±SD (n).

ARS = audience response system.

Table 3.

Gain Scores by Instructional Method

	ARS	Small‐group Discussion
Pre–post	3 ± 1.9	3.1 ± 1.8
Pre–delayed post	1.9 ± 1.7	1.5 ± 2.2

Data are reported as mean ± SD.

ARS = Audience response system.

There was no significant difference between the pretest scores and immediate posttest scores between the intervention groups (mean difference attributable to teaching method = 0.18, p = 0.62, 95% confidence interval [CI] = −0.52 to 0.88; Figure 1), nor between the pretest and retention test scores (mean difference attributable to teaching method = −0.42, p = 0.36, 95% CI = −1.32 to 0.47). The level of training, as measured by PGY, did have a significant effect on the difference between pre‐ and posttest scores, with a decrease in 0.31 points per PGY (p = 0.034, 95% CI = −0.34 to −0.59). The PGY did not have a significant effect on the difference between immediate posttest and delayed posttest scores (mean difference attributable to PGY = −0.55, p = 0.07, 95% CI = −1.12 to 0.28; Figure 2).

Figure 1.

Difference between pre‐ and immediate posttest scores by topic and intervention difference between pre‐ and posttest.

Figure 2.

Difference between pre‐ and delayed posttest scores by topic and intervention difference between pretest and retention test.

DISCUSSION

In this study, there was no significant difference in immediate‐ and long‐term knowledge gain between instructional methods assessed. Both methods demonstrated immediate knowledge gain, although this gain was attenuated on delayed testing. These findings are in line with prior research demonstrating the effectiveness of these educational methods. ¹⁹ , ²⁰ , ²² , ²³ , ²⁴ , ³¹ , ³² The use of active learning is well supported by the literature. ⁵ , ⁶ , ⁷ , ⁸ , ⁹ , ¹⁰ , ¹¹ , ¹² , ¹³ , ¹⁴ , ²⁹ Active learning techniques promote engagement and interactivity which is important for successful transfer of information. ³⁶ , ³⁷ , ³⁸ Additionally, active learning methods are aligned with millennial preferences. ³⁹ This “buy‐in” from learners may also augment the impact of these methods. The results of this study support the idea that it is not the specific instructional modality that matters but rather the use and promotion of active learning that is important, as has been proposed by prior literature. ⁴⁰ If these results are replicated in future larger studies comparing various active learning methods, this could support educators in choosing from a variety of active methods for their instructional sessions based on their knowledge, experience, and resources without fear of compromising the learning of their trainees, thus providing enormous flexibility. Given the diversity of faculty and technological resources across institutions, this flexibility is important.

Interestingly this study found a negative association between PGY and immediate knowledge gain. While this may seem counterintuitive, this effect can be explained by the fact that more senior residents generally had a higher pretest score and thus had less room to increase their scores on the posttest. We did not find an association between PGY and long‐term knowledge gain suggesting that knowledge retention is stable across the studied learner levels.

LIMITATIONS

This study took place at a single institution, so the results may be difficult to generalize. We did not block randomize and, consequentially, the two comparison groups were different in terms of sex and training level. As mean pretest scores were not significantly different between the groups, and we evaluated differences in gain scores effectively using individuals as their own control, we do not believe that these group differences impacted our results in a significant way. Additionally, all tests were identical, so it is possible that our results may have been influenced by a testing effect. As correct answers and explanations were not provided, we expect this impact to be minimal. It is possible that a ceiling effect of the test could have limited our ability to detect differences between modalities; however, as few participants achieved the maximum score on any test, we do not believe that this effect had a large impact. We did not assess quality of instruction and it is possible that this may have influenced our results. However, both instructors were of the same level of training and had both undergone specialized education in instructional delivery, and each taught both active learning modalities so we do not believe that this impacted our results greatly. Based on feedback from instrument piloting, one test item was deleted. We did not have enough data to perform a point biserial. This could be considered in a future study if the same set of questions is to be used again. Importantly, this study only compared two techniques and assessed content knowledge gain in two domains. This study did not assess processes, skills, or behaviors so it is unknown if differences between instructional methods would be seen in these other outcomes. Our sample size was small and a sample size calculation was not performed prior to study initiation. However, a post hoc power calculation determined that our sample size had 97% power to detect a difference of 2 points in the “difference of the difference” (the difference in the mean difference between pretest and posttest scores compared between teaching methodology groups, assuming a 5% type I error and a subject‐level random effect of 1.5). The minimum sample size that would have been required to produce 80% power under the same assumptions would have been 10. Larger studies comparing other forms of active learning techniques on various educational content and outcomes are needed.

CONCLUSIONS

In this small study, there was no significant difference between the instructional methods assessed on immediate‐ and long‐term knowledge gain in emergency medicine subinterns and residents.

Supporting information

Data Supplement S1. Supplemental material.

AEM Education and Training 2021;5:1–6