As Hora acknowledges (1), our paper calls for second-generation research in science, technology, engineering, and mathematics (STEM) education that is focused on comparing modes or intensities of active learning (2). As part of this effort, we agree with Hora that researchers should use one of the many validated instruments now available to quantify teaching practice in undergraduate STEM classrooms (e.g., refs. 3 and 4). However, we do not agree with his skepticism about the validity of analyzing existing “first-generation” experiments that do not use one of these instruments (1).
The 225 studies in our analysis all identified their comparison groups as traditional lecturing versus active learning. Each reported the outcome of a contrast between a transmission-intensive, teacher-centric approach and a constructivist, student-centric approach in the same college course. The dichotomy that Hora objects to, then, simply captures the contrast designed and implemented by the experimenters.
Hora (1) is correct in pointing out that not all implementations of traditional lecturing are identical, just as we took pains to point out that not all implementations of active learning were identical. We were well aware that variation in implementation occurred in both treatments and controls; it was our major reason for analyzing the data using random-effects models, which are explicitly designed to account for variation in implementation (5). If all of the implementations had been identical, we would have used fixed-effect models. Indeed, understanding broad patterns, even in the face of variation across studies, is one of the major goals of a research synthesis like ours. Cooper and Hedges illustrate this point with an analogy: If individual studies are like bricks, the meta-analyst’s task is to use a preestablished coding protocol to stack them into a structure, all the while recognizing that “no two bricks are exactly alike” (6).
Thus, variation in implementation does not undermine our analysis, as Hora (1) implies. Instead, it strengthens our findings. If not all of the control conditions represented “pure” traditional lecturing, then our results underestimate the actual impact of active learning compared with “pure” lecturing. With examination data, for example, active learning—compared with strictly expository teaching—may lead to even larger benefits than our summary estimate of 0.47 indicates. We regret not including this insight in our Discussion section (2).
Supplementary Material
Footnotes
The authors declare no conflict of interest.
References
- 1.Hora MT. Limitations in experimental design mean that the jury is still out on lecturing. Proc Natl Acad Sci USA. 2014;111:E3024. doi: 10.1073/pnas.1410115111. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Freeman S, et al. Active learning increases student performance in science, engineering, and mathematics. Proc Natl Acad Sci USA. 2014;111(23):8410–8415. doi: 10.1073/pnas.1319030111. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Nunn CE. Discussion in the college classroom: Triangulating observational and survey results. J Higher Educ. 1996;67(3):243–266. [Google Scholar]
- 4.Turpen C, Finkelstein ND. Not all interactive engagement is the same: Variations in physics professors’ implementation of Peer Instruction. Phys Rev ST Phys Educ Res. 2009;5(020101):1–18. [Google Scholar]
- 5.Raudenbush SW. Analyzing effect sizes: Random effects models. In: Cooper H, Hedges LV, Valentine JC, editors. The Handbook of Research Synthesis and Meta-Analysis. New York: Russell Sage Foundation; 2009. pp. 296–315. [Google Scholar]
- 6.Cooper H, Hedges LV. Research synthesis as a scientific process. In: Cooper H, Hedges LV, Valentine JC, editors. The Handbook of Research Synthesis and Meta-Analysis. New York: Russell Sage Foundation; 2009. pp. 3–16. [Google Scholar]