Abstract
Objectives
To develop a valid and reliable active-learning inventory tool for use in large classrooms and compare faculty perceptions of active-learning using the Active-Learning Inventory Tool.
Methods
The Active-Learning Inventory Tool was developed using published literature and validated by national experts in educational research. Reliability was established by trained faculty members who used the Active-Learning Inventory Tool to observe 9 pharmacy lectures. Instructors were then interviewed to elicit perceptions regarding active learning and asked to share their perceptions.
Results
Per lecture, 13 (range: 4-34) episodes of active learning encompassing 3 (range: 2-5) different types of active learning occurred over 2.2 minutes (0.6-16) per episode. Both interobserver (≥87%) and observer-instructor agreement (≥68%) were high for these outcomes.
Conclusions
The Active-Learning Inventory Tool is a valid and reliable tool to measure active learning in the classroom. Future studies are needed to determine the impact of the Active-Learning Inventory Tool on teaching and its usefulness in other disciplines.
Keywords: active learning, assessment, peer observation, perceptions, faculty development
The concept of student engagement is becoming more than just educational rhetoric. Active-learning techniques have emerged as strategies for instructors to promote engagement with both discipline material and learning.1 Reports by the National Survey of Student Engagement (NSSE) demonstrate that a high level of student engagement increases learning and retention of material particularly in underrepresented minority students.2,3 While instructors' engagement with students may occasionally occur outside the classroom (eg, experiential learning activities, casual conversation), it most often occurs in the classroom. Student engagement in the classroom involves the student participating in a Didaktik triangle interaction between the instructor, fellow students, and the discipline material.4 The most challenging aspect of this relationship for instructors to establish is the student interaction and engagement with material beyond the basic level of knowledge and comprehension. We have begun to think of this learning process as “the continuum of engagement,” where students are presented with multiple pathways to engage in learning that must begin with being actively engaged in the classroom.
The advantages of active learning are numerous. Integrating active-learning strategies into the classroom results in a strong model of teaching because active learning promotes the application of material while it is still being presented.5,6 Active-learning techniques engage students more deeply in the process of learning course material by encouraging critical thinking and fostering the development of self-directed learning.7,8 Use of active-learning techniques not only benefits students by allowing them the opportunity to practice skills and ask questions, but also benefits instructors by affording them the opportunity to assess the students' understanding and remediate important points on a nearly “real time” basis. Classroom environments that include active learning engage students in their learning, encourage “doing” with understanding, provide the students with opportunities to revise and improve their own thinking (formative assessment), and help the students to connect the information from the classroom to practice in the outside world.9
These skills are critical to the personal, professional, and intellectual development of health care providers. The Accreditation Council for Pharmacy Education (ACPE) and the American Association of Colleges of Pharmacy (AACP) both encourage a doctor of pharmacy (PharmD) curriculum that directly involves students, facilitates the transition from “dependent to independent learner,” and develops in graduates “the ability to integrate and apply learning to both the present practice of pharmacy and the advancement of the profession.”10,11 Active learning can create an environment that facilitates “learning in real time” either in collaboration with others or independently, and this is central to the philosophy of practice-oriented education at our institution.12 Lastly, active learning can be viewed as the first step along an experiential learning continuum that promotes more substantive learning outcomes.
While active learning has been validated as an effective way to increase student learning, and is increasingly being incorporated into the classroom, a search of the literature failed to identify an “inventory tool” to quantify and characterize the use of active-learning techniques by faculty members. Development of a standardized inventory tool that is both valid and reliable provides the ability to document the type and quantity of active learning occurring in classes and establishes a basis for qualitative evaluation of active-learning techniques. In addition, an inventory tool will help determine the type of active learning that is best suited to teach or assess a particular level of knowledge, serve faculty members and program administrators as they seek to evaluate teaching skills and provide comments for improvement, and be valuable for those conducting research in this area. An additional advantage of having an instrument is to provide documentation for the increasing emphasis on measuring outcomes in pharmacy education, but to our knowledge there are no tools available to objectively document the use of effective teaching techniques (such as active learning) to attain these outcomes. We therefore sought to develop a valid and reliable active-learning inventory tool to quantify the use of active learning in large courses. In addition, we compared instructor perceptions of the amount and type of active learning used in the classroom to the amount and type quantified using the Active-Learning Inventory Tool as we considered potential faculty development uses for this tool.
While there is no standard definition for active learning, for the purpose of this paper, we defined active learning as “…students doing things and thinking about what they are doing.”6 In addition, we considered active learning as successfully occurring in the classroom when 3 key components were observed: (1) the context of the activity was explained, (2) the students were engaged in the activity, and (3) there was closure to the activity via reflection. We think of these elements as the CER (context, engagement, reflection) components of successful active learning.
METHODS
A draft of the Active-Learning Inventory Tool was prepared using articles, textbooks, and online references related to active learning, as well as classroom assessment materials which emphasize active-learning techniques as a means to formatively monitor ongoing learning through a course. All materials were reviewed in order to find types of active-learning techniques and methods that have been described to quantify and categorize these active-learning techniques. As our intentions for the use of this tool went beyond mere quantification of activities, we reviewed the literature on how to create change in faculty teaching practices.13,14 Change theory provided the context for conceptualizing the approach we used to develop, introduce, and debrief this tool to instructors. Each active-learning technique was initially categorized according to faculty risk, which was estimated based on the intensity and ease of use of the activity in the classroom. Hypothesizing that a simpler tool would have greater acceptance among faculty members, we consciously grouped similar items together and kept the tool as short as possible. The proposed Active-Learning Inventory Tool that was sent out to reviewers identified, and briefly explained 22 widely accepted active-learning techniques and ranked them by degree of faculty risk as determined by the amount of time and effort it takes to design, implement, and extract learning data from the activity.
To establish the validity of the Active-Learning Inventory Tool, we consulted with expert reviewers who had published and researched extensively in the field of education. The experts were asked to review the Active-Learning Inventory Tool and then comment on the use of terminology and descriptions, appropriateness of the specific activities included, overall validity of the assessment, ease of use, and generalizability to other academic disciplines, and provide general comments. The tool was subsequently modified based on the results of their written and verbal feedback, including reorganizing the rank order of activities based on complexity versus faculty risk. Based on expert feedback, coding schemes and descriptions of active-learning techniques were clarified and the rank order of techniques was changed to reflect complexity of the active-learning activity as opposed to faculty risk. The complexity of an activity was assigned based on a combination of findings from the literature review and consensus of the authors.15 The section of tool that asked for qualitative comments about the use of active learning was also clarified.
After approval by the Northeastern University Institutional Review Board, the revised Active-Learning Inventory Tool was tested for reliability in 2 stages: assessment using 3 videotaped 1-hour pharmacy lectures and then in 6 live lectures in large, school of pharmacy courses. Four trained observers (1 educational expert and 3 pharmacy faculty members) participated in this phase of the investigation. Of the 4 observers, 2 had received prior formal training in adult teaching and learning. All observers participated in extensive discussions to develop a common understanding of the definition of active learning and to recognize the CER elements that would lead to successful implementation of an active-learning activity. We felt that it was important to include faculty members with and without prior extensive education in active learning to ensure usability by our target audience of doctoral trained faculty members who may not have training in the concepts of adult learning and active learning.
Lectures were selected during the summer 2005, fall 2005, or spring 2006 semesters when at least three fourths of the observers could attend and when the instructor was willing to participate. Nine lectures (3 videotaped and 6 live) were used for reliability evaluation. Lectures were given in the summer (N = 1) and fall semesters of 2005 (n = 4), and spring semester 2006 (n = 4) in the Therapeutics (n = 6), Self-Care Therapeutics (n = 1) and Pathophysiology (n = 2) courses. Students were in the third-professional year (P3) of a doctor of pharmacy degree program with an approximate class size of 100. Immediately following each lecture, the reviewers met to compare initial results and propose modifications to the Active-Learning Inventory Tool, including changes to improve its ease of use and clarity. The most difficult item to capture on the Active-Learning Inventory Tool was “Code A: Question & Answer.” Differences around this item were resolved by consensus and the Active-Learning Inventory Tool was revised accordingly prior to the next classroom evaluation.
Seven of the instructors were interviewed following their lecture using a scripted interview guide to elicit perceptions of their lesson that included: their definition of active learning, the perceived merits of active learning in the classroom, the types of active-learning activities used in the lecture, the rationale for the use of the specific active-learning activities chosen, the estimated amount of class time that was devoted to active-learning activities, the estimated time required to prepare the lesson and active-learning activities, any perceived barriers to the use of active learning, and the impact of using active-learning techniques on the amount of content covered.16,17 All data from the instructor interviews were coded and analyzed using analyst triangulation with 3 independent coders.
The following outcomes were used to measure agreement among the observers using the Active-Learning Inventory Tool and between the observers and each instructor: number of active-learning episodes used, time per active-learning episode, and the number of different active-learning episodes included in each lecture. Given the small number of episodes of active learning that were observed in each lecture and the revisions that occurred after some of the lectures, interrater reliability could not be estimated using common statistical testing such as the average measure intraclass correlation. Therefore, percent agreement was calculated between each observer for the 3 outcomes and between the observers as a whole and the instructor. Data analysis was performed using Excel and SPSS 6.11 (SAS, Cary, NC).
RESULTS
The Active Learning Inventory Tool was constructed to allow a trained peer observer to record the type, amount, length, and complexity of any observed active-learning teaching behaviors. Each active-learning activity is recorded as a separate “episode” and asks the observer to comment on the quality of the classroom environment during the activity, the overall class atmosphere, and the perceived ease and skill of the instructor.
Through the 8 pilot assessments, 12 additional modifications were made to the Active-Learning Inventory Tool. Most modifications pertained to “Code A: Question & Answer,” with other adjustments made for clarity. Changes to each draft of the tool are described in Table 1 and included clarification of descriptions and a summary page for the reviewer's comment. The frequency of modifications decreased over the development process. The final version of the Active-Learning Inventory Tool is presented in Appendix 1.
Table 1.
Description of Significant Revisions to Active Learning Inventory Tool
Over these 9 lectures, an average of 13 (range: 4-34) episodes of active learning were observed that took an average of 2.2 minutes (range: 0.6-16) each to complete. Three (range: 2-5) different types of active learning were observed per lecture.
Average percent agreement among faculty observers was excellent for each outcome and is presented in Figure 1. The percent agreement for the total number of active-learning episodes in all lectures was 88% (61%-100%), the number of different types of active learning observed was 90% (67%-100%) and the time per active-learning episode was 87% (64%-100%). Although not statistically significant, agreement among the observers improved over time as experience with the Active-Learning Inventory Tool increased (Table 2).
Figure 1.
Percent agreement among observers for each lecture.
Table 2.
Agreement Between Observers and Instructors, Median (Range)
Definitions of active learning varied widely, but all included elements of “doing” and “processing.” The primary perceived merits of using active learning that were cited by instructors included better retention of material (57%) and improved application and critical thinking (29%). The most frequently reported types of active learning used were cases (100%), think/pair/share activities (66%), and the use of a computer-based personal response system (33%). Past exposure or familiarity with a particular active-learning technique was cited as the most common reason for its use. The reported amount of time spent completing active-learning activities varied widely among instructors (range 10%-50% of total classroom time). Most instructors (71%) reported that the use of active learning required more preparation time, particularly if the technique was new to them, if they were unfamiliar with active learning overall, or if this was a new lecture. Barriers to the use of active learning included lack of time (86%), need to remove lecture content (43%), lack of technology in the classroom (14%), and large class size (14%). Most instructors believed that any increase in the time devoted to active-learning came at the expense of lecture content (86%).
Results showed general congruence between instructor perception and the Active-Learning Inventory Tool observations with respect to the time spent completing active-learning activities. With the exception of instructors underreporting their use of student questioning, instructor perceptions about the types and quantity of active learning used in lectures was similar to that observed using the Active-Learning Inventory Tool. Agreement among observers and instructors was good for all outcomes: number of active-learning episodes used (82%; range, 58%-100%); number of different types of active learning used (78%; range, 62%-100%), and average time per active-learning episode (68%; range, 50%-82%).
DISCUSSION
The Active-Learning Inventory Tool is the first tool that utilizes qualitative and quantitative information to capture the amount and type of active learning in the classroom that has been evaluated for validity and reliability. Through an extensive review of the literature on active engagement, consultation with experts in the field, and continual and thoughtful informed reflection, we have created a tool that can clearly measure the amount and type of active learning occurring in a professional curriculum. This tool also helps to identify the gap between faculty perceptions of classroom activities and actual usage of active learning based on the 3 key components that need to be observed (context, engagement, and reflection) to say active learning was occurring in the classroom. In addition, the tool was modified to provide qualitative feedback regarding (1) faculty approach in the activity, (2) quality of classroom environment during the activity, and (3) overall atmosphere. In order for users to obtain maximum benefit from the Active-Learning Inventory Tool, observers should participate in a multi-pronged training session consisting of 3 components. First, an in-depth discussion of the various active-learning activities described in the tool. This ensures all observers have a similar understanding of what each of these activities should involve and will help them recognize the activity when it occurs in the classroom. The second component includes training on how to perform the observation involving discussion and practice of what to observe and how to observe it. Observers should understand that successful active learning is more than just having an activity during class; successful active learning involves CER. The last and perhaps most important segment of the training is the feedback that observers are able to provide to colleagues to motivate them to reflect more deeply on their use of active learning. In our development of the Active-Learning Inventory Tool, we did not observe all the active-learning activities listed on the tool, but with sufficient training and practice as described above, we feel that the tool can capture episodes of successful active learning in classes.
Preparing pharmacy students for practice in the modern healthcare system requires that we rethink pharmacy teaching methodology and go beyond the traditional lecture-based delivery of factual material to incorporate those methods that allow for effective application and problem solving in the classroom. With the changing climate of accountability in higher education, it is no longer sufficient to say we are “good teachers”; we need to have mechanisms that promote real change and growth in faculty teaching skills and that capture the reality of the classroom. We believe the Active-Learning Inventory Tool is a strong tool to serve both these functions. There are broad potential applications for this inventory tool in other professional disciplines as diverse as accounting and nursing education.18-20 Based on current educational standards from these and other professions that promote the routine inclusion of active-learning strategies in professional curriculum, a tool to evaluate active learning will be useful to instructors in various learning venues.
In our pilot testing we found some interesting preliminary themes. Although instructors perceived that they developed full active-learning experiences, the Active Learning Inventory Tool documented only engagement – context and reflection were absent. For example, several instructors asked students to respond to questions, but the questions were perceived as rhetorical or could not be answered by students in the time allowed. Also, most of the instructors interviewed said they highly valued active learning in the classroom, but were reluctant to take class time away from content to conduct activities. They believed that active learning increased retention and involvement, but felt it was too time intensive to use when presenting complex materials. This is a key barrier that needs to be explored further if the use of active learning is to increase.
We were faced with many challenges as we worked to design and modify this Tool. Since nomenclature for active-learning activities varies, we made several changes that focused on clarifying the description of certain active-learning activities. Also, in our experience, pharmacotherapeutics courses lend themselves easily to the use of cases and student questioning, but other disciplines may rely on different techniques. It will be important to expand the evaluation of this tool to other health, science, and humanities disciplines in order to fully comment on the generalizability of the Tool. The Active-Learning Inventory Tool primarily focuses on the quantitative evaluation of active learning rather than qualitative evaluation since quantitative evaluation is far easier for observers who are not experts in active learning to complete. We chose not to have 4 formally trained observers, but rather a mix of 2 formally trained in adult teaching and learning and 2 practice-based faculty members. We believed doing this added to the external validity of the Tool's usefulness by any trained observer. Ideally, any trained observer could use this tool to increase awareness of teaching techniques or perhaps stimulate change in the amount of active learning used in a classroom. Our reliability was generally good, however, the low number of episodes of active learning and the revisions made to the Active-Learning Inventory Tool after each lecture did not allow us to calculate a formal measure of agreement. In addition, observers and instructors were not chosen randomly, nor blinded to the assessment. Lastly, we designed and tested this tool for use in a large class, and its appropriateness for smaller settings still needs to be evaluated.
There are a number of areas in which the Active-Learning Inventory Tool requires further evaluation. Its role as a basis for providing feedback to instructors should be investigated. One potential valuable contribution would be as part of a peer-evaluation process to raise instructor awareness of their usage of active learning and facilitate change in teaching techniques. It may also serve as the foundation for future research or faculty development programs to reduce barriers to the use of active learning in all disciplines and types of content. Lastly, we hope to explore the role of this tool as a change agent to increase use of active learning in large classes or to stimulate a dialog that results in improved student learning in large classes.
CONCLUSION
A valid and reliable tool to measure the type and amount of active learning in large pharmacy classrooms was developed. Use and testing of this tool will be expanded to other university disciplines and other colleges and schools of pharmacy. Most importantly, we hope to increase dialog about the use of effective techniques to improve student learning in institutions across the country.
ACKNOWLEDGEMENTS
We would like to thank our expert reviewers, Charles C. Bonwell, PhD, President, Bonwell Associates and Professor Emeritus of History; Eric H. Hobson PhD, Associate Dean for Academic Affairs and Assessment at South University; and Linda B. Nilson, PhD, Director, Office of Teaching Effectiveness and Innovation at Clemson University, for their valuable feedback. We would also like to thank Dr. Eric Hobson for the use of his active-learning activities chart, the instructors who were kind enough to let us use their classes to test reliability and determine their perceptions, the Northeastern University Provost for the funding to support this work, and Dr. Jennifer Trujillo for critically reviewing the manuscript.
Appendix 1. Final version of the Active Learning Inventory Tool © 2006 Van Amburgh, Devlin, Kirwin, Qualters
REFERENCES
- 1.Qualters DM. Do students want to be active? J Scholarship Teach Learn. 2001;2:51–60. [Google Scholar]
- 2.Kuh G. Assessing what really matters to student learning: inside the national survey of student engagement. Change. 2001;33:10–7. [Google Scholar]
- 3.Wasley P. Underrepresented minority students benefit most from engagement. Chronicle Higher Educ. 2006;53:A39. [Google Scholar]
- 4.Hudson B. Holding complexity and searching for meaning: teaching as reflective practice. J Curriculum Stud. 2002;34:43–57. [Google Scholar]
- 5.Moffett BS, Hill KB. The transition to active learning: a lived experience. Nurse Educator. 1997;22:44–7. doi: 10.1097/00006223-199707000-00015. [DOI] [PubMed] [Google Scholar]
- 6. Bonwell CC, Eison JA. Active Learning: Creating Excitement in the Classroom. ASHE-ERIC Higher Education Report No. 1. Washington, DC: The George Washington University, School of Education and Human Development; 1991.
- 7.Gokhale A. Collaborative learning enhances critical thinking. J Technol Educ. 1995;7:22–30. [Google Scholar]
- 8.Brown MN, Freeman K. Distinguishing features of critical thinking classrooms. Teaching Higher Educ. 2000;5:301–9. [Google Scholar]
- 9. Bransford JD, Brown AL, Cocking RR. How People Learn: Brain, Mind, Experience and School. Washington, D.C.: National Academy Press; 2000. Available at: http://www.nap.edu/html/howpeople1/. Accessed January 04, 2005.
- 10. Background Paper II: Entry level, curricular outcomes, curricular content and educational process. Am J Pharm Educ. 1993;57:377-85.
- 11. Accreditation Council for Pharmacy Education. Accreditation Standards and Guidelines for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree. Adopted January 15, 2006. Available at: http://www.acpe-accredit.org/pdf/ACPE_Revised_PharmD_Standards_Adopted_Jan152006.pdf. Accessed January 17, 2007.
- 12. Center for Work and Learning. Practice-Oriented Education at Northeastern University. Available at: http://www.poe.neu.edu/about/whatispoe.html. Accessed January 06, 2005.
- 13.Prochaska JO, DiClemente CC, Norcross JC. In search of how people change. Am Psychol. 1992;47:1102–14. doi: 10.1037//0003-066x.47.9.1102. [DOI] [PubMed] [Google Scholar]
- 14. Qualters DM, Sheahan T, Isaacs J. An electronic advice column to effect teaching culture change. To Improve the Academy (Professional and Organizational Development Network, http://www.podnetwork.org). 2006;24:201-16.
- 15.Angelo TA, Cross KP. Classroom Assessment Techniques: A Handbook for College Teachers. 2nd ed. San Francisco, Calif: Jossey Bass, Inc; 1993. [Google Scholar]
- 16.Patton M. How to Use Qualitative Methods in Evaluation. London, England: Sage Publications; 1987. [Google Scholar]
- 17.Bogdon R, Iklen S. Qualitative Research for Education: An Introduction to Theory and Methods. Boston, Mass: Allyn and Bacon; 1992. [Google Scholar]
- 18.Accounting Education Change Commission. Objectives of education for accountants: position statement number one. Issues Accounting Educ. 1990;5:307–12. [Google Scholar]
- 19. Albrecht WS, Sack RJ. Accounting Education: Charting the Course Through a Perilous Future. Accounting Education Series No. 16. Sarasota, Fla: American Accounting Association; 2000.
- 20. Commission on Collegiate Nursing Education. Standards for Accreditation of Baccalaureate and Graduate Nursing Education Programs. Available at: http://www.aacn.nche.edu/Accreditation/standrds.htm. Accessed January 06, 2005.