Systematic review of the implementation of audience response systems and their impact on participation and engagement in the education of healthcare professionals

Abstract

Background

Audience response system provides a mechanism to engage larger groups as active participants in teaching sessions. However, they are traditionally based on ‘fixed’ closed loop system, which limits their functionality to a single geographical location, thus has the effect of confining their use to universities and other larger institutions, with a primary focus on education. Conversely, in the healthcare education context, the majority of formal education is undertaken through postgraduate training programmes, largely conducted in smaller cohorts in clinical settings.

Objective

The purpose of this review is to evaluate audience response systems in terms of feasibility of implementation and the impact on participation within the field of education of healthcare professionals, in comparison to the non-healthcare education.

Study selection

Therefore, systematic structured searches of PubMed and Medline databases for healthcare education were conducted, and Scopus, Education Resources Information Center, British Education Index, Education Abstracts, Education Administration Abstracts and PsycINFO databases for non-healthcare education databases.

Findings and conclusions

Consistent and fundamental differences were found in the studies evaluating healthcare education compared with other fields, with more difficulties encountered in implementation and a less significant impact on engagement seen. Here we discuss the consequences of these findings on the use of audience response systems and beyond.

Introduction

Postgraduate medical education for junior doctors forms a core part of their continuing professional development and serves to guide the acquisition of core clinical knowledge.¹ These seminars are delivered at a local level in individual hospitals and are delivered to groups of between 20 and 40 junior doctors at the early years of their postgraduate career. These are traditionally composed of a series of didactic lectures around critical topics, with a final summative component. The formative component largely takes the format of either a collection of key slides from previous presentations or questions focused on key points asked to individual audience members. The aim is to highlight key aspects of the topics, and aims to reinforce the acquisition of knowledge and highlight the areas of application to future practice, this leading to higher levels of learning, as discussed in Bloom’s taxonomy of learning.² The disadvantage of the first approach is that it largely delivers the same information using the same style, and can therefore prove to be unengaging. In the second approach these questions are usually directed to individual members of the audience, leading to uneven level of engagement across the cohort and therefore excluding the majority of the audience, contrary to evidence that higher levels of engagement lead to higher levels of meaningful learning.³

However, audience response systems or classroom response systems (CRS) have been recently adopted as an effective way of enhancing the levels of engagement, particularly in higher education.⁴ The major drawback of CRS systems is the required technological infrastructure to administer and deliver, confining its use to larger institutions and universities, and effectively limiting the use of CRS within the context of medical education to undergraduate training. This, in turn, effectively excluded postgraduate seminars due to the limitations of infrastructure.

Advances in technology have recently enabled the utilisation of a web-based CRS, such as Poll Everywhere and Conferences i/o among others, offering CRS that is easy to design and administer, as well as being free at the point of use, which may facilitate more widespread adoption. This platform has been demonstrated in the undergraduate setting to good effect,⁵ although no demonstration of CRS or this platform has been made in the healthcare education setting.

This review aims to systematically analyse the available literature, within and without healthcare professional education, and phenomenographically assess for differences between the two contexts.

CRS in education outside healthcare

A systematic review of Scopus and EBSCOHost databases was conducted, using the search terms ‘ audience response system ’, ‘ audience response system ’, ‘ electronic voting system ’, ‘ clickers ’ and ‘ Poll Everywhere ’, with the addition of ‘adult education’, ‘higher education’, ‘undergraduate’ and ‘postgraduate’. In the EBSCO platform the following databases were included: Education Resources Information Center, British Education Index, Education Abstracts, Education Administration Abstracts and PsycINFO. The search was limited to journal articles and reviews published following 2000. The references of included articles were additionally surveyed for further studies with additional narrative value. These choices were made to enable the capture of all relevant publications, while maintaining relevance in a technologically dynamic context. Figure 1 presents a summary of the literature search strategy of this review using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow chart.⁶

Figure 1.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow chart of the search strategy on the use of classroom response systems in educational settings outside of healthcare. Adapted from Moher et al.⁶

Thirty-three studies were reviewed in full, of which 22 identified studies attempted to evaluate the ease of instituting and implementing a CRS system, or addressed the use of CRS in relation to enhancing participation and engagement. Of these, six studies evaluated the ease of implementation, with all indicating that CRS can be incorporated into existing teaching programme without difficulty. Interestingly, MacGeorge et al ⁷ highlighted that anxiety regarding the initial use of CRS exists in lecturers as well as students, with Walklet et al ⁸ emphasising that additional planning time may be required when utilising CRS. Additionally, Bunz⁹ and Smith et al ¹⁰ suggested that a more simplified system with streamlined features may offer greater ease at the point of use; Graham et al ¹¹ added that increased lecturer experience with the technology may facilitate the integration of CRS more effectively into the teaching programme. The studies concerned with ‘cloud’-based CRS platforms suggested that the complexity was centred around appropriate setting of questions, rather than use of the platforms themselves.^{12 13}

Of the 19 studies evaluating student engagement and participation with CRS, 2 studies did not find a statistically significant link between the use of CRS and student engagement^{13 14}; furthermore, an additional study found mixed results in engagement with CRS, with suggestion that some groups may be resistant to the inclusion of the technology into lectures.¹⁵ The remaining studies indicated that subjectively reported engagement was aided by the use of CRS.

An important note was that the use of CRS has been reported to provide a more captivating and fun experience for the students.^{11 16 17} Curiously, the evidence to whether this leads to a motivational boost to engage with the materials outside of CRS lecture is divided, with Hoekstra¹⁸ and Smith et al ¹⁹ suggesting that anticipating participation led to students undertaking preparatory work; in contrast, Laxman²⁰ and Oigara and Keengwe²¹ did not find that engagement continued outside of the lectures.

Several studies suggested that CRS may be of specific benefit to groups that may otherwise be under-represented in other modalities of lectures and small group teaching.^{17 22} However, Heaslip et al ²³ highlighted that students with physical disabilities can encounter difficulties in utilising the CRS systems.

CRS in healthcare education

A systematic review of Medline and PubMed databases was conducted using terms of ‘audience response system’, ‘audience response system’, ‘electronic voting system’, ‘clickers’ and ‘Poll Everywhere’. Figure 2 presents a summary of the literature search strategy according to the PRISMA flow chart.¹ These databases were chosen as they include health education journals. The references of included articles were additionally surveyed for further studies with additional narrative value.

Figure 2.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow chart of the search strategy on the use of classroom response systems in educational settings within healthcare. Adapted from Moher et al.⁶

Of the 149 abstracts screened, 8 was excluded as duplicates, and 36 studies were identified for full assessment. After evaluating the full articles, 13 studies were included in the review, with an additional 2 studies identified through screening the reference list of included studies. All the studies included in the review used CRS systems ‘fixed’ to a set lecture hall or auditorium. Two studies did not find a clear suggestion of CRS improving participation^{24 25}; in addition to these, three further studies stated that significant difficulties, either in integration of CRS into the existing teaching programme or in the technical set-up, hampered its implementation.^26–28 In contrast, Llena et al,²⁹ while not explicitly set out to assess the implementation of CRS, did find the platform easy to use. However, with a third of studies in the review highlighting difficulties in implementing CRS, significant impediments to CRS may exist in healthcare education.

Interestingly, the impact on participation exhibited a wide range of variability, with Stevens et al ³⁰ establishing only a minor benefit in student participation, and the majority of studies quantifying student participation demonstrating a greater benefit.^31–33 Pettit et al ³⁴ commented that increased participation was established with the addition of a competitive element to CRS. Furthermore, the effects on learning have been reported as mixed, with Sternberger³⁵ suggesting that exposure and utilisation of CRS foster the development of critical thinking earlier in training, in contrast to Filer²⁷ and Rahman et al ³⁶ suggesting that although increased participation occurred with CRS, this did not translate into more effective learning. Patterson³² was the sole study that commented on reduced participation following technical difficulties, citing this as the major drawback of CRS systems.

Only one study evaluated the use of CRS in postgraduate education of healthcare professionals, with a study following up of a small cohort of trainees through multiple sessions of CRS-enhanced sessions throughout the study period.²⁶ Limitations of this study include the small number of participants, with a maximum of 10 trainees included at any point. This suggests that CRS is an under-researched area in healthcare education, particularly in postgraduate medical education, which may be a legacy of earlier technology.

Analysis and discussion

Due to a large degree of heterogeneity in study design and research methodology, no meta-analysis was possible based on these systematic reviews. It is interesting to note that while CRS is used easily and without any significant obstacles, incorporation of CRS into healthcare education programmes has been hampered by significant difficulties at various stages. There may be a variety of reasons for this discrepancy, with the lack of support from dedicated information technology services within healthcare education being a distinct justification, particularly in contrast to higher education institutions. Second, the paradigm of healthcare education may not lend itself readily to a more time-consuming method of group learning.

It is also worth noting that there was an additional significant incongruity with regard to the impact on participation between healthcare education and non-healthcare education, with a far greater benefit derived from CRS outside of healthcare education. While there is no clear reason for this divergence of impact, it may be due to the baseline level of participation and motivation in the two context.

Conclusions

The use of CRS may be seen as advancing active, rather than passive, learning. The advantages of pursuing an active learning approach were discussed by Cleveland et al ³⁷ and Torre et al,³⁸ who suggested that utilising this style leads to the development of a deeper understanding, enhances engagement with the material at hand, and encourages discussion and peer learning. This is on a background of a growing body of evidence that peer learning provides a unique learning opportunity, with a reciprocal benefit for both those receiving teaching and the individuals providing it.^39–41 Therefore, the fostering of this relationship through encouraging a group discussion and learning may be a useful side effect of using CRS.

It may be that CRS’ greatest strength is to encourage learning through an active rather than a passive and receptive process. It has been suggested that active learning leads to a greater understanding of the topic and appreciation of the impact of the information, as well as encouraging the students to take more ownership of their own learning.^{37 42–44} When applied to professional education, active learning can nurture the development of a competency and growth of the ‘professional identity’.⁴⁵ When applied to healthcare education, active learning techniques lead to greater student satisfaction, in addition to improved academic performance and enhanced resilience.^{46 47}

The incongruity between healthcare education and non-healthcare education settings on both implementation and engagement in the literature suggests that findings in educational research in general may not always be directly applicable to healthcare education, although the reasons for this are unclear and could be investigated in more detail in the future. It would therefore be prudent to caution against accepting educational research from settings outside healthcare without trialling them first.

Therefore, based on the findings presented here, the integration of CRS into postgraduate healthcare education programmes is likely to be beneficial. As these programmes, out of practical necessity, include didactic lectures, CRS may serve to promote a more active learning process, to the benefit of both the audience and their patients. Consequently, the additional preparation and planning required for successful implementation of CRS, with alignment to the learning objectives, may be justified. However, adequate preparation and technical support are vital to success. Furthermore, within healthcare educational settings, CRS may not be relied on to singularly improve audience participation, but may form part of a more comprehensive array of tools for this purpose.