E-learning for Critical Thinking: Using Nominal Focus Group Method to Inform Software Content and Design

Abstract

Background:

Undergraduate nursing students are often confused by multiple understandings of critical thinking. In response to this situation, the Critiique for critical thinking (CCT) project was implemented to provide consistent structured guidance about critical thinking.

Objectives:

This paper introduces Critiique software, describes initial validation of the content of this critical thinking tool and explores wider applications of the Critiique software.

Materials and Methods:

Critiique is flexible, authorable software that guides students step-by-step through critical appraisal of research papers. The spelling of Critiique was deliberate, so as to acquire a unique web domain name and associated logo. The CCT project involved implementation of a modified nominal focus group process with academic staff working together to establish common understandings of critical thinking. Previous work established a consensus about critical thinking in nursing and provided a starting point for the focus groups. The study was conducted at an Australian university campus with the focus group guided by open ended questions.

Results:

Focus group data established categories of content that academic staff identified as important for teaching critical thinking. This emerging focus group data was then used to inform modification of Critiique software so that students had access to consistent and structured guidance in relation to critical thinking and critical appraisal.

Conclusions:

The project succeeded in using focus group data from academics to inform software development while at the same time retaining the benefits of broader philosophical dimensions of critical thinking.

1. Background

University students are required to demonstrate critical thinking skills, as are new graduates when commencing initial employment (1, 2). Academics and employers advocate the need for critical thinking skills (3, 4) which are ‘typically rated higher by employers than academic qualifications’ (2). However there is limited agreement about what is involved in the demonstration of these skills (4-9) with Beyer and Dryden (10) claiming that critical thinking is a commonly misused term. For example, critical thinking is considered an essential requirement for health professional practice (3, 6, 8, 11) and questioning assumptions is regarded as central to critical thinking (1, 3, 12). However in actual practice nurses who question health care practice may be praised, ridiculed or even dismissed depending on precisely what is being questioned and the context in which practice may be challenged (13). For nursing undergraduates this situation can be confusing as students may be criticised for not thinking critically when there is no consensus among academic or clinical staff about what critical thinking entails, or when there are cultural and ethnic differences within the workplace environment (14-17). It was in this context that the Critiique for critical thinking (CCT) project was implemented to provide students with structured guidance to promote critical thinking skills.

Central to the CCT project was the modification of e-learning software called Critiique to produce Critiique for Critical Thinking (CCT), a version of Critiique designed specifically to develop critical thinking skills. To maximise learning opportunities, all student-based activity within CCT can be linked to assessment. Therefore the CCT project was different from most university E-Learning, due to the complete integration of teaching and learning software with assessment (18).

Allen et al. (19) noted that assessment linked to students’ construction of knowledge from web content provided strong motivation for students, and could lead to their development of high quality content. As a knowledge construction and evaluation tool, CCT can achieve this while also providing students with detailed and consistent guidance on assignment construction. Furthermore, the CCT reports produced by students are in themselves potentially valuable, as they provide critical evaluation and synthesis of high quality web-based content, and may be further refined with peer review.

Strong theoretical arguments support this work, which draws upon evidence-based practice, critical thinking (20), authentic assessment (21) and knowledge construction in relation to web 2.0 environments (19). The pedagogic process used with Critiique is novel in that the knowledge resource is not the lecturer, nor a learning management system (LMS) such as Moodle or Blackboard, but rather web-based resources, with the lecturer’s role being to facilitate efficient and effective critical thinking and appraisal processes. The student perspective is also novel, as the student is focused on the end product and associated knowledge construction, with learning occurring as and when the student engages in appraising and synthesising web resources. In his recent Australian Teaching and Learning Council Fellowship report, Allen suggests that:

“Assessment is a powerful mechanism for developing new approaches to teaching since students normatively pay greater attention to, learn more from, and devote most time to assessment, for various reasons. Assessment thus is not just about testing knowledge, but about developing it in the first place; assessment drives student learning and so should be the main focus for pedagogic innovation” (21).

CCT is customised software that aims to teach critical thinking skills by engaging students in an assessment task. CCT was developed at an Australian university to guide large authentic assessment task development, by taking students step by step through the process of writing a critical review of research evidence.

The initial versions of CCT focused on the application of critical thinking to the appraisal of web resources, particularly focusing on the critical appraisal of research papers. CCT was then enhanced so all content was authorable, allowing the software to be adapted to cover specific aspects of research appraisal and more general theoretical content about critical thinking. From the student perspective using CCT is somewhat similar to using e-portfolio software as students are assigned their own data storage area. However, with CCT students work step-by-step through a structured process of guided critical appraisal to construct an assignment, with each component of the assignment stored in a database. CCT is then used to retrieve the content constructed by each student in an automated comparative table and report format. CCT uses an online process that mimics a hard copy filing cabinet. While this process is quite structured, the embedded content still challenges students philosophically and critically by exposing them to new ways of thinking while maintaining sufficient scaffolding to guide the application of critical thinking skills. Specifically CCT helps students learn about critical thinking as they read, hear or view content about critical thinking in small discrete steps at the time they are constructing relevant assignment content. This high level of support and scaffolding for learning provides the basis subsequent more complex critical thinking activities such as research synthesis and reflection upon the processes used.

Figure 1 contains a screen shot of CCT illustrating the authorable section where staff can modify content and the text box where students construct their assignment that is ultimately retrieved into a preformatted report.

Figure 1. Screen Shot of CCT.

The CCT project involved the conduct of nominal focus groups to obtain a measure of consensus about critical thinking, with the focus group data used to inform software content and modification.

The initial textual content in CCT was informed primarily by critical thinking and evidence-based practice theory. In addition, Scheffer and Rubenfeld (22) conducted an international Delphi study to develop a consensus statement on critical thinking in nursing and this work was used as a starting point for the nominal group method implemented in the CCT project.

2. Objectives

To ascertain academic staff views about priorities for teaching critical thinking to undergraduate nursing students. To explore the use of Critiique, a software tool for the teaching of critical thinking to undergraduate nursing students.

3. Materials and Methods

This project employed a modified nominal group process informed by Van de Ven and Delbecq (23) and Levine et al. (24) to develop CCT as a generic assignment construction tool, able to be subsequently customised for specific assessment items. Van de Ven and Delbecq defined nominal group processes as ‘…structured meetings which seek to provide an orderly procedure for obtaining qualitative information from target groups who are most closely associated with a problem area’ (24). This study explored broad, open-ended questions with the desired outcome of obtaining structured, priority-ranked critical appraisal questions. The modified nominal group process involved three main steps: brainstorming, categorising and ranking. Permanent academic staff employed in an Australian University School of Nursing and Midwifery were invited via email invitation to participate in the modified nominal group workshop. Staff members who were unable to attend were given the opportunity to provide electronic input about critical thinking priorities after the workshop, with this input added to the brainstorming step of the nominal group session. Seven staff participated in the study, with 3 staff attending the initial focus group and 4 more staff participating electronically. The study received ethics approval from the Flinders University Social and Behavioural Research Ethics Committee (approval number 6144).

3.1. Data Analysis

The method used involved categorisation during the focus group in accordance with nominal focus group methods. Two researchers then independently summarised the data by reviewing transcribed audio-recorded data and then conferred to establish categorisation and initial ranking. This summarised document was then circulated back to all 3 participants and to a further 4 participants who could not attend the initial focus group. Participants were then asked to re-rank or confirm ranking of the identified categories with 10 categories 41 sub-categories identified. The categorising step of the nominal group process was conducted in a manner that paralleled CCT software structure to produce data outputs that could be used to guide software modification. A participation target of 10 - 20 academic staff was sought for the workshop.

While seven staff in total participated, the data generated were extensive and rich, and provided the basis for software modification with inbuilt feedback mechanisms allowing for further ongoing modification.

During the 90-minute nominal focus group workshop, academic staff considered and addressed the following questions:

• - How can critical thinking be introduced to students?

• - How should we assess critical thinking?

• - What are your preferred theoretical approaches to critical thinking?

• - What critical thinking resources should be available for students?

• - How should critical thinking applied to the appraisal of web-based resources?

• - What factors should students consider when applying information from web-based resources?

The session was audio recorded with participants de-identified. The session was noted and summarised by the lead researcher with the preliminary summary document emailed to electronic participants for feedback. Electronic participants indicated items where they supported consensus, noted points of disagreement and added additional information in response to the questions. The lead researcher then tallied comments in terms of their frequency and rating, with emerging themes thus identified. A second researcher then reviewed the audio and summary data to verify these themes.

3.2. Ethical Considerations

The study was approved by the social and behavioural research ethics committee at the university where the study was conducted. Participant data was de-identified in accordance with this ethical approval.

4. Results

The nominal focus group results are listed sequentially under respective focus group questions with the more frequent responses listed first.

How can critical thinking be introduced to students?

Participants suggested relating critical thinking to nursing by providing direct examples of the application of critical thinking to nursing practice. Other responses with less frequent identification included teaching application of critical thinking in everyday life, introduction of philosophy related to critical thinking and encouraging self-responsibility:

“Start with how they use critical thinking in everyday life and nursing practice. They are put off by the language, place it in the everyday.”

There were interesting participant comments about younger students and their ability to use critical thinking:

“Gen Y are the least able to employ critical thinking because parents have done everything for them.”

“Look at games that they play, do their games influence them? Do they play games that they need to think about or problem-solving games? Instant and reactive.”

How should we assess critical thinking?

Participants advocated assessing the application of theoretical knowledge to the clinical environment using authentic patient scenarios as the basis of assessment items. Other suggestions included students explaining rationale for actions in laboratory settings, simulated handover and the use of oral assessments.

Participants highlighted difficulties for students demonstrating written critical thinking:

“Writing as they are speaking … Writing is as spontaneous as speech, generation who don’t understand that this is permanent- don’t revise, rewrite etc. Disconnect from context occurs this applies to local students. No editing of what they write.”

What are your preferred theoretical approaches to Critical Thinking (CT)?

Participants did not respond by citing particular authorities in the field of critical thinking but more generally identified approaches that acknowledge cultural aspects of CT, while highlighting a preference for Western approaches to critical thinking. Other aspects suggested included emphasising rationale for decision making and involving the art and culture of other thinkers.

What critical thinking resources should be available for students?

Participants suggested e-learning resources that require students to actively apply concepts to patient scenarios, rather than just passively watching videos and reading. Other suggestions included the use of branched patient scenarios with embedded videos, explicit teaching of critical thinking, a staged approach to teaching with well-scaffolded first-year content and the inclusion of broad content about critical thinking.

“Include video that shows clinical branching where students have to make decision”

How should CT be applied to the appraisal of web-based resources?

Participants strongly advocated the use of appraisal tools, noting that the same appraisal processes should be used for content whether web-based or not. Specific areas highlighted included peer review status, currency, context (country/medical system) author credibility and whether content is opinion or research based:

“Use of appraisal tools-good for students as they give the students a way to start looking at the information, Need to start with something that allows them to base their thinking on”.

What factors should students consider when applying information from web-based resources?

Participants emphasised the need for critical appraisal of content before applying this to nursing practice, suggesting students should go beyond the websites that are for the general public and examine relevance, fit and ethical application to practice. They suggested that students should be questioning all the time, even of their own behaviour.

“How would you apply this for your practice, how would you develop/appraise yourself, students should be questioning all the time”

4.1. Further Results

Other themes emerging from the data but not directly related to the focus group questions included the need for face-to-face teaching of critical thinking, adequate time for instruction and discussion, and the importance of social responsibility. Participants highlighted cross-cultural differences in critical thinking, particularly in relation to Asian students. However this issue was not discussed in depth by participants.

In summary, as anticipated, there was not complete consensus obtained from participants in relation to the focus group questions. However, the nominal focus group process was effective in obtained sufficient agreement to guide the development of teaching resources and specifically CCT software.

5. Discussion

While consensus was not obtained in the nominal focus group about many aspects of critical thinking, emerging themes suggested critical thinking should be taught through the application of critical thinking skills to nursing practice problems; the use of specific appraisal tools for the critiique of web-based content; the use of face-to-face as well as online teaching, and by using interactive approaches for online teaching. These findings are consistent with existing literature on critical thinking and also suggest a strong disciplinary preference within the health disciplines, and particularly nursing, relating to the application of critical thinking to clinical problem-solving (11, 25-30). This suggested more emphasis on the cognitive skills related aspects of critical thinking rather than the affective elements (22, 31).

Nominal focus group data were used to inform modification of CCT content where this was indicated and feasible. Much of the focus group feedback aligned with existing CCT content which already includes several embedded critical appraisal tools, modified slightly in response to findings. Similarly, since the prototype version of CCT was developed as an assignment construction tool, Critiique contained high levels of interactivity as students would always be engaged in an assessment-related task. Other focus group feedback was addressed by adding content to Critiique, such as the addition of broad theoretical content related to critical thinking. Therefore the revised CCT included generic related to critical thinking as well as content specific to nursing and health care. Further interesting findings arising from the focus groups were comments about Gen Y related to ‘writing as they speak’ and their preference for instant and reactive online games, and for short videos over large amounts of online text. While the internet and the development of web-based resources have dramatically influenced the nature of contemporary educational design (32), it is important to note that when students use Critiique they will engage in an assessment task that requires lengthy critical analysis and synthesis of content. Therefore while Critiique is contemporary e-Learning software, it will not provide students with ‘learning by entertainment’ but will instead provide scaffolded support for critical thinking skills as students engage in a major, complex assessment task.

It is notable that some feedback was not amenable to representation by CCT. Focus group findings advocating the application of critical thinking to clinical problem-solving could potentially be addressed by including patient case studies in Critiique, thereby promoting active teaching strategies to develop critical thinking skills amongst nursing students (33). This would also achieve the use of active strategies to promote critical thinking advocated by The Australian university concerned already had case-based learning resources that included high quality, authentic, multimedia case studies, and so the logical step was to use Critiique as a tool to guide problem-solving when students use these case studies. A scaled-back version of CCT will also be developed for this purpose, combining the basic logical steps of scientific critical thinking with evidence-based practice to provide a practical, basic, decision support tool.

There were several limitations associated with the nominal group process such as low participant numbers, volunteer bias, the subjective nature of the data collected and difficulties obtaining a measure of consensus across broad subject areas. It is important to note that Critiique software is authorable so that ongoing modifications can be made as more staff and students use CCT. However, Critiique remains early in its development with large scale implementation with students planned in 2016. This implementation will be comprehensively evaluated over several years, with use of CCT likely to be extended to include the UK, and possibly Indonesia and Malaysia.

Critiique may be applicable not only across multiple topics in the undergraduate and postgraduate nursing curriculum, but also across all health sciences disciplines and indeed any professional practice discipline that draws upon web-based resources. The CCT project also has the potential to contribute to the development of higher education pedagogy and the ways in which Learning Management Systems are used.