The use of multiple-criteria decision-making theory to measure students’ perceptions of high-fidelity simulation

Abstract

Background

High-Fidelity Simulation (HFS) has great potential to improve decision-making in clinical practice. Previous studies have found HFS promotes self-confidence, but its effectiveness in clinical practice has not been established. The aim of this research is to establish if HFS facilitates learning that informs decision-making skills in clinical practice using MultipleCriteria DecisionMaking Theory (MCDMT).

Methods

The sample was 2nd year undergraduate pre-registration adult nursing students.

MCDMT was used to measure the students’ experience of HFS and how it developed their clinical decision-making skills. MCDMT requires characteristic measurements which for the learning experience were based on five factors that underpin successful learning, and for clinical decision-making, an analytical framework was used. The study used a repeated-measures design to take two measurements: the first one after the first simulation experience and the second one after clinical placement. Baseline measurements were obtained from academics. Data were analysed using the MCDMT tool.

Results

After their initial exposure to simulation learning, students reported that HFS provides a high-quality learning experience (87%) and supports all aspects of clinical decision-making (85%). Following clinical practice, the level of support for clinical decision-making remained at 85%, suggesting that students believe HFS promotes transferability of knowledge to the practice setting.

Conclusion

Overall, students report a high level of support for learning and developing clinical decision-making skills from HFS. However, there are no comparative data available from classroom teaching of similar content so it cannot be established if these results are due to HFS alone.

Introduction

Simulation is a creation of a set of problems or events that can be used to teach how to do something.¹ Within healthcare education, it is a term which encompasses a wide range of approaches to create real-life clinical tasks or scenarios in a learning environment.² Research into simulation use in healthcare education has been ongoing from the 1990s to measure its effectiveness in supporting the development of skills and knowledge for healthcare professionals to perform tasks and inform their clinical practice. To date, the evaluations of the effectiveness of High-Fidelity Simulation (HFS) has relied mainly on subjective measurements of individual participants perceptions of their abilities and experiences and not an objective measure of their actual knowledge or skill development.³ A major limitation of this is students who over estimate their skills and knowledge depth in relation to clinical practice.⁴ There is a recognised need to establish more objective measures for evaluating the impact of HFS on healthcare practitioner’s clinical skills and ultimately its impact on clinical service.²

This study aims to obtain an objective measurement of the effectiveness of HFS and its ability to support the development of skills clinical decision-making in undergraduate nursing students by using MultiplecCriteria Decision-Making Theory (MCDMT). The use of MCDMT as research methodology is a novel and innovative approach within healthcare education that will enable more accurate, reliable and valued added data on the efficacy of HFS to be assimilated. This tool has been adapted for the purpose of this study from an original tool used to measure customer satisfaction⁵ for details (see online supplementary material 1).

Background

The use of flight simulators was first developed in the 1920s by Edwin Link and has developed into fully computerised flight simulating cockpits which are believed to have had a significant impact on improving aviation safety.⁶ Healthcare educators have observed the effectiveness of simulation technology in aviation safety and sought to emulate their success. Simulation using manikins in healthcare education evolved in the 1950s, when a collaboration between two anaesthetists and Asmund Laerdal, a Norwegian toymaker, resulted in the development of a part-task trainer known as Resusci Annie for developing airway management and resuscitation skills, with the computerisation of manikins being developed from the 1980s.⁷

The fidelity of the simulation is the degree to which the experience represents realism.⁸ Fox-Robichaud and Nimmo⁹ suggest that the fidelity of simulation is ranging from those that support the development of specific skills and impart major facts (low) to a realistic clinical scenario experience that allows for clinical reasoning and decision-making, using realistic patient simulators or patient actors (high).

Literature review

The theory of simulation learning

Race¹⁰ promotes five factors that support students to learn successfully, intrinsic motivation or the ‘want to learn’ factor, extrinsic motivation or the recognition of the ‘need’ to learn, experiential learning or ‘learning by doing’, feedback and making sense of things. Learning is a process undertaken by students, for lecturers the challenge is to create an environment that facilitates learning to happen.¹⁰ Eraut¹¹ suggests that for students, who are studying for professional qualifications and are required to develop professional capability, higher education does not facilitate learning that promotes effective transfer of knowledge from the classroom to the workplace and therefore the workplace is the most effective environment to support learning. When workplace learning is the sole approach for healthcare professionals, the concern is that this approach to learning is simply based on the belief when students spend enough time being exposed to clinical practice they will eventually understand what to do.¹² These beliefs are supported by MacDowell¹³ and Bricker and Pardee¹⁴ who identify that learning in the workplace with acutely ill patients is not only stressful for students but also potentially unsafe for patients. Conversely, if students are only exposed to the safe environment of a lecture room with limited experience in the clinical environment they can only absorb the technicalities of knowledge without understanding the impact and complexity of the clinical environment.¹⁵ Lendahls and Oscarsson¹⁶ showed that midwifery students placed high value on simulation of challenging clinical scenarios that allow them to learn risk free. In the current climate of medical litigation Strump, Husman and Brem¹⁷ found that nurses who have low confidence in their skills and fear making mistakes avoid tasks and consequentially fail to provide adequate patient care. By allowing healthcare professionals to practice in simulated scenarios that replicate the realism of actual clinical scenarios introduced in a safe learning environment, it is possible to potentially improve safe levels of care for patients.¹⁸

Although, several studies^{15 19 20} suggest that simulation promotes self-confidence, critical thinking skills, skill performance, decision-making and problem-solving, a comparison of two groups of students by Secomb et al ²¹ found that there was no significant difference in the clinical decision-making ability of students following instruction by high-fidelity clinical simulation and computer-based simulations with no faculty instruction. These findings were supported by Maneval et al ²² and Goodstone et al ²³ who conducted studies which measured the effectiveness of HFS versus classroom-based activities and found no statistical significance in the critical thinking and clinical decision-making abilities of the participants.

Students’ perceptions of simulation education

Students enjoy simulation learning.¹³ Furthermore, Jeffries¹⁵ states that learners reported high levels of satisfaction following simulation activities. Feingold et al ²⁴ found that students believed simulation enhanced their learning and they valued its contribution to their education. Bambini et al ²⁵ and McCaughey and Traynor²⁶ identified student’s feelings of increased confidence and competence following simulation activities. Stroben et al ²⁷ demonstrated that HFS increased medical student’s self-efficacy and helped them to feel prepared for clinical practice after graduation. Sleeper and Thompson²⁸ and Lendahls and Oscarsson,¹⁶ showed that students believed being exposed to simulated scenarios had reduced their anxiety in relation to meeting similar scenarios in real practice. However, Walton et al ²⁹ found that simulation is itself a cause of anxiety for students, who suffered from performance anxiety and were worried about the equipment and opinions of their peers. This was supported by Deegan and Terry²⁰ where students reported emotional responses to the emergency scenarios. In a study by Liaw et al, ³⁰ students identified that despite the stress of the emergency situation being simulated they were able to remain calm, they felt properly prepared and believed feeling stress in simulated emergency scenarios is a true reflection of the feelings that would be invoked in clinical practice therefore learning how to deal with these normal responses is an important part of the learning experience.

Measurement of learning outcomes

Research into simulation use in healthcare education to measure its effectiveness in the ability of healthcare professionals to perform tasks and inform their clinical practice has relied mainly on individuals perceptions of their abilities and experiences.³ Liaw et al ⁴ suggest that simple knowledge tests and self-evaluations lack objectivity and rigour and cannot predict learners’ clinical ability. This is supported by Kardong-Edgren et al ¹² who identify the need for valid and reliable evaluation tools for students’ performance during simulation. Self-reporting studies have highlighted increased confidence as a major finding following simulation activity15 19 20 25 yet Liaw et al ⁴ identified a major concern of their study was students reporting much higher levels of confidence in their ability to manage a simulated activity than matched faculty assessment of their actual clinical performance and ability. Shinnick and Woo³¹ suggest that students should expect low scores of self-efficacy because to be over confident as a student may lead to a reluctance to think decisions through carefully or ask for help appropriately thus risking patient safety.³¹

Transfer of knowledge from the classroom to clinical practice

The ultimate aim of all professional education is to facilitate the development of knowledge and skills that inform clinical practice¹¹ and allow students to develop along the professional novice-expert trajectory.³² Reader³³ suggests simulation is a powerful tool for facilitating education because it allows students to use their memory of past experiences, known as episodic memory³⁴ to inform their practice. Bricker and Pardee¹⁶ noted that although students identified that their confidence in patient's assessment and clinical judgement skills increased following simulation experience, evidence that it translates into their clinical practice is purely anecdotal. A survey by Feingold et al ²⁴ reported that 100% of faculty staff believed learning facilitated with HFS would transfer to the workplace, whereas only 50% of the students believed their learning experience would transfer into clinical practice.

The study

The study took place in a Higher Education Institution in the North of England. The BSc (Hons) Adult Nursing programme was selected following the introduction of HFS as an educational tool to facilitate learning in relation to the care of acutely ill patients. Students’ response to this approach was positive and the students were requesting for HFS to be used more within their curriculum. To support this request, it was important to establish, through research, that HFS is a credible tool for facilitating clinical learning environments. We address the following research question:

Does utilising simulation technology facilitate professional capability in undergraduate student nurses?

This study aims are as follows:

1. Identify to what extent exposing students to simulated clinical scenarios informs their clinical decision-making skills in a practice setting.

2. Establish the quality of HFS in relation to knowledge transfer from the education setting into clinical practice.

Methods

Multiple Criteria Decision-Making Theory

MCDMT was used to establish the level of support from simulation education in relation to clinical decision-making. Two main areas (frameworks) for measurement were identified: clinical decision-making and the quality of HFS. MCDMT required each of the framework factors to be ranked in order of perceived importance.

Theme 1: clinical decision-making

The characteristics of clinical decision-making for the MCDMT are based on the analytical framework for clinical decision-making by Carroll and Johnson.³⁵

1. Recognition of situation

2. Formulation of explanation

3. Alternative generation of other explanations

4. Information: clarify choice and available evidence

5. Judgement or choice

6. Action

7. Feedback

Theme 2: quality of the HFS

Race’s¹⁰ five factors that underpin successful learning were used as the framework.

1. Wanting to learn

2. Needing to learn

3. Learning by doing

4. Learning through feedback

5. Making sense of things

Participants

Convenience sampling was used. Although this is acknowledged as a weak form of sampling, it ensures all potential participants are accessible by the researcher and have been exposed to simulation education.

Participants required for calculating the MCDMT weighting coefficients data were 13 experts (academics in adult nursing) and 160 novices, (second-year pre-registration nursing students) Novices were recruited from two consecutive cohorts of nursing students (n1=59, n2=101). The student cohorts were taught in groups of 15 for 2 hours.

The novices’ judgements of how well simulation allowed them to meet each part of the two frameworks were collected once for the quality framework and in two separate stages for clinical decision-making (the two separate stages of data collection are described fully under data collection) and repeated using two cohorts of students.

Research ethics

Research ethics approval of the study was granted by the University’s ethics committee.

Materials and equipment

The questionnaire did not ask for any demographic data. For calculation of the weighting coefficients the questionnaire asked for each part of each of the two frameworks to be ranked (1–7 for clinical decision-making and 1–5 for quality of HFS).

The study questionnaires which were used to measure the effectiveness of simulation learning employed an 11-point scale, with end points of 0 (simulation does not support the stated aspect of learning) and 10 (simulation supports this aspect of learning fully).

The study took place in a purpose-built simulation suite using Laerdal Simman technology and Scotia Medical Observation Training System (SMOTS). The topic for the simulation was recognition of an acutely ill patient.

Procedure

All respondents were advised that responses would be analysed anonymously.

Data collection for calculation of the weighting coefficients academics/experts

Before the teaching sessions, a questionnaire was distributed to all academics involved in teaching the BSc (Hons) Adult Nursing programme by email. They were requested to rank the importance of the characteristics for clinical decision-making and quality of HFS, and return their answers anonymously printed on paper or non-anonymously via email.

Students/novices

An information sheet requesting volunteers was given to all students by the researcher at the start of the simulation session. In a first questionnaire, students were requested to rank the importance of the characteristics for clinical decision-making and for the quality of HFS before exposure to the simulation class.

Data Collection for measuring the effectiveness of the simulation environment

Data for the study were collected from the students in two stages and across two cohorts of students.

Stage 1

Immediately following the 2 hours simulation class, all the students were given a second questionnaire sheet that asked them to rate how well they believed their experience of simulation teaching supported the factors that facilitate successful learning and clinical decision-making.

Stage 2

Following the taught weeks of the module where the students experience HFS, they had two clinical placements. One of these was an 8-week critical care placement, which potentially exposed them to clinical situations with acutely ill patients who had been simulated during the taught simulations sessions. On their return to university, they were given a final questionnaire printed on paper asking them to rate how well their experience of simulation teaching had supported their understanding of the seven characteristics to facilitate safe clinical decision-making in clinical practice.

Students were asked to complete all questionnaires within the classroom setting. They returned the completed questionnaires anonymously at the end of the session.

Data analysis

Processing the enquiry data

Kendall’s rank correlation was used to calculate the weighting coefficients for the quality characteristics. First agreement of the opinion among experts and among novices needed to be calculated. Then with agreement established, the judgements of all students for each characteristic gave an overall evaluation for the level of support with this particular characteristic of HFS.

Graphical presentation of effectiveness measurement results

The support measurement results are represented graphically with the weighting coefficients projected on the x-axis and the level of student support with each characteristic (in %) projected on the y-axis. Four areas of outcomes can then be defined (see figure 1).

Figure 1.

Possible outcomes for each rated characteristic of high-fidelity simulation as a learning tool.

According to MCDMT,⁵ an acceptable level of support is above 50%.

Results

Theme 1: Clinical decision-making

The results of Kendall correlation calculation for ranking the importance of the characteristics for clinical decision-making was 0.99 which demonstrated concordance of opinion and so enabled the weighting coefficients to be calculated. Table 1 shows the results for the calculation of weighting coefficients for clinical decision-making.

Table1.

Weighting coefficients for clinical decision-making

	Recognition of situation (y1)	Formulation of explanation(y2)	Alternative generation of other explanations (y3)	Information: clarify choice and available evidence (y4)	Judgement or choice (y5)	Action (y6)	Feedback (y7)
Weighting coefficients	0.26	0.13	0.11	0.18	0.14	0.14	0.02
Rank	1	5	6	2	3.5	3.5	7

Figure 2 shows the overall support from simulation learning for clinical decision-making.

Figure 2.

Level of support received by high-fidelity in relation to clinical decision-making.

The initial conclusions by students in both cohorts (denoted by blue diamond in figure 2) following exposure to HFS shows support levels for all characteristics of clinical decision-making in the range of 84%–92%. This suggests that all characteristics of clinical decision-making received strong support from HFS, according to the students. Recognition of situation this was considered by academics and students to be the most important characteristic in clinical decision-making. This had a high level of support, with both groups of students rating the level of support as 86%. Feedback was considered the least important characteristic and yet was scored the highest in terms of level of support, 88%–92%. The differences from the two cohorts in percentages for each characteristic were 0%–4% suggesting overall agreement in level of support from simulation.

Following clinical placement, the results from cohort 1 students (number of completed questionnaires reduced to 45) showed significant reduction ranging from 14% to 17% in the level of support they perceived from simulation education for each characteristic.

Students voluntarily added qualitative comments on their questionnaires about why they believed the level of support from HFS had not transferred to their clinical placement. Their comments identified that there was a significant length of time between being taught the theory in the simulation laboratory and actually being in the acute setting for clinical placement. For cohort 2 students (number of completed questionnaires reduced to 61), a change in the programme delivery meant this lapse in time did not occur and the level of support for the whole simulation stayed at 85%.

Theme 2: Quality of the HFS

The results of Kendall correlation calculation for ranking the importance of the factors that facilitate successful learning was 0.95 which demonstrated concordance of opinion and so enabled the weighting coefficients to be calculated.

Table 2 shows the results for the calculation of weighting coefficients for factors that facilitate successful learning.

Table 2.

Weighting coefficients for factors that facilitate successful learning

	Wanting to learn (y1)	Needing to learn (y2)	Learning by doing (y3)	Learning through feedback (y4)	Making sense of things (y5)
Weighting coefficients	0.35	0.15	0.24	0.09	0.17
Rank	1	4	2	5	3

Figure 3 shows the overall support from simulation learning for the factors that facilitate successful learning.

Figure 3.

Level of support received by HSF for successful learning.

Overall support for facilitating successful learning ranged from 87% to 93%, suggesting a high level of support for all factors from simulation.

Wanting to learn was ranked by academics and students to be the most important factor in successful learning. The rated level of support was from 89% to 92%. Learning through feedback was considered the least important factor and yet was scored the highest in terms of level of support, from 91% to 93%.

The differences between the two studies in percentages for each factor were 0%–4%, suggesting overall agreement in the level of support from simulation.

Discussion

Studies have demonstrated that HFS creates an environment that students enjoy¹³ and promotes satisfaction in learners.¹⁵ However, satisfaction is a subjective perception. To justify the resources needed to facilitate HFS, it needs to be objectively evaluated to establish a firm evidence base that it is a credible educational tool for healthcare education.² Using MCDMT allows the introduction of objectivity. Each quality characteristic can be characterised by quantitative and qualitative valuations. The qualitative valuation defines the level of importance of the quality characteristic. The quantitative valuation gives the level of support associated with this quality characteristic provided by HFS. The combination of these two valuations gives a complex evaluation of the level of support from HFS associated with the chosen quality characteristic.⁵ By using this tool to measure the effectiveness of HFS in undergraduate student nurse education, the results show that for the quality of learning framework effectiveness ranged from 87% to 93%. For supporting clinical decision-making effectiveness ranged from 84% to 92%. This puts all results in the area of high effectiveness, which suggests that HFS is a credible educational tool.

Key themes in relation to existing literature

The theory of simulation learning

Simulation allows student nurses to practise in simulated scenarios that replicate the realism of actual clinical scenarios and so create an experiential learning environment.³⁶ This can also be referred to as active experience or what Race¹⁰ refers to simplify as ‘learning by doing’. Light and Cox³⁷ advise not all experiences and environments achieve what is perceived as experiential learning therefore the facilitator needs to ensure the correct parameters are addressed which include the learning experience and reflection. The effectiveness of HFS in achieving learning by doing and then supporting effective feedback suggests that it meets these requirements and is an effective tool to support experiential learning.

In contrast to wanting to learn, Race¹⁰ suggests that needing to learn is the extrinsic motivation for students. Teachers have a duty to ensure students understand their needs and help them to develop a sense of ownership in relation to their learning. Needing to learn, was considered to be third in ranking for this study but yielded a high level of support from the simulation experience.

Effective feedback is considered an essential component of simulation education.²⁴ It is also an essential component of Kolb’s experiential learning theory.³⁶ It was ranked the least important component of learning yet it was the component most highly supported by the simulation experience for both groups.

Student’s perceptions of simulation education

Simulation creates an environment that students enjoy¹⁸ and promotes satisfaction in learners.¹⁸ Race¹⁰ suggests that wanting to learn can also be considered as intrinsic motivation that students may be unaware of and teachers need to initiate. Wanting to learn was ranked the top factor in successful learning. The rated level of support was from 89% to 92% suggesting simulation supported this aspect of learning fully.

Race¹⁰ suggests making sense of things is the most important of his five stages and was ranked the second highest factor. However, Race¹⁰ also suggests that effective feedback is what helps students to make sense of the learning experience. Making sense of the situation could be linked to the clinical decisions made as part of the learning. Support for making sense of things was the lowest scoring factor for each group from 87% to 90%, however, this still indicates high levels of support for the learning outcomes from HFS.

Transfer of knowledge from the classroom to clinical practice

The study aimed to measure students’ perceptions of how HFS education facilitated learning in clinical practice. Therefore, the second stage of data collection took place after students had experienced clinical placements in acute care and collected data on clinical decision-making only. This gap in data collection introduced a change in the number of completed questionnaires per cohort. For students in cohort 1 (n=45), there was a long time delay between the delivery of the taught component and this exposure to acute clinical practice. The results from this analysis showed level of support for clinical decision-making fell from 85% to 70%. This suggested initially that transferability of knowledge from HFS to the workplace was limited. For the cohort 2 students (n=61), this lapse in time did not occur and level of support for the whole simulation stayed at 85%. This suggests that HFS can be an appropriate tool for transferring knowledge from the classroom to the clinical setting when the taught sessions are delivered in a timely manner for transferring the knowledge to practice.

Conclusion and recommendations

Overall, the levels of support from simulation on the education experiences of undergraduate student’s nurses are high. However, there is no comparative data available from classroom teaching of similar content so it cannot be established if these levels are due to simulation alone. Further studies need to be developed to objectively compare simulation teaching with traditional classroom-based activities.

In line with much of the previous research into simulation use in healthcare education, this study has used a tool, which although tries to balance the perceptions of students with the opinions of experts, ultimately uses a self-reporting tool to measure its effectiveness. There is still a need for an objective study of student’s clinical performance following simulation education in the workplace.