Predictors of Knowledge Gains Using Simulation in the Education of Prelicensure Nursing Students

Abstract

Gains in knowledge and self-efficacy using human patient simulation (HPS) in the education of prelicensure nursing students have been reported. However, the predictors of improved learning outcomes using this teaching methodology are not known. Using a two-group (participated in HPS, did not participate in HPS), repeated-measures, experimental design, we examined the predictors of higher scores on a Knowledge Questionnaire in 162 students (age = 25.7 ± 6.6, gender = 85.5% female) from four prelicensure cohorts at three nursing schools. Statistical analysis consisted of t-tests, ANOVA and stepwise logistic regression. Covariates included age, gender, learning style, baseline critical thinking, baseline self-efficacy, group membership (control or experimental), and school. Membership in the experimental group was the only statistically significant independent predictor (P <. 001) of knowledge gains among the covariates entered into the regression analysis. Members of the control group were two times less likely than those in the experimental group to be in the higher scored group (P <. 001), yet this changed once the control group participated in HPS. Our findings show that HPS can independently improve test scores. This study provides evidence that HPS; is an effective teaching methodology for prelicensure nursing students regardless of age, learning style, or critical thinking ability.

Human patient simulation (HPS) is a time- and cost-intensive teaching modality that consists of both hands-on experience with a lifelike manikin and a debriefing session. The impact of HPS on gains in knowledge is uncertain in prelicensure nursing students because previous studies of in these subjects have been primarily descriptive and subjective. However, nursing faculty emphasize gains in clinical knowledge in their students (American Association of Colleges of Nursing, 2008; Innovation in nursing education, 2004). Although such knowledge gains with HPS have been demonstrated in basic and advanced life support training (Monsieurs, De Regge, Vogels, & Calle, 2005; Palmisano, Akingbola, Moler, & Custer, 1994; Verplancke et al., 2008), increases in other forms of clinical knowledge with the use of HPS have rarely been reported.

Background

There is a paucity of objective, quantitative research on knowledge acquisition in prelicensure nursing students using HPS. A literature review on HPS by Ravert (2002) revealed a positive effect on knowledge and skill acquisition in 76% of the cases, but none of these were done on prelicensure nursing students. Since Ravert’s review of the literature, three published studies have looked at the effect of simulation on knowledge gain specifically with prelicensure nursing students. Despite positive results in the studies, neither investigated if HPS was an independent predictor of those gains (Brannan, White, & Bezanson, 2008; Hoffman, O’Donnell, & Kim, 2007; Howard, 2007; Jeffries & Rizzolo, 2006; Shinnick, Woo, Horwich, & Steadman, 2011).

Thus, although many institutions use HPS as a teaching method in prelicensure nursing education, there are no published multivariate analyses that have determined if simulation is an independent predictor of greater knowledge gains when compared with those who have not participated in simulation. Identifying whether HPS or other factors objectively induce gains in knowledge could enable faculty to maximize those variables and optimize the learning experience of simulation.

Self-efficacy is a factor that is widely believed to increased knowledge (and believed to be increased in HPS). Self-efficacy is a person’s degree of confidence in performing a desired action or skill and has been measured using researcher-developed Likert scale instruments. (Bambini, Washburn, & Perkins, 2009; Goldenberg, Andrusyszyn, & Iwasiw, 2005; Gordon & Buckley, 2009; Jeffries & Rizzolo, 2006; Ravert, 2004; Scherer, Bruce, & Runkawatt, 2007; Wagner, Bear, & Sander, 2009). Although gains in self-efficacy were found using HPS in these studies, it has not been linked to gains in knowledge or been investigated as a predictor for greater gains in clinical understanding.

Despite the lack of research in this area, there is a widespread desire to incorporate HPS into nursing curriculums (Bandali, Parker, Mummery, & Preece, 2008; Seibert, Guthrie, & Adamo, 2004; Wilford & Doyle, 2006). This is a problem that both educators and clinicians should take seriously, as the commitment to changing nursing education with the incorporation of HPS is not only a financial one but a time-intensive one as well. Many institutions already have the problem of underutilization of purchased equipment (King, Moseley, Hindenlang, & Kuritz, 2008), most likely related to the uncertainty of the efficacy of HPS to improve knowledge.

It is clear that advancement in HPS technology use has exceeded the evidence to support its effectiveness. As acquisition of knowledge is fundamental in nursing education, this research sought to identify whether HPS would be an independent predictor of heart failure (HF) knowledge gains in prelicensure nursing students.

Specific Aim and Hypotheses

The specific aim of this study was to determine predictors of higher scores on the Heart Failure Knowledge Questionnaire during an HPS experience. Study hypotheses were (a) participation in HPS of a common adult HF clinical situation is an independent predictor of HF clinical knowledge; (b) self-efficacy (as measured by the Self-Efficacy for Nursing Skills Evaluation Tool) after HPS is an independent predictor of HF clinical knowledge; (c) self-efficacy (as measured by the Self-Efficacy for Nursing Skills Evaluation Tool) before HPS is a negative predictor of HF clinical knowledge; (d) learning style (as measured by the Kolb Learning Style Inventory) is an independent predictor of HF clinical knowledge in those participating in an HPS; (e) disposition toward critical thinking and critical thinking ability (as measured by the California Critical Thinking Disposition and the Health Sciences Reasoning Test, respectively) are independent predictors of HF clinical knowledge in those participating in an HPS; and (f) age and gender (as measured in the demographic questionnaire) are not independent predictors of HF knowledge in those participating in an HPS.

Method

Study Design

This study used a two-group (students who have an HPS experience on HF compared with students who do not have an HPS experience on HF), repeated-measures, experimental design.

Sample

A convenience sample of four cohorts (N = 162) of prelicensure nursing students were recruited from three Schools of Nursing for the study. All schools used the same simulation equipment (Sim Man Laerdal Medical Corp., Wappinger Fall’s, NY), yet only one cohort had any experience with simulation. Institutional review board approval was obtained from all three schools prior to the study. Power analyses indicated that a sample size of 128 subjects would allow detection of moderate (0.25) effect sizes on a paired t test and a one way analysis of variance at a P value of. 05 and power of 0. 80 and that a sample of 98 subjects would be sufficient to detect a moderate effect size (0.15) on a multivariate logistic regression of six covariates with alpha of. 05 and power of 0.80. Inclusion criteria were students in the same course at each school that had successfully completed instruction in care of patients with HF. This point in the prelicensure curriculum is the standard equivalent of a Medical Surgical Course, Level III, traditionally taken in the third year of a 4-year nursing program. Exclusion criteria were students who either had HF or had family members with HF. Students were randomly assigned by blocks (all students for the day at that school were assigned to the same group) to experimental (participated in HPS) or control (no HPS) groups. To ensure an equal learning opportunity for all students, the control group participated in simulation after the posttest was completed.

Scenario Development

The clinical situation of acute decompensated HF was chosen to be the focus of the simulation. Simulation of patients with HF in the education of nurses in acute care is important because HF is the most common hospital discharge diagnosis in the United States in patients 65 years and older (Schocken et al., 2008). In addition, there are more than 6 million people with HF, and more than 550,000 are newly diagnosed each year in the United States (Schocken et al., 2008). Thus, patients with HF can be found in many medical units of hospitals. This high incidence of HF makes simulating scenarios for patients with HF important components in the clinical training of nurses at all levels.

Three simulation scenarios of clinical cases of acute decompensated HF were created for this study. They were identical to each other in design with the exception of the patient history and gender. Parallel simulations were developed to decrease cross talk between groups and scenario predictability. Validity for HF scenario accuracy was done by three experts in HF management (two doctorally prepared nurses with HF expertise and one physician from an HF clinic) with 100% agreement.

The study scenarios were designed to elicit basic nursing responses such as elevating the head of the bed for a dyspneic patient, applying oxygen as appropriate, choosing the priority medication from physician’s orders, and monitoring appropriate electrolytes in a patient receiving a diuretic. Simulation objectives, which are usually given to students prior to the experience, could not be given in this case as the subject matter would have been revealed.

Data Collection Instruments

The students were studied in groups of five at the same point in their curriculum using an HF simulation and several assessments before simulation (pretest). Students were asked to complete three online questionnaires prior to their assigned simulation day. These were the Kolb Learning Style Inventory (a questionnaire consisting of 12 sentence-completion items; Kolb, 1999), the California Critical Thinking Disposition Inventory (a 75-item critical thinking inventory tool; Facione & Facione, 1996), and the Health Sciences Reasoning Test (a 34-item multiple choice instrument that measures core critical thinking skills; Facione & Facione, 1996). Each of these assessments is well established and has been used in the assessment of health care students. These results were later entered into the bivariate analyses prior to the multiple logistic regression model determination. The posttest was given to the experimental group after the completion of the HPS, whereas the control group took this assessment prior to their HPS experience.

The HF clinical knowledge pretest and posttests, developed by the investigator, focused on the symptom management of a patient with HF. Each version of the HF clinical knowledge test was different but considered parallel to the others (Table 1). The questions did not mention HF by name so the participant was blinded to the topic of the simulation. However, the questions focus on desired nursing interventions for common issues associated with HF.

Table 1.

Examples of an HF Clinical Knowledge Question on Parallel Tests

Knowledge pretest	Knowledge Posttest 1	Knowledge Posttest 2
The reason Harold has crackles in his lungs is: Fluid volume overload Increased pressure in the pulmonary vasculature An upper respiratory infection Preexisting pulmonary edema	Harold’s chest X-ray reveals pneumonia. How can the nurse determine the cause of his crackles in the lungs? Use percussion to assess the lungs Check for jugular venous distension Base assessment on his history of HF Check for weight gain	The reason Harold has crackles in his lungs is: Fluid volume overload Drinking too much An upper respiratory infection Preexisting pulmonary edema

Content validation of the HF clinical knowledge tests was done by three experts in the nursing care of patients with HF and by one cardiologist who practices at a large HF specialty clinic. Each version of the HF clinical knowledge test had 100% agreement on the content by the panel of judges for this study.

The HF clinical knowledge tests were scored by via Scantron. Item analysis and test scores were computerized and entered into an Excel file, which was then uploaded into SPSS version 16.0 (SPSS Inc., Chicago, IL). The HF clinical knowledge test consisted of 12 items (maximum score possible was 12 points). Scantron scores were entered into the database as a raw score of the number of answers that were correct.

For the purpose of the multivariate analysis section of this study, knowledge scores for the pretest and posttest scores on the HF clinical knowledge test were transformed into bivariate data (“good” vs. “poor”) to run this analysis. Consultation with nurse educators led to the decision that a score greater than 80% (10 or more answers correct out of the 12 questions) were categorized as a good score on the knowledge test and a score of less than 80% (9 or less questions correct) as a poor score.

In the pilot study of this project, a negative correlation was found between student self-efficacy and knowledge on the pretest (r = −.181, P = .022), which indicated that students were coming to the HPS experience overconfident. This finding led to the inquiry of the predictability of self-efficacy to knowledge.

The Self-Efficacy for Nursing Skills Evaluation Tool (a 12-item Likert-type questionnaire evaluating a student’s confidence or self-efficacy in care of a pulmonary patient; Ravert, 2004) was edited with the original author’s permission. It included items such as “self-efficacy in prioritizing doctor’s orders,” “self-efficacy in the management of postop pulmonary complications,” and “self-efficacy in taking vital signs.” This assessment was given to participants with the Knowledge Questionnaires at the pretest and posttest time points. Reliability of this instrument had a coefficient alpha of. 87 (Ravert, 2004).

Data on age, gender, school affiliation, and HF experience were completed as part of a demographic questionnaire at the time of the posttest.

Data Collection Protocol

“Standard Care” for all participants included the Med–Surg III course at their school and any accompanying clinical time in the same quarter. A two-day data collection interval for this study was scheduled at each site within 3 weeks of the HF lecture.

A coin toss determined the study day to be experimental or control with the subsequent day being the reverse. Participants in groups of five rotated together through testing, whereas simulations were done one-on-one using a random numbers table to determine scenario selection. Debriefing was done in groups of five.

Both the experimental and control groups completed the pretest assessments prior to simulation. Following HPS, the experimental group took the posttest. However, the control group took the posttest before the simulation. Once this assessment was completed, the control group then participated in the simulation and took another posttest (Posttest 2). The study protocol is depicted in Figure 1.

Figure 1.

Study protocol depicting randomization, experimental and control groups, testing completed, and the crossover of the control group to simulation.

With the exception of the online assessments, all other assessments were completed via Scantron. The item analysis and test scores were computerized and loaded into an Excel file, which was then uploaded into SPSS version 16.0 (SPSS Inc.).

Statistical Analyses

Of the original multiple variables collected in the study (Table 2), the covariates included in the model were those with a P <. 05 on the bivariate analyses for good versus poor HF test scores (dependent variable). The six covariates used in the multivariate analysis were gender, school, baseline self-efficacy in prioritizing doctor’s orders, baseline self-efficacy in the management of postoperative pulmonary complications, baseline self-efficacy taking vital signs, and group membership (control or experimental).

Table 2.

Univariate and Multivariate Analysis: Predictors of Good Knowledge (N = 162)

Variable	M(SD)	Bivariate P	Multivariate P	Exp (B)	B
Age	25.71 ± 6.62	.20	–	–	–
Gender	1.88 ± .330	.041	.189	–	–
Employed as nurse helper	0.59 ± 1.43	.853	–	–	–
Ethnicity	2.99 ± 1.24	.177	–	–	–
Order of attending simulation	5.97 ± 3.62	.444	–	–	–
School	2.61 ± 1.09	.009	.095	–	–
SE vital signs	3.96 ± 0.759	.079	.482	–	–
SE prioritizing physician orders	2.58 ± 0.87	.036	.358	–	–
SE managing HF	1.99 ± 0.91	.765	–	–	–
SE monitor O2 saturation	3.67± 1.01	.527	–	–	–
SE apply O2 cannula	3.34 ± 1.16	.509	–	–	–
SE apply O2 mask	3.20 ± 1.45	.443	–	–	–
SE fluid overload	2.63 ± 0.97	.551	–	–	–
SE manage pulmonary infection	1.99 ± 0.90	.393	–	–	–
SE postop complications	1.90 ± 0.90	.253	.542	–	–
SE signs respiratory decompensation	2.06 ± 0.94	.786	–	–	–
Learning style	1.93 ± 1.08	.980	–	–	–
Health science reasoning test	21.72 ± 4.68	.030	–	–	–
Group (control or experimental)	1.56 ± 0.50	.000	.000*	.137	−1.986

Note. SE = self-efficacy

^*Statistical significance at P <.05

Multivariate analysis was performed by stepwise logistic regression using SPSS version 16.0 (SPSS Inc.) statistical package to determine predictors of higher knowledge scores. As stated, the model included variables found to be significant on bivariate analyses (P <. 05), with the dependent variable being good versus poor score on the HF clinical knowledge tests.

Results

A total of 162 prelicensure nursing students from three schools of nursing completed the study. The control and experimental groups (n = 72 and 90, respectively) were not equal in size due to the variability in prelicensure cohort sizes at the data collection sites. However, there were no statistically significant differences in age, gender (Table 3), or baseline knowledge scores (pretest, Figure 2) between groups, which supports the contention that they were equivalent at baseline.

Table 3.

Demographic Data on Study Sample

	Control	Experimental	P
n	72	90	–
Age (years)	27 ± 7	25 ± 6	.43
Gender, females (%)	67 (93.1)	74 (85.5)	.06

Figure 2.

The columns represent the scores on the HF clinical knowledge pretest between the control and experimental groups. Using paired t tests, no significant differences in mean knowledge scores between the groups were seen at baseline (pretest).

The HF clinical knowledge test percentiles on the pretest ranged from 41% to 100% (raw scores = 5–12) with a mean of 64% (eight correct); the Posttest 1 test percentiles 15%–89% (raw scores = 2–11) with a mean of 58% (seven correct); and the Posttest 2 percentiles of 33%–100% (raw scores = 4–12) with a mean of 71% (nine correct).

Predictors of Knowledge in HPS

A logistic regression with the transformed variable of HF knowledge (good or poor) with the above-mentioned covariates (gender, school, baseline self-efficacy in prioritizing doctor’s orders, baseline self-efficacy in the management of postoperative pulmonary complications, baseline self-efficacy taking vital signs, and group [control or experimental]) revealed group membership (control or experimental) as the only independent predictor of a good score (P < .01; Table 3).

Control group members did not participate in HPS at the first posttest time point and were found to be twice as likely to score in the poor knowledge group (knowledge scores <80%) in comparison to the experimental group who did participate in HPS. Consequently, the hypothesis of HPS participation predicting HF clinical knowledge was supported. The control group mean knowledge scores at baseline (64.02% ± 13.12) did improve to after they participated in HPS as measured by Posttest 2 (69.51% ± 13.34) (Figure 3). Mean HF clinical knowledge scores improved after HPS, but the goal of a good score (>80%) was achieved by only 25% of the participants in either group.

Figure 3.

Knowledge scores on the pretest and posttest between groups depicting score differences at the two time points. Mean knowledge scores for both groups were the same prior to HPS (P < .001); significantly different at Posttest I (P < .001), where the experimental group had HPS and the control group had not; and Posttest 2 for the control group revealed gains in knowledge above baseline after HPS (P < .001) with the control group.

Self-efficacy scores after HPS in the multivariate analysis were not found to be statistically significant for predictability of HF clinical knowledge. Therefore, the hypothesis of self-efficacy scores being a predictor for HF clinical knowledge is not supported. Finally, self-efficacy scores before HPS were also not found to be negative predictors of HF clinical knowledge, so this hypothesis was also not supported.

Discussion

The finding of control group membership being a negative predictor for gains in knowledge in this investigation further reinforces the value of HPS. Using a randomized, experimental design, this study has demonstrated that prelicensure nursing students participating in HPS clearly have higher knowledge scores on an HF clinical knowledge test. In addition, it has demonstrated student disadvantages for those who do not participate in HPS when compared with those who have. Although mean knowledge scores increased after the HPS experience, only 25% of the participants had scores deemed to be good (>80%) as defined by nursing educators. This interesting finding could be related to factors not accounted for in this study, such as student grades or knowledge retention. This finding may be an indication that HPS should not be a stand-alone form of teaching. Although all students attended lectures that included HF within 2 weeks of HPS, this study did not provide the participants with separate assigned reading or information about the HPS topic prior to the experience. Adequate preparation for the simulation in this manner may increase student knowledge and should be researched.

Because of a preponderance of females in the sample, one may surmise that gender played a role in the findings. However, a logistic regression run with male gender alone revealed the same results. In addition to gender not being a predictor for higher knowledge scores, this study has also eliminated suspected predictor factors such as age, school, or self-efficacy scores in multiple areas. In addition, it clarifies for educators that students with diverse learning styles or critical thinking ability can learn in a simulation experience, as learning style and critical thinking were not found to be predictable for higher knowledge scores. This may make HPS attractive to nursing faculty members, as students in a course are typically composed of learners with many different strengths and weaknesses. In the midst of clinical sites becoming more difficult to establish, faculty can be more confident of HPS as a learning methodology that has value.

Study Limitations

Efforts were made to minimize study limitations, although some were unavoidable. For the lecture component of the course at each research site, different, resident faculty gave their usual cardiac lecture, which included HF. To eliminate study bias, the lecture at the home site of the principal researcher was done by a faculty not involved with the study. In addition, the emphasis on HF may have varied from school to school as it was part of a larger, cardiac topic lecture. However, the randomization of subjects for this study support the contention that variations in school teaching and content regarding HF was not a significant predictor of HF knowledge in this project.

Students may have had different and unequal clinical experiences in HF. Attempts were made to control this by scheduling the study within 2 weeks of the lecture at each site and by randomization to group assignment. As the study was done at each site over a 2-day period, contamination of the study content may have occurred with students discussing content of the simulation among themselves (cross talk) despite confidentiality agreements. This was only apparent at the end of the last study day at one site.

Previous simulation experiences differed slightly between the groups as one of the four study cohort had experienced simulation in other courses. This cohort seemed more comfortable in the simulation and did not need as much prompting as the others though they did not score die highest on the knowledge scores. All students were oriented to the HPS manikin and the environment prior to the simulation to decrease the effect of this limitation.

Implications

This study has demonstrated simulation to be an effective learning modality for a clinical situation in HF in prelicensure nursing students. It has clarified the variables that matter (participating in HPS) for knowledge gains in this type of teaching methodology and eliminated many that do not. Educators can now feel more confident in this new learning strategy as knowledge gains occur with HPS for different ages of learners and without preference for learning style. It also clearly identifies value to students who may not be strong critical thinkers. This is great news for educators as many have already invested in expensive simulation devices and programs as well as for others who were waiting for evidence of HPS value. However, further study is needed to determine optimal preparation and simulation exposure time necessary for improved knowledge scores.