The effect of clinical simulation assessment on stress and anxiety measures in emergency care students

Abstract

Background

Clinical simulation has become widespread as a training and assessment tool across a range of health professions, including emergency care. As with any form of assessment, simulations may be associated with stress and anxiety (“distress”) which may have a negative effect on student performance if demands required by the simulation outweigh the available resources. This study aimed to assess the effect of participation by students in an emergency care simulation on an objective measure of stress and a subjective measure of anxiety.

Methods

Heart rate variability (HRV) and scores from a validated state anxiety instrument (the State-Trait Anxiety Inventory) were assessed in 36 emergency medical care students participating in scheduled simulation assessments. Data recorded during a resting control period were used for comparison.

Results

HRV variables showed changes in the simulation assessment group suggesting decreased variability and parasympathetic withdrawal, however these were not significantly different to control. Heart rate in the simulation assessment group increased significantly (73.5/min vs. 107.3/min, p < 0.001). State anxiety scores increased significantly both before (33.5 vs. 49.1, p < 0.001) and after (33.5 vs. 60, p < 0.001) the simulation assessment, compared to control. No linear relationship was found between any HRV variables and anxiety scores.

Conclusion

Participating in an emergency care simulation assessment significantly elevated levels of anxiety in a group of 36 students, however an objective measure of stress did not identify changes significantly different to those at rest, with the exception of heart rate. The high levels of anxiety documented before and after simulation assessments may have a negative effect on performance and require further investigation.

African relevance

• •Clinical simulation is used for both training and assessment of emergency care students in Africa
• •No data are currently available on the levels of stress and anxiety experienced by African emergency care students during simulation assessment
• •African emergency care students' levels of anxiety were substantial before and after a simulation assessment, possibly affecting performance

Introduction

Clinical simulation has become widespread in the training of a range of emergency care professionals [[1], [2], [3], [4]]. The use of clinical simulation has extended to prehospital emergency care, for both training and assessment [[5], [6], [7]]. Simulation assessments allow assessors to expose a cohort of students to the same reproducible cases in order to produce more reliable assessments. Simulated cases can also be tailored to the specific needs of a program, module or learning unit and allow assessors to focus on the assessment of specific learning outcomes. Importantly, simulation assessment also eliminates the practical and ethical complications of assessing student performance using real patients [8].

As with all forms of assessment, simulation assessments may be associated with stress and anxiety experienced by students [9], however simulation assessment exposes students to unique stressors. Being required to make patient care decisions and perform procedures in real time places stress on students, especially when the simulated cases involve emergencies. Unlike written assessments, where there is no immediate feedback, simulation assessments provide patient responses in real time which may further complicate management of the case, or suggest to students that they have made an incorrect decision or action – adding further pressure to be dealt with. In some cases, depending on local practices and available facilities, students may be aware of assessors observing them – another factor which, like the others above, tends to make simulation assessments uniquely stressful.

Tomaka et al. propose possible stress responses in such a condition based on a two-stage situational appraisal process [10]. The first stage involves an appraisal of the demands perceived to be required to deal with the situation, while the second stage involves an appraisal of personal and environmental resources available to meet the demands. Resources perceived as being adequate to meet demands give rise to a situational assessment that a “challenge” is present, while a perceived deficiency of resources gives rise to an assessment of “distress”, typically resulting in varying degrees of anxiety. While what is perceived as a challenge may have positive effects on performance, states of distress may negatively affect aspects of performance such as memory and decision-making [10,11].

By its nature, because it involves often critically ill or injured patients, emergency care – and simulated emergency care – is stressful. However, it is possible that the nature of a simulation assessment may add other sources of stress, or distress, to student performance which may have a negative impact. Few studies have measured stress or anxiety during emergency care simulation assessment, particularly those simulating prehospital emergency care [[12], [13], [14], [15], [16]]. The aim of this study was to assess the effect of participation by students in a clinical emergency care simulation assessment on an objective physiological measure of stress (heart rate variability) and a subjective measure of anxiety (the state component of the State-Trait Anxiety Inventory score) and to assess whether a linear relationship existed between these measures.

Methods

This research used a prospective, repeated measures experimental design with one independent variable, exposure to a simulation assessment and no exposure (control), and five dependent variables: four Heart Rate Variability (HRV)-derived variables (explained under Data collection) and State-Trait Anxiety Inventory scores. It was hypothesized that HRV-derived variables under control conditions would indicate a significantly lower measure of stress, that anxiety scores under control conditions would be significantly lower and the a significant linear relationship would be established between HRV-derived data and anxiety scores under simulation assessment and control conditions.

The population comprised of students from all four academic years of study in the Bachelor of Health Sciences in Emergency Medical Care program offered by the Department of Emergency Medical Care at the University of Johannesburg. The Bachelor of Health Sciences in Emergency Medical Care is a four-year professional degree program leading to registration as an Emergency Care Practitioner with the Health Professions Council of South Africa. Emergency Care Practitioners are the top tier of Advanced Life Support prehospital emergency care personnel in South Africa. Clinical simulation is used extensively throughout all four years of the program for both training and assessment.

The sample was comprised of those students who consented to participate on one of the predetermined emergency care simulation assessment dates during 2018 and were present for both simulation assessment and control data collection, which occurred on different days.

The research was conducted in the Faculty of Health Sciences Simulation Laboratory before, during and after emergency care simulation assessments. The assessments generated a mark that contributed towards each student's overall subject mark. Students in first through to third year were assessed on a 15–20 min adult emergency care case while students in fourth year were assessed on a 20–30 min pediatric emergency care case. All of the cases involved students interacting with the simulator and performing a range of diagnostic and therapeutic procedures, however none of the cases involved strenuous physical activity.

In all assessments three external assessors observed and rated each student's performance using a rubric. For first and second year assessments the assessors were situated in the same room as the students and were thus visible to them while for third and fourth year assessments live video footage recorded from two viewing angles was transmitted to and viewed by assessors in a separate room. A prompter was also present in all of the simulation assessments, to provide clinical information to students during the assessment that was not available from the simulator.

Data collection

Heart rate variability

All students were requested to avoid caffeine-containing substances and exercise before data collection, and were screened for medications with known autonomic effects or side-effects. Shortly before each simulation, a heart rate monitor (Actiheart, CamNtech, Cambridge, United Kingdom) was attached to each participating student using two adhesive disposable cardiac monitoring electrodes. The heart rate monitor remained in place and recorded the student's electrocardiogram throughout the simulation assessment. Immediately after the simulation assessment ended, the heart rate monitor was removed and the recorded data were immediately downloaded to a notebook computer using a software application (Actiheart Analytical Software, CamNtech, Cambridge, United Kingdom). A control recording of cardiac activity was made on a separate occasion, on a day when each student was not scheduled for any other assessment. On this day, students were also requested to avoid caffeine-containing substances and exercise before data collection. For the control recording, the heart rate monitor was attached in the same way that it was for the simulation assessment recordings and students sat quietly in an office on their own for the duration of the 15-minute control recording.

After collection of all electrocardiographic data for all participants, the recorded data were imported into a heart rate variability analysis application (Kubios HRV Standard, Version 3.1, Kubios Oy, Finland) which was used to generate heart rate variability reports for each student. The first 15 min of each recording was used for HRV analysis. The following variables were extracted from each report and entered into an electronic spread sheet application: (i) standard deviation of all NN intervals (SDNN), (ii) square root of the mean of the sum of the squares of differences between adjacent NN intervals (RMSSD), (iii) the high-frequency spectral component (HF) and mean heart rate.

Anxiety

Anxiety was measured using a validated anxiety measurement tool, the State-Trait Anxiety Inventory (STAI) for Adults (Form Y) [17]. The STAI is a 40-item questionnaire with 20 items devoted to state anxiety (i.e. at the present moment) and 20 items devoted to trait anxiety (i.e. longer term relatively stable differences in personal anxiety). Each item is in the form of a statement to which participants select their agreement on a four-point scale. Students were requested to complete the STAI 5–10 min before their simulation assessment started, and before attachment of the heart rate monitor as described above. Immediately after the simulation assessment, after the heart rate monitor had been removed, students were again requested to complete the STAI. A control measure of anxiety was recorded on the same day as the control recording of cardiac activity (described above) with each student completing the STAI after their control cardiac activity recording was completed. Both state and trait STAI scores were calculated, however only the state component is reported here as a reflection of anxiety experienced at the time of the simulation assessment and during control data collection.

Descriptive data are presented as medians and interquartile range as three data groups out of 11 were found to be non-normally distributed. (One-sample Kolmogorov-Smirnov tests, performed on each group of data). Differences between simulation assessment and control HRV variables and STAI scores were assessed with paired samples t-tests (if normal) or Wilcoxon matched-pair signed-rank tests (if non-normal). Correlation between HRV variables and STAI scores in simulation assessment and control groups were assessed by calculating Pearson correlation coefficients. For all tests, p < 0.05 was considered significant. IBM SPSS Statistics (version 25, IBM Corporation, New York, USA) was used for all data analysis.

This research was ethically approved by the Faculty of Health Sciences Research Ethics Committee at the University of Johannesburg (ethical clearance REC-01-80-2017).

Results

A total of 66 students consented to participate over eight different days of simulation assessments and data collection. Of this group, 36 students (55%) returned after the simulation assessments for control data collection as described above. Characteristics of the sample of 36 students with simulation assessment and control data are summarized in Table 1:

Table 1.

Student characteristics (n = 36).

	n (%)
Academic year
First	11 (31)
Second	4 (11)
Third	8 (22)
Fourth	13 (36)
Gender
Male	22 (61)
Female	14 (39)

	Median (IQR)
Age (years)1	21 (3)

IQR = interquartile range, 1 = age was not determined for one participant, median and IQR are calculated from n = 35.

Descriptive HRV data and STAI scores are shown in Table 2 for both the simulation assessment and control data sets. Total power (SDNN) was reduced in the simulation assessment group, as were HRV variables associated with parasympathetic activity (RMSSD and HF). Heart rate, on the other hand, was increased by 37% in the simulation assessment group. Mean STAI scores for participants during control data collection were below the male (lower) reference range for college students as a population (36) [17]. The mean pre-simulation STAI score increased by 38% compared to control and the mean post-simulation STAI score increased from 33.5 to 60 in the control group. Dispersion of HRV variable observations and STAI scores was similar in both simulation assessment and control data sets.

Table 2.

Descriptive data (n = 36).

Group	Variable	Median	Interquartile range
Simulation assessment	SDNN (ms)	75.1	47.1
	RMSSD (ms)	82.5	65.1
	HF (ms2)	1739	3976
	HR (/min)	107.3	25
	STAI PRE	49.1	23
	STAI POST	60	21
Control	SDNN (ms)	77.5	45.8
	RMSSD (ms)	87.5	58.3
	HF (ms2)	2245.5	3949
	HR (/min)	73.5	21
	STAI	33.5	16

SDNN = standard deviation of all NN intervals, RMSSD = square root of the mean of the sum of the squares of differences between adjacent NN intervals, HF = high-frequency spectral component, HR = mean heart rate, STAI = State-Trait Anxiety Inventory, STAI PRE = STAI administered before the simulation assessment, STAI POST = STAI administered after the simulation assessment.

Results of paired significance tests between the two data sets are shown in Table 3. HRV variables showed relatively small differences between the groups that were not significant. The exception to this was heart rate, with a significant 34 beat/min difference between simulation assessment and control group means. Significant differences were found between both pre- and post-simulation median STAI scores and control. Effect sizes for SDNN, RMSSD and HF were small, those for heart rate and both STAI comparisons were larger.

Table 3.

Paired differences: simulation assessment vs. control groups (n = 36).

Paired simulation assessment/control difference	Median/mean difference	95% CI	p	Effect size
SDNN (ms)	−2.4	N/A	0.413	0.227
RMSSD (ms)	−5.0	N/A	0.730	0.193
HF (ms2)	−4966.5	−11,882; 1949	0.154	0.044
HR (/min)	33.6	27.4; 39.7	<0.001	2.382
STAI PRE	15.6	N/A	<0.001	0.924
STAI POST	26.5	N/A	<0.001	1.672

CI = confidence interval, SDNN = standard deviation of all NN intervals, RMSSD = square root of the mean of the sum of the squares of differences between adjacent NN intervals, HF = high-frequency spectral component, HR = mean heart rate, STAI = State-Trait Anxiety Inventory, STAI PRE = STAI administered before the simulation assessment, STAI POST = STAI administered after the simulation assessment.

In addition to the above analysis, the sample was divided into junior students (first and second year, n = 15) and senior students (third and fourth year, n = 21) and post-simulation STAI scores were compared between these two groups. A mean difference of −9.75 in scores was observed between the groups (p = 0.029, effect size = 0.813).

Correlation between HRV variables and post-simulation STAI scores yielded small coefficients for both the simulation and control groups (Table 4). Correlation coefficients were generally smaller in the simulation group compared to the control group.

Table 4.

Heart rate variability vs. STAI correlations (n = 36).

Group	Variable	r	p
Simulation1	SDDN (ms)	0.028	0.873
	RMSSD (ms)	0.092	0.593
	HF (ms2)	0.033	0.849
	HR (/min)	0.311	0.065
Control2	SDDN (ms)	0.131	0.446
	RMSSD (ms)	0.155	0.366
	HF (ms2)	0.184	0.282
	HR (/min)	0.162	0.346

1 = correlated with post-simulation STAI score, 2 = correlated with control STAI score.

SDNN = standard deviation of all NN intervals, RMSSD = square root of the mean of the sum of the squares of differences between adjacent NN intervals, HF = high-frequency spectral component, HR = mean heart rate, STAI = State-Trait Anxiety Inventory.

Discussion

In this study, aimed at investigating the effects of a clinical simulation assessment on stress and anxiety measures in students, no significant differences were observed in all but one of four HRV variables between a simulation assessment and at rest, while significant differences in anxiety, measured with a validated tool, were observed between these groups. No significant relationships were observed between HRV variables and anxiety scores in either the simulation assessment or control groups. These data suggest that students participating in an emergency care simulation assessment subjectively experienced significantly more anxiety than at rest but this seemed to be associated predominantly with increased sympathetic activity, as indicated indirectly by an increase in heart rate, and not parasympathetic withdrawal.

HRV variables chosen for this study were those associated with overall variability and parasympathetic influence. When exposed to a cognitive stressor, a typical sequential response is seen involving parasympathetic withdrawal initially followed by activation of the sympathetic nervous system [18]. Consequently, HRV changes suggestive of a stress response include a reduction in magnitude of variables representing parasympathetic activity (RMSSD and HF) and a decrease in overall HRV (SSDN). Although low frequency (LF) power and the LF: HF ratio have previously been thought to provide a valid measure of sympathetic tone and sympatho-vagal balance, the validity of these variables has been questioned [[19], [20], [21]]. Consequently, in this study, no HRV variable was assumed to convey valid quantification of sympathetic activity.

The reduction in parasympathetic influence observed in the current study was small – a 6% reduction in RMSSD values and a 25% reduction in HF power in the simulation assessment group with small effect sizes (Table 3). While the reduction in HF power was greater, this must be interpreted with some caution as speech may have an effect on this variable [19,20] and participants did speak intermittently throughout all of the simulations. Thus RMSSD probably gives a more accurate picture of parasympathetic activity. Despite a seemingly small degree of parasympathetic withdrawal, the heart rate increase of 37% to a median of 107 beats/min. during simulation assessments suggests a substantial increase in sympathetic tone although it was not possible to corroborate this with any HRV variables.

Published research involving a mix of simulated emergency cases and other forms of clinical care have identified changes in HRV associated with stress some of which were greater than those observed in the current study, and many of which were statistically significant when compared to a control measurement [13,22,23]. However, these results are difficult to compare for a variety of reasons. Hunziker et al. also measured RMSSD and heart rate during simulated resuscitation, however the resuscitation involved physical exertion in the form of cardio-pulmonary resuscitation which may have affected these variables beyond the effects of cognitive stress alone [13]. Nakayama et al. documented significant changes in heart rate and HF power during one component of a simulation compared to other components, however the simulated case was not an emergency and they report no other measure of parasympathetic activity other than HF power which may have been affected by speech (their students spoke throughout the simulation) [22]. Rieber et al. documented a significant increase in LF: HF ratio in students during simulations compared to a control measure, however the simulated case was not an emergency and again, the LF: HF ratio is very much in doubt as a valid measure of sympatho-vagal balance [23]. The small effect sizes for SDNN, RMSSD and HF observed in this research suggest that HRV as a marker of stress in similar contexts may be of limited value.

State-Trait Anxiety Inventory scores from the current study documented during both pre-simulation and post-simulation questionnaires were significantly higher than those obtained under control conditions (Table 3). Moreover, post-simulation STAI scores were significantly higher than pre-simulation scores (mean difference = −8.139, p < 0.001, effect size = 0.601). This was probably an indication of greater levels of anxiety present immediately after completion of the assessment compared to anticipation of the assessment beforehand.

The difference in post-simulation STAI scores between junior and senior student participants indicates that senior students in their third and fourth year of study experience greater anxiety during simulation assessments than their more junior counterparts. This may be due to the greater complexity of clinical cases used for assessments as students' progress across academic years of study, or it may also be influenced by the introduction of paediatrics simulation assessments in fourth year. The closer proximity of program completion in fourth year, with this final milestone contingent in part on the outcome of simulation assessments, may also explain the heightened anxiety. Regardless of the cause, it appears that students are not able to adapt to and moderate the stress of simulation assessments as they progress from first to fourth year.

Post-simulation STAI scores were high compared to others documented during states of stress related to simulations that are available in the literature. In the study by LeBlanc et al., a mean STAI score of 46.1 was observed amongst paramedics who had just completed a 20-minute “high stress” simulated emergency case [14]. Harvey et al. observed a mean STAI score of 40 immediately after completion of a “high stress” trauma simulation in a group of general surgery and emergency medicine residents [15]. Piquette et al. documented mean STAI scores of 42 and 40 before and after “high stress” critical care simulations in a group of critical care residents [16]. Only the scores from LeBlanc et al. and Piquette et al. involved assessment, but in the former the assessment (of drug calculation ability) occurred after completion of the STAI by participants. By contrast, the pre-simulation and post-simulation scores in the current study were 49 and 60 respectively. The fact that the simulations were formal assessments contributing towards a final mark could possibly explain why the STAI scores were so much higher than those observed under other, possibly less threatening, circumstances. Such an environment, which exposes students to socio-evaluative stressors, may elicit a distress response resulting in higher STAI scores [11].

The lack of any significant relationship between subjective and objective stress measures in this study was unexpected. Only one other study could be found in the literature that assessed the relationship between HRV and STAI scores. In the study by Jones et al., assessing HRV and STAI scores in eight surgeons performing colorectal surgery, a significant strong correlation (r = 0.766) was observed [24]. It must be noted, however, that reliance was made on a single HRV variable (LF:HF ratio) with questionable validity as a measure of sympatho-vagal tone making these results doubtful. It is unclear why a stronger linear relationship between HRV and STAI scores was not observed in the current study.

A strength of this study is that it involved an authentic assessment environment with an assessment that carried a mark weighting for students, unlike other simulation assessment studies [[12], [13], [14], [15], [16],22,23]. The response from participants thus includes a realistic component of stress based on academic importance of the event which, in reality, is present in all assessments contributing to an overall determination of success. A second strength is that the control data collection was carried out on a different day to the simulation assessment. This prevented stress arising from anticipation of an assessment which would most likely have been present if control data was collected just before the assessment, as was the case with other studies [[12], [13], [14], [15], [16],22,23].

There are also several limitations to this research. Collection of control data from 55% of the original 66 students from whom simulation assessment HRV and STAI data was collected resulted in a smaller sample than originally anticipated. Without control data for comparison, no inferences can be made about the effect of the simulation assessment. A smaller sample and fairly large dispersion of data, as observed in the interquartile range of many of the HRV variables, raises the possibility of a Type II error in the HRV simulation assessment and control comparisons. Many HRV variables are known to be affected by changes in posture and resulting baroreceptor reflexes. Although it was not possible to match the posture of any of the participants during the simulation assessments exactly, the control measurements were recorded in a seated position as it was felt that this most closely approximated the simulation assessment conditions. Lastly, it is possible that a non-linear relationship may exist between HRV and STAI data that was not detected by correlation analysis.

Conclusion

This study found that students participating in an emergency care simulation assessment experienced significantly elevated levels of anxiety and increased heart rate during the assessment, compared to at rest. An objective measure of stress, in the form of HRV, showed changes suggestive of decreased parasympathetic influence, but these changes were not significantly different to those observed at rest. Anxiety and objective measures of stress were not linearly related. Further research is required to investigate the nature of autonomic balance resulting from simulation assessment-induced stress and to explore the effects that this response may have on performance. Results of this research suggest that SDNN, RMSSD and HF HRV variables are of limited value as a markers of stress in similar contexts.

Dissemination of results

Results have not been disseminated beyond this publication.