Could low Safety Attitudes Questionnaire scores be indicative of an environment where it may be difficult to get new training practices established?

Abstract

Introduction

Simulation-based, multiprofessional team training (SBMPTT) is used widely in healthcare, with evidence that it can improve clinical outcomes and be associated with a positive safety culture. Our aim was to explore the impact of introducing this type of training to a gynaecological team.

Methods

In this interrupted time-series study, ‘Safety Attitudes Questionnaire’ (SAQ) data was collected both before and after SBMPTT was introduced to a gynaecological team.

Results

Low baseline SAQ scores coincided with difficulty in establishing the training, meaning that at the end of our study period only a small proportion of staff had actually attended a training session. Despite trends towards improvement in scores for safety climate, teamwork climate and job satisfaction, no statistically significant difference was observed. There was however an improved perception of the level of collaboration between nursing staff and doctors after the introduction of training.

Conclusions and Discussion

In this paper we explore a hypothesis that low baseline SAQ scores may highlight that the multiprofessional teams most in need of training work in environments where it is more challenging to implement. There is evidence from other specialties that multiprofessional team training works, now we need to understand how to address the barriers to getting it started. In this paper we suggest how the SAQ could be used as a directive tool for improvement; using the detailed analysis of the local safety culture it provides to both inform future training design and also provide management with an objective marker of progress.

Introduction

This study assesses the introduction of simulation-based multiprofessional team training (SBMPTT) to a gynaecology unit in South West England using the Safety Attitudes Questionnaire (SAQ) scores as an outcome measure.

A recent meta-analysis has shown that the use of team training in healthcare settings is effective at improving all four areas of Kirkpatrick's ‘training evaluation criteria’.¹ Namely participants’ positive reactions to training, the learning they achieve (of knowledge, skills and abilities), and the transfer of that learning to the job and results as measured by organisational or patient outcomes (including safety culture measured by the SAQ). There is also evidence that any learning acquired through team training can impact positively and sequentially on both transfer and results.² SBMPTT is already practiced widely within the paired specialty of obstetrics. In England (where training in the management of obstetric emergencies is mandatory for all professionals providing maternity care³) the well-established PRactical Obstetric Multi-Professional Training (PROMPT) course is used to provide annual training in most maternity units. Since PROMPT was introduced as a mandatory annual training event for all healthcare professionals working in a busy obstetric tertiary referral centre in South West England (the same hospital in which this study took place), there has been strong evidence of improved clinical outcomes.^4–6 SBMPTT is now strongly embedded in this unit and staff attitudes towards safety and teamwork, as measured by the SAQ,^{7 8} have been found to be very positive.⁹ In the same hospital where SBMPTT has been introduced in general surgery, SAQ scores were found to increase for safety and teamwork climate.¹⁰ Evidence that positive SAQ scores correlate with improved patient outcomes¹¹ was a driver for use of the questionnaire in this study.

PROMPT has been successfully implemented in a number of countries worldwide (and a similar course called Advanced Life-Support in Obstetrics (ALSO) runs in America). In England, no equivalent training course exists for gynaecology despite the Royal College of Obstetricians and Gynaecologists (RCOG) recommending to gynaecology units that ‘training in the management of emergencies must be given priority’.¹² There is though the potential for life-threatening emergencies that would require management by a multiprofessional team occurring in the gynaecological setting, including sepsis, collapse and haemorrhage (both postoperatively and during a miscarriage) as well as ruptured ectopic pregnancy. Local successes of SBMPTT in obstetrics and in general surgery^{4–6 9–10} alongside mounting evidence that team training in healthcare settings is associated with improved outcomes,² inspired the development of local gynaecology SBMPTT based on the PROMPT model.

It is of relevance that a team of healthcare professionals in South West England developed the PROMPT course and one of the authors of this paper (DS) is a senior member of the PROMPT Maternity Foundation. The other authors of this paper work both clinically and academically within the specialty of obstetrics and gynaecology and all have an interest in medical education.

Methods

This interrupted time-series study took place at a tertiary referral teaching hospital with a catchment area of 400 000 patients and gynaecology services admitting both elective and emergency cases. The majority of elective patients had undergone major surgery (grade 3 or 4, as defined by the National Institute of Health and Care Excellence).¹³ Emergency referrals into the service were received from the early pregnancy clinic, the emergency department, general practitioners and other acute specialties.

Two SBMPTT days were run during the study period, in April of 2013 and 2014. Originally four sessions had been planned with the intention of capturing all clinical staff working regularly in the gynaecology ward. The curriculum was designed in concordance with the authors of the local obstetric SBMPTT course, following several of the key principles of PROMPT:

• Multiprofessional participants/trainers

• Locally run, using facilities in own unit

• Multiprofessional drills, supported by in-house clinical champions

• Participant debrief using clinical and teamwork checklists

All doctors, nurses and healthcare assistants (HCAs) working in the gynaecology ward were invited to attend a training session. The faculty was derived from the local department and comprised of senior gynaecologists and anaesthetists (with a background in medical education or experience of teaching on the local PROMPT course), senior gynaecology nurses, the PROMPT lead midwife and a medical student. All faculty members were fully briefed verbally on the scenarios in advance of starting training and had written information about running their station. The morning session was aimed at information giving and included a team building exercise, updates on new local guidelines and introduced some key causes of morbidity and mortality in gynaecology (sepsis and haemorrhage). In the afternoon participants were given the opportunity to practice working in teams to manage common gynaecological emergencies. Multiprofessional groups of four to six learners rotated through four ‘drills’—collapse, postoperative haemorrhage, sepsis and bleeding postmiscarriage. Low-fidelity models were used to aid real-time simulations of these emergencies in a familiar environment (the gynaecology ward), following the PROMPT model of using local clinical facilities. Each drill had case-specific learning objectives relating to diagnosis, initial emergency management, definitive treatment and medium and long-term patient support. Teamwork and safety climate specific objectives included the longitudinal themes of safe teamworking, multiprofessional values and communication within each drill. Each station lasted 30 min including time for a scenario briefing and debrief. Checklists were used to facilitate debrief on clinical aspects of managing the emergency. Facilitators were prompted to provide feedback to participants on the observed communication and teamworking within the scenario.

The SAQ (Labour and Delivery version UK) was used to generate both quantitative and qualitative outcome data initially in April 2013 then again in July 2014. The SAQ is a validated survey for measuring safety culture within healthcare teams.^{7 8} Doctors’ rotas and nurse/HCA ‘off duty’ rosters were used to assess those who have worked in the gynaecology ward for at least 4 weeks and would be eligible to complete the SAQ. The training course faculty and the study authors were excluded. There were 51 eligible staff in April 2013 and 50 in July 2014. Eligible staff were invited to complete the SAQ via their National Health Service (NHS) Trust email using existing departmental contact lists. In addition they were sent a paper copy and researchers (SC and NR) promoted completion of the SAQ during team meetings. Once anonymously completed, the questionnaires were placed in a sealed box kept in a locked corridor only accessible by NHS staff working in the ward. In June 2014, a list of eligible staff (using the same criteria) was created and they were invited to complete the SAQ. Staff did not need to have completed the SAQ in April 2013 to be eligible and they may not have attended a training day.

The SAQ was adapted to ensure specialty-appropriate language (with the term ‘midwife’ replaced by ‘nurse’ and ‘labour ward’ replaced by ‘gynaecology ward’). This validated questionnaire^{7 8} assessed the safety culture of the ward as a whole by asking the individuals working there questions within six different domains; teamwork climate, safety climate, job satisfaction, stress recognition, perceptions of management, working conditions. The SAQ asks for the respondents’ level of agreement (on a five-point Likert scale) with 47 positively worded statements, giving a score of 1–5 (1=Strongly disagree, 2=Disagree, 3=Neutral, 4=Agree, 5=Strongly agree). In the case of the 10 negatively worded statements, adjusted scores were calculated automatically by a formula in MS Excel (adjusted score=6—score). The SAQ also asks the staff to rate their level of collaboration and communication with the different members of the multiprofessional team (1=very low, 2=low, 3=adequate, 4=high, 5=very high). We planned to exclude all invariant responses from the analysis. Demographic data were collected to ensure that there were no duplicate responses. Free-text responses to the question ‘What are your top 3 recommendations for improving patient safety on the gynaecology ward?’ were transcribed word-for-word and analysed for commonality. All data from the SAQs were stored on a password-protected NHS computer.

Data are expressed as percentages (for proportions of staff) and means (for individual item scores). Percentage scores for the six domains (teamwork climate, safety climate, job satisfaction, stress recognition, perceptions of management, working conditions) were calculated using the average score for items within those domains and applying the formula (score—1)×25 (thus converting the five-point Likert scale to a 100-point scale (1=0, 2=25, 3=50, 4=75, 5=100)). Statistical significance between the two cohorts was investigated using STATA software. The Mann-Whitney U test was used for non-parametric data. In order to use this method, medians and means were calculated and compared for alignment. Divergence confirmed a non-parametric distribution. Statistical significance was defined as p<0.05.

Approval was granted to conduct the study and disseminate results by North Bristol NHS Trust Research and Development, reference number 3345.

Results

Who completed the SAQ?

In April 2013, 82% (n=42/51) of eligible staff completed the SAQ (25/30 doctors, 12/15 nurses and 5/6 HCAs). In June 2014, 58% (n=30/50) of eligible staff completed the SAQ (18/28 doctors, 7/17 nurses, 4/5 HCAs and 1 with role undeclared). There were no questionnaires with invariant responses. Through analysis of demographic data provided by respondents, no duplicate responses were found. Around a third (n=12) of the respondents in the second cohort had attended a simulation training session.

Collaboration and communication scores

In April 2013, only 36% of doctors felt that the levels of collaboration and communication they experienced with nurses were high or very high. By June 2014 this had reached 71%. Analysis showed this to be a statistically significant improvement (mean ratings=3.17/5.00 (April 2013) and 4.06/5.00 (June 2014), p=0.02). Nurses more consistently rated their collaboration and communication with doctors as high or very high (mean ratings=4.50 (April 2013) and 4.00 (June 2014)/5.00), p=0.17) throughout the study period.

Scores by SAQ domain

Table 1 displays the pretrainingand post-training SAQ scores by domain and allows for comparison with studies that have used the SAQ to assess the impact of training; and also with SAQ scores from the obstetric unit in the same hospital where SBMPTT is strongly embedded. During the study period there was a trend towards improvement in scores for questions relating to safety climate (p=0.19), teamwork climate (p=0.4) and job satisfaction (p=0.1), none of which reached statistical significance. Therefore, the introduction of SBMPTT failed to significantly improve SAQ scores in this setting.

Table 1.

SAQ scores—comparison with other studies.^{14 9 10 15}

	Baseline scores—this study (SD)	Post-training—this study (SD)	P Value	Pretraining—SaFE study	Post-training—SaFE study	Embedded training—LW NBT	Pretraining—surgical wards, NBT (van der Nelson et al)10	Post-training—surgical wards, NBT (van der Nelson et al)10	Pretraining—Paediatric ED, USA	Post-training—Paediatric ED, USA
Teamwork climate	59.9 (14.45)	62.2 (13.68)	0.4	72.5	71.9	76.1	72.8	82.5	69.2	73.1
Safety climate	62.4 (13.91)	66.5 (12.93)	0.19	69.3	74	74	67	77.8	73.2	78.6
Job satisfaction	57.7 (18.10)	63.4 (18.10)	0.1	65.5	65.1	71.9
Stress recognition	69.4 (11.41)	67.5 (11.29)	0.8	70.8	70.9	65
Perceptions of management	43.8 (15.93)	44.4 (12.83)	0.5	47.5	49.2	47
Working conditions	53.2 (16.59)	53.6 (19.)	0.9	59.9	62.2	62.6

LW NBT, Labour Ward, North Bristol NHS Trust; NBT, North Bristol NHS Trust; SAQ, Safety Attitudes Questionnaire.

Scores by clinical role and SAQ domain

The demographic data and scores for the different domains of SAQ by clinical role are shown in table 2. Splitting the data in this way demonstrates that the trend towards improvement in scores for safety climate can be mostly accounted for by the ratings of nurses and HCAs. Conversely, any trend towards improvement in the score for teamwork climate was mostly due to higher ratings from doctors.

Table 2.

Scores for domains of SAQ by clinical role

	Doctors		Nurses		HCAs
	April-13	June-14	April-13	June-14	April-13	June-14
Number eligible for study	30	28		17	6	5
Number responding (%)	25 (83)	18 (64)	12 (80)	7 (41)	5 (83)	4 (80)
Number working in April and June		10		13		4
Number that attended training	n/a	7	n/a	3	n/a	2
Mean age (lowest-highest)	33.17 (26–60)	31.33 (27–54)	40 (23–53)	30.60 (24–37)	41.25 (26–54)	34.33 (28–45)
Years specialty experience (min–max)	6.42 (0–31)	5.5 (1–30)	12.5 (5–23)	6.44 (1–12)	7.5 (6–9)
Teamwork climate score	60.85	65.45	57.56	58.93	61.46	55.1
Safety climate score	64.43	65.18	56.93	67.35	65.89	71.43
Job satisfaction score	59.79	65.56	51.83	53.57	60.25	64.58
Stress recognition score	70.05	67.01	68.23	75.00	67.50	60.94
Perceptions of management score	47.92	48.96	34.9	34.82	45.00	37.5
Working conditions score	58.85	58.33	42.71	44.64	51.25	47.92

HCA, healthcare assistants; min-max, minimum and maximum; SAQ, Safety Attitudes Questionnaire.

Responses to individual SAQ questions

When responses to individual questions were examined it was noted that throughout the study period there was a perception among staff that high levels of workload were common in the gynaecology ward (mean scores=4.60 (April 2013) and 4.57 (June 2014)/5.00). Staff were unconvinced that staffing levels were sufficient for the number of patients (mean scores=1.69 (April 2013) and 1.97 (June 2014)/5.00). When asked if morale was high in the clinical area, staff generally disagreed (mean scores=2.14 (April 2013) and 2.48 (June 2014)/5.00). Staff agreed that they were less effective when fatigued (mean scores=4.10 (April 2013) and 4.17 (June 2014)/5.00) and believed that their performance was affected by the high workload (mean scores=4.12 (April 2013) and 3.93 (June 2014)/5.00).

All respondents were asked to give their top three recommendations for improving patient safety on the gynaecology ward. Overall, 69% of respondents suggested more staff and 30% suggested improvements in teamworking (namely communication or better multidisciplinary team (MDT) working). During the study period there was a large increase in the number of staff advocating increased input of more senior medical staff on the ward (10% April 2013 and 46% July 2014). Other suggestions included having better stocking of equipment and less bureaucracy and form filling.

Discussion

What does this study show?

During the study, there was a statistically significant improvement in how doctors’ perception of their level of communication and collaboration with nurses. This was also reflected in this group's contribution to the trend towards improvement in the teamwork climate score after implementation of training. Doctors’ scores for individual statements pertaining to working relationships with nursing staff, such as ‘Nurses input is well received in this clinical area’, ‘The doctors and nurses here work together as a well-coordinated team’ and ‘Disagreements here are resolved appropriately (ie, not who is right but what is best for the patient)’, were particularly improved. It was disappointing that only 12/50 staff in the second cohort had attended a training session as this makes the finding of improved collaboration and communication between doctors and nurses difficult to attribute to SBMPTT directly.

However, this may indicate the power of SBMPTT in establishing a set of values and ways of working applicable to the local clinical setting. Newcomers to the environment may observe the behaviours and attitudes of more established staff and choose to apply these principles to their own practice (Albert Bandura's ‘Social Learning Theory’¹⁶). The knowledge and practice gained through SBMPTT by those that did attend training may have diffused to those that did not, and in doing so could have contributed to a cultural change on the ward. This cannot be proven through the methodology deployed in our study; however, it does provide a theoretical basis to the observed increased perception of doctor/nurse collaboration.

Study strengths and weaknesses

This study set out to assess the impact of introducing SBMPTT to a gynaecology ward, with the hypothesis that SAQ scores would improve after training. Unfortunately this was not proven by this study. When considering the possible reasons for this, it is notable that during this study the implementation of training was impeded by practicable considerations. Initial plans for up to four training days to be held proved unworkable due to clinical commitments of the faculty and workforce planning issues; ultimately it was only possible to hold two training days and consequently only 12/50 staff in the second cohort of respondents actually attended training. In response to the challenge of enabling staff release for facilitation or participation in a session, training has been shortened to a half-day. Facilitated by three faculty members (five were previously required) and with up to 30 staff in attendance, a ‘ward and risk update’ is followed by two emergency drills covering haemorrhage and sepsis with cardiac arrest (run twice if required, with participants alternating between participating and actively observing). The reduced faculty requirements enable us to hold a higher number of sessions throughout the year, enabling more staff to attend while minimising the disruption to service delivery.

The study had a small sample size, potentially leading to an underpowered study unable to detect statistical significance. However, sufficient powering was limited by the practicalities of actual staff numbers; that is there was only a finite number of staff that could be recruited.

One of the strengths of the study was the high response rate in April 2013 (82%), facilitated by a local champion (SC) working in the hospital and distributing questionnaires at meetings and staff handovers. A relative weakness is the lower response rate in June 2014 (58%), a possible source of selection bias.

Low baseline scores

When considering the baseline SAQ scores in the gynaecology ward in this study, it may be of significance that scores for teamwork and safety climate were much lower than were found in other studies where introduction of training did significantly improve SAQ scores in these domains.^{10 15} Could it be that the comparatively high baseline SAQ scores seen in these two studies contributed to the success of the interventions? SAQ scores as low as our baseline have been reported in the literature,¹⁷ but there are currently no other published studies where a team training intervention with the aim of improving safety culture has been applied to a unit with baseline SAQ scores as low as found in our study for comparison. It is clear that a unit's underlying safety culture is influenced by a complex set of interacting factors. Studies in several countries have found safety culture to be a highly localised concept; with scores for all domains of the SAQ varying significantly between units within institutions.^{8 18–20} It therefore is not unexpected to have found differences in scores between our gynaecology ward and the adjoining obstetric unit. We hypothesise that the problems to which SBMPTT seeks to redress (reflected in low SAQ scores) might be the very barrier to its implementation. Indeed, further interrogation of our baseline SAQ data reveals comparatively low scores in the domains of ‘perceptions of management’, ‘working conditions’ and ‘job satisfaction’. Deeper analysis of responses to questions within these domains demonstrated an overwhelming perception of heavy workload and insufficient staffing of respondents. Indeed in this study, staff most frequently suggested that the way to improve safety on the ward was ‘more staff’. Similarly negative perceptions of workload and staffing were found in our adjoined obstetric unit,⁹ but interestingly job satisfaction remained high. The research team in that study hypothesised that the positive attitudes towards teamwork within the unit may have helped to counteract concerns about workload and staffing. Could universally low starting scores and poor morale indicate that a department will be difficult to motivate and engage in an improvement initiative?

The relatively high SAQ scores in the adjoined obstetric unit suggest that a strongly embedded team training programme may contribute to a positive safety culture.⁹ The key now is to understand the barriers to establishing a sustainable training programme in gynaecology and to break this vicious cycle.

Moving forward

1. Using SAQ data to inform initiatives aimed at improving patient safety.^{21 22}

Now that necessary changes have been made to shorten the duration of our training day, it will be crucial to ensure that the content is relevant. Therefore, a department-wide SAQ will be repeated to help further refine the programme to address the temporally pertinent issues influencing the local safety culture. For example we have already noted that scores for teamwork climate were much lower than in the adjoined obstetric unit⁹ (59.9 compared to 76.1), and within this domain the lowest scoring item was ‘the doctors, nurses and HCAs here work together as a coordinated team’. If this is a consistent finding, then specific teamwork skills training could be introduced into the training day programme. Another suggestion is to engage attendees to the training sessions in discussion about how safety could be improved in the ward using previous free-text SAQ responses as a starting point. Engaging attendees in this way makes them stakeholders in the process of improving local safety culture and is an example of how sharing of SAQ results could be used to empower and motivate a department towards self-improvement and shift the focus away from poor staffing. Once >75% of eligible staff attend training, we will repeat the SAQ with the expectation of improved scores (as achieved locally in general surgery with 79% attendance¹⁰).

• 2. An objective and validated way to measure improvement

Successful implementation of a sustainable training programme will also require clear organisational level support,,^23–25 including financial, to ensure that roster gaps are filled so that staff can have study leave to attend the training. The SAQ could be used to monitor improvement, providing managers with a measureable indicator of success of training. A similar application of the SAQ, with goals being set for departmental-level score improvement, has been used successfully in a hospital in the USA.²⁵ Another study has shown that improving SAQ scores can be associated with decreasing levels of patient harm and mortality rates,²⁶ providing validity to the suggested practice of using the SAQ as a marker of progress.

Conclusion

In conclusion, this study has not been able to clearly demonstrate that safety culture in a gynaecology ward can be improved through multiprofessional team training. However, our results still have value, as our baseline safety culture scores were very low and there was a trend towards improved scores for teamwork climate, safety climate and job satisfaction. Our comparatively low baseline safety culture scores coincided with challenges in getting training off the ground. This may highlight that the teams most in need of training are working in the very units in which it is difficult to implement. There is evidence that multiprofessional team training works and there is an unmet need in gynaecology, but we now need to understand how to address the barriers to initiating training and how to enable departments (staff and management) to strive towards an improved safety culture and ultimately better patient outcomes. Using the SAQ as a directive tool for improvement, by using the data it provides to inform future training design and also providing management with an objective marker of progress, may be the way forward.