Pediatric Early Warning System Scores: Lessons to be Learned

Abstract

The objective was to evaluate the use of a pediatric early warning system (PEWS) score in Dutch general and university hospitals, 4 years after the introduction of a national safety program in which the implementation of a PEWS was advised. An electronic cross-sectional survey was used. All general and university hospitals ( n  = 91) with a pediatric department in The Netherlands were included in the study. The response rate was 100%. Three-quarters of all Dutch hospitals were using a PEWS score in the pediatric department. A wide variation in the parameters was found leading to 45 different PEWS scores. Almost all PEWS scores were invalidated, self-designed, or modified from other PEWS scores. In one-third of the hospitals with an emergency room, a PEWS was used with a wide variation in the parameters leading to 20 different PEWS scores, the majority of which are invalidated. Three-quarters of the hospitals did implement a PEWS score. The majority implemented an invalidated PEWS score. This may lead to a false sense of security or even a potentially dangerous situation. Although these systems are intuitively experienced as useful, the scientific evidence in terms of hospital mortality reduction and patient safety improvement is lacking. It is recommended to establish a national working group to coordinate the development, validation, and implementation of a wide safety program and a PEWS usable for both general and university hospitals.

Introduction

In 2008, the Dutch Hospital Association introduced a safety management system (SMS) called 'Prevent Damage, Work Safely' in response to a national study on potentially avoidable care-related patient damage in Dutch hospitals. ¹ On the basis of SMS, a pediatric safety management program was developed in 2011. The aim of this program was to reduce the number of cases of potentially avoidable harm by 50% in 5 years. One of the six themes in this program was the ‘early identification and treatment of critically ill children.’ ¹ In this respect, the expert group recommended implementing a so-called Emergency Intervention System based on a pediatric early warning score (from here on referred to as PEWS score). This PEWS score, as an afferent component of the system, should be linked to a standardized call procedure for a step-up care with accompanying rules on how to react on a decline of the score as an efferent component of the system. ^{2 3} These scores combined with the manner of response to these scores are known as the Pediatric Early Warning System (PEWS).

A PEWS score validated according to Dutch guidelines and national health care system had not yet been available at that time. It was recommended at the time to use a PEWS score validated according to foreign guidelines, such as that by Duncan et al ² or Parshuram et al, ⁴ or the invalidated PEWS score developed by the British National Health Service Institute for Innovation and Improvement. This recommendation was provided although the two validated systems were tested in completely different settings. The systems of Duncan et al and Parshuram et al were validated in a tertiary care setting in which the final outcomes, 'resuscitation' and 'urgent admission to the pediatric intensive care unit (PICU)' were used to identify critically ill children. They were validated in emergency departments only for predicting the level of medical care needed but not for triage. The systems were never validated for settings in general hospitals. So, although the Dutch Hospital Association had recommended implementation of early warning systems in all hospitals, no PEWS score had been validated for use in Dutch general hospital settings. The purpose of this study was to investigate how many hospitals complied with this recommendation and what kind of PEWS scores hospitals choose to use in The Netherlands, 4 years after the introduction of the national safety program, and to determine lessons learned and to be learned.

Materials and Methods

With the aim to establish how many Dutch hospitals did use a PEWS score at their pediatric department and emergency room (ER), and which PEWS score and parameters were used, a cross-sectional survey was conducted by means of an electronic survey ( Supplementary Material , available in the online version). The respondents were asked to state their motivation for implementing a PEWS score, and the satisfaction (dichotomous yes-no questions) with the functioning of the PEWS. All Dutch hospital locations with a pediatric department were included in this study. Hospital locations that provide only outpatient treatment were excluded. At the time of the survey, there were 83 general hospitals and 8 university hospitals in the Netherlands. In total, 91 (general and university) hospital locations with a pediatric department were included in the study. The emergency pediatric patients are primarily seen at the ER in 82 of these 91 locations. At the other nine hospitals, emergency patients are primarily seen at the pediatric ward.

An electronic survey was sent by email to one of the pediatricians of these 91 hospitals between October 2014 and November 2014, with the request to fill out the survey on behalf of their departments or to forward the survey to the person responsible for the PEWS. The pediatricians selected are responsible for the PEWS or have an affinity with acute care or are approached by the department as the representative to fill in the survey. They are consultants to the pediatric ward, the ER, and the outdoor patient clinic. They reflect the consensus of the department in satisfaction questions, for both pediatricians and nurses. Non-responders were sent reminders after 2 and 4 weeks. The missing three hospitals were contacted by a phone call in which the survey was completed. The results were analyzed using SPSS statistics 20.0 (IBM Corp., Armonk, USA); descriptive statistics were used to present the results.

Since this study included no human subjects, approval of the study by the Institutional Review Board was not necessary.

Results

Pediatric Departments

The response rate was 100%. In total, 75% (68/91) of the Dutch hospital locations with a pediatric department implemented a PEWS score at their pediatric department ( Fig. 1 ), including four of the eight university medical centers. The SMS safety program was the most important reason for implementing a PEWS score in 75% (51/68) of the hospitals. Another reason for implementing a PEWS score was the need for better monitoring of the clinical course (22%; 15/68) or after a severe incident (3%; 2/68).

Fig. 1.

Pediatric early warning system (PEWS) score at Pediatric Departments and Emergency Rooms in The Netherlands.

The 68 hospitals with a PEWS score used 45 different versions of PEWS scores, with 20 different parameters in various combinations ( Table 1 ). None of the parameters was used in all systems.

Table 1. Parameters used in pediatric early warning system scores at pediatric ward and emergency room.

	Pediatric ward ( n  = 68)		Emergency room ( n  = 26)
Parameter	No.	%	No.	%
Heart rate	66	97.1	25	96.2
Respiratory frequency	66	97.1	26	100.0
Oxygen saturation	61	89.7	24	92.3
Oxygen therapy	53	77.9	20	76.9
Blood pressure	45	66.2	18	69.2
Temperature	43	63.2	17	65.4
Nurse's worried sign	42	61.8	20	76.9
Consciousness	41	60.3	16	61.5
Capillary refill	29	42.6	9	34.6
Parents' worried sign	22	32.3	11	42.3
Respiratory effort	22	32.2	8	30.8
Diuresis/urinary production	6	8.8	3	11.5
Nebulization frequency	3	4.4	2	7.7
Behavior	3	4.4	1	3.9
Color	3	4.4	3	11.5
Convulsion	3	4.4	1	3.9
Persistent postoperative vomiting	2	2.9	1	3.9
Pain score	1	1.5	0	0.0
Chest retractions	1	1.5	1	3.9
Diarrhea	1	1.5	0	0.0

Out of the 68 hospitals using a PEWS score, 15 (22%) indicated that they implemented a validated PEWS score (Brighton PEWS, ⁵ 4/15 or Parshuram, 10/15). One of these 15 respondents did not know what type of validated system was used and 11 did modify the PEWS score by adding parameters. Finally, 3 of the systems were identical to the Parshuram PEWS and none to the Brighton PEWS score. The other hospitals used invalidated scoring systems (85%; 58/68) or could not indicate what their scoring system was based on (9%; 6/68). The invalidated scoring systems were acquired from other hospitals (31%) or designed by the staff of the hospitals themselves (69%).

Out of the four academic hospitals using a PEWS score, three indicated that they implemented a validated PEWS score but none of these scores were identical to Brighton or Parshuram PEWS score ( Table 2 ). These three were all modified by removing or adding parameters. The parameters added were ‘Nurse's worried sign’ and/or ‘Parents' worried sign’.

Table 2. Pediatric early warning system scores used in academic and general hospitals.

		Which PEWS score at pediatric ward?
		Validated	Invalidated		Don't know	Total
			Copied (other hospital)	Self-designed / modified
Type of hospital	General	3	18	37	6	64
	Academic	0	0	4	0	4
Total		3	18	41	6	68

Of the respondents using a PEWS, 76% (51/67) indicated that they were satisfied with the functioning of the PEWS score at their department. One hospital with a PEWS was not included in this assessment because the PEWS had been implemented only a few days before the survey was conducted.

In 66 (including the four academic hospitals) of the 68 hospitals (97%) with a PEWS, the first response to an elevated PEWS score was to contact a physician (pediatrician or pediatric resident). Depending on the severity of the score and/or the physicians' opinion, a Rapid Response Team in general hospitals (if this did exist) or a Pediatric Rapid Response Team in academic hospitals was called.

Emergency Rooms

In 26 out of 82 hospital locations with an ER, a PEWS score was used ( Fig. 1 ) including one of the eight university medical centers. The SMS safety program was available in 73% (19/26) hospitals, the most important reason for implementing a PEWS score on the ER. Another reason for implementing a PEWS score was the need for better monitoring of the clinical course (15%; 4/26) or for another reason not further defined (12%; 3/26). The 26 hospitals were using 20 different versions, with 18 different parameters in various combinations ( Table 1 ). In 7 out of the 26 hospitals (27%), the respondents did indicate that they were using a validated PEWS score (Brighton or Parshuram). However, the parameters of these seven 'validated' scoring systems show that only two of the systems were exactly the same as the Brighton or Parshuram PEWS score. The other five systems were modified by the user by adding parameters. This means that validated scoring systems were used in only 8% (2/26) hospitals. Four out of 22 invalidated early warning scores were unpublished systems acquired from other Dutch hospitals, and 18 were scoring systems designed by the staff of the hospitals themselves. Most of these self-designed scoring systems were designed by pediatricians in collaboration with ER nurses. Respondents of 19 hospitals using a PEWS score in their ER (76%) indicated that they were satisfied with the functioning of the PEWS without formal evaluation. One hospital could not respond in this item since the PEWS had been implemented only a few days before the survey was conducted.

Discussion

In summary, we found that three-quarters of the hospitals did implement a PEWS score. The majority implemented an invalidated PEWS score. This may lead to a false sense of security or even a potentially dangerous situation.

Motivation to Implement PEWS

The main reason for implementing a PEWS was by far the SMS safety program. This result indicates that hospitals have implemented invalidated scoring systems with the extrinsic motivation to merely “check the box” to implement some scoring system rather than out of intrinsic motivation. Implementation of a new scoring system or strategy in safety management in hospitals only based on extrinsic pressure is less likely to be successful than based on intrinsic motivation. The implementation of a new scoring system, not based on intrinsic motivation, needs some outside pressure for a successful change. ⁶ The willingness to accept this new system depends mainly on the level of and the perceived goals behind this pressure. ⁶ In a time that healthcare professionals are more and more occupied with documentation of all kind of data for quality indicators and certifications, the introduction of the new scoring system will only be successful if the usefulness is clear and distinguishable for them. ⁷ Change is possible only if a well-designed intervention and implementation is used. This means that the implementation has to be well-prepared, unequivocal with a clear education of medical and nursing staff and guidance on the workplace, and evaluation on a regular basis. ^{7 8}

The difference in PEWS rate between university (50%) and general hospitals (77%) is substantial. However, no difference in the motivation was found that can explain this discrepancy. This might indicate that there is a reduced feasibility to implement a PEWS score in large-volume hospitals.

Despite the fact that extrinsic motivation was the main reason for implementing a PEWS score, three-quarters of the respondents who used a PEWS score stated that pediatricians and nurses were satisfied with how the PEWS score was functioning at their ward. It appears that the system has an added value for healthcare professionals, even when it occasionally fails; therefore, users also have negative experiences with it.

PEWS at the Emergency Room

Despite the fact that research has shown that a PEWS score is not suitable for a triage of patients at the ER, ⁹ our results show that in one-third of the ERs, a PEWS score could be used as the single triage system or in combination with another triage system. In the latter, it was used mostly as a monitoring system to evaluate the medical condition of the patient over time. PEWS scores at the ER have been validated only in university hospital settings and for predicting the level of medical care needed, particularly with regard to admission to a PICU, but never for triage. The predictive value of PEWS score is based on the progress over time and is not suitable as a single-scoring-triage system. ⁹ Scoring systems like the Manchester triage system, developed and validated as triage systems for children at the ER should be used as predictive triage tools. ¹⁰ After first triage with such a scoring system, the PEWS score in the ER might be useful if the patient is admitted to the ward as a first scoring point for later trends. The professionals who are using a PEWS score at the ER should be aware of the incapability of the system as a triage instrument and the potential pitfalls. ⁹

PEWS at the Pediatric Ward

This study shows that the parameters used in the PEWS score did vary in a wide range and that almost all PEWS scores in use are not validated. It is remarkable that vital parameters such as heart rate, oxygen saturation, and blood pressure were not used in up to 30% of the PEWS scores, where especially heart rate and oxygen saturation are early signs of deterioration. Almost all incorporated PEWS scores were invalidated, self-designed, or modified from other PEWS scores. The Dutch situation corresponds to a large degree with the results of a study conducted in Great Britain. ¹¹ Since the same results are found in Great Britain and The Netherlands, it is quite plausible that also in other countries PEWS scores are in use, that are not validated at all or used in a different setting (general hospital instead of university hospitals) with a different case mix of patients. The early warning scoring systems that have not been validated and currently used in Great Britain and The Netherlands have an unknown positive and negative predictive value. They might be better or worse than the validated scoring systems. In a time in which patient safety has become the main topic of governments and hospital policies, it is remarkable that hospitals develop their own early warning systems without validation. In this way, another checklist is introduced that can create false expectations and sense of safety among the users, especially when they are not aware of the pitfalls of their own PEWS score.

One of these pitfalls is demonstrated by several international studies that showed that validated PEWS scores have a relatively low sensitivity (on average not higher than 70%) and specificity for predicting resuscitation at the pediatric department or admission to a PICU. ^{3 4 12} Another pitfall is the fact that the systems (Brighton, Duncan, and Parshuram) were validated in a tertiary care setting in which the hard final outcomes, 'resuscitation' and 'urgent admission to the PICU' were used to identify critically ill children. These systems have false-negative (underestimate the seriousness of condition resulting in a potential delay of treatment) and false-positive (overestimate the seriousness of condition resulting in a potential over-use of resources and additional costs) results. Although these PEWS scores have limited reliability, at least they have shown to be capable of identifying clinical deterioration in an early phase. ^{3 4 12} Modifications to these scoring systems may alter the sensitivity and specificity of the existing systems, for better or for worse, which may lead to unknown pitfalls. Besides this, the PEWS is only validated in a tertiary care setting. This means that the sensitivity and specificity of PEWS scores in general hospitals and their patient populations are unknown. Remarkably, the four PEWS scores used by tertiary care settings in The Netherlands are all invalidated since they are self-designed (¼) or modified (¾). Despite the lack of evidence for using ‘worried sign’ as a parameter, all four academic centers added “worried signs” to their PEWS scores. Of all general and academic hospitals that are using a PEWS score, 77% added ‘nurse's worried sign’ and 42% added ‘parents' worried sign’ as a parameter. The need to include ‘worried signs’ as a parameter is possibly caused by trying to minimize the possible false-negative results.

Besides the fact that the PEWS is only validated in tertiary care setting, it also has to be questioned if the end points of the validation studies of the PEWS in university hospitals are the same that should be used in general hospitals.

Validation studies on the benefits of PEWS in general hospitals and the end points to use are generally lacking, ^{12 13} and needed.

Lessons to be Learned

The current situation with mainly invalidated, self-designed, or modified scoring systems in use can create a false sense of security, and interfere with proving that there is an added value of PEWS for healthcare in general hospitals. A lesson to be learned is that making a safety recommendation to implement the use of a PEWS score, when such a validated score does not exist for use in general hospitals, may result in an improper use of those that are available in an attempt to comply with the recommendations rather than implement a system (including an efferent component) to improve patient safety. The added value of PEWS is not only the scoring system itself (afferent component), but rather the awareness of vital signs created by the implementation of a PEWS score at the ward, and the accompanying rules on how to react on a decline of the score. The imbedding of the system and rules of escalation or de-escalation of care (efferent component) are important factors in the PEWS as a valuable instrument. Two of the lessons learned from this study is that hospitals in The Netherlands are keen to comply with the safety recommendations to utilize an early warning score for pediatric patients and that the opportunity to implement a validated score across the nation exist. This has to be well prepared, with a clear education of medical and nursing staff, support to the ward, and evaluation on a regular basis.

Recently, a working group has been established in The Netherlands, which is supported by the Dutch Pediatric Association. This working group will coordinate the development of a PEWS usable for both general and university hospitals, and studies to validate these PEWS. It will also give recommendations for the implementation of this system. It is our opinion that in other countries as well, such a working group appointed and supported by national pediatric associations might play a key role in the development and implementation of pediatric SMS in hospitals.

Limitations of the Study

This is a survey in which the respondents had knowledge of the research topics. The data of each department was obtained from one person. Although the respondent was asked to give the opinion of the whole department of pediatrics including doctors and nurses, there may be a respondent bias. The parameters of PEWS scores were provided by the respondents. These given parameters that will form the PEWS scores were not verified during site-visits or by a document. Moreover, the opinion of nurses was not obtained directly in this study. In a subsequent study, their opinion will also be included.

Conclusion

Four years after the introduction of a national safety program in which the implementation of a PEWS was advised, three-quarter of the hospitals did implement a PEWS score. As there was no standard validated PEWS score available according to Dutch guidelines, the majority implemented a modified or self-designed, invalidated PEWS score. This current situation may lead to a false sense of security or a potentially dangerous situation. To improve this situation, a ‘PEWS Taskforce’ has been established. The above-mentioned situation is also likely to exist in other countries. For each country, it is recommended to establish a national working group to stimulate and organize the development and implementation of a PEWS score usable for both general and university hospitals.

Funding Statement

Funding None.