An assessment of the psychometric properties of the Stoplight Pain Scale in a Canadian paediatric emergency department

Abstract

Objective

This study aimed to validate a novel, three faced, colour-coded, action-oriented tool: The Stoplight Pain Scale (SPS).

Methods

A prospective observational cohort study was conducted at a Canadian paediatric emergency department from November 2014 to February 2017. Patients aged 3 to 12 years and their caregivers were asked to rate pain using the SPS and the Faces Pain Scale-Revised (FPS-R). Pain was measured just before analgesia administration, 30 minutes after analgesia administration, and immediately following a painful procedure.

Results

A total of 227 patients were included; 26.9% (61/227) were 3 to 5 years old while 73.1% (166/227) were 6 to 12 years old. Using Cohen’s κ, agreement for SPS and FPS-R was ‘fair’ for children (0.28 [95% confidence interval {CI} 0.20 to 0.36]) and ‘poor’ for caregivers (0.14 [95% CI 0.07 to 0.21]), at initial measurement. The SPS had ‘fair’ agreement between child and caregiver scores, (0.37 [95% CI 0.27 to 0.47]), compared to FPS-R which showed ‘poor’ agreement (0.20 [95% CI 0.12 to 0.29]). Absolute agreement between child and caregiver SPS scores improved with repeat exposure; 30 minutes after analgesia administration, caregivers and children had fair agreement (κ=0.38, 95% CI 0.28 to 0.48); they had moderate agreement directly following painful procedures (κ=0.46, 95% CI 0.34 to 0.59). Overall, 72.4% (139/192) of children and 60.2% (118/196) of caregivers preferred SPS over FPS-R.

Conclusion

The SPS demonstrates fair agreement with FPS-R for children and fair-moderate agreement between children and caregivers; agreement improved with repeat use. The SPS is simple and easy to use; it may have a role in empowering direct child and family involvement in pain management.

Graphical Abstract

Graphical Abstract.

Children frequently experience acute pain (1–6) and adequate pain management is fundamental to providing excellent care (1,4,7–9). Paediatric patients remain at high risk for poor pain management despite recent advances in the assessment and treatment of pain (2,3,5–7,10–12). Unfortunately, children who experience pain are at risk of developing significant short- and long-term negative effects (4,9,13). Appropriate pain management has been shown to significantly improve patient, parent, and staff satisfaction with medical care (1,11,14).

Consistent use of assessment tools to measure, document, and reassess pain can improve recognition and treatment of pain (1,5,6,8,10,12). Numerous pain assessment tools have been developed and tested for use in children (10,12,15–18). The Faces Pain Scale-Revised (FPS-R) has been validated for use in young children, making it the preferred self-reported pain assessment tool for children < 4 years (12,15,16,18,19). However, FPS-R lacks actionable direction, assessing pain without giving the caregiver direct guidance on how to respond to the score.

Stoplight Pain Scale (SPS) was developed to be a simple, practical, and reliable tool for the assessment of paediatric pain (15). SPS is modeled after the ‘Stoplight Construct’, where Red means ‘Stop’ (i.e., pain requires immediate intervention), Green means ‘Go’ (i.e., pain is currently well-managed and intervention is not required), and Yellow means ‘Yield’ (i.e., pain requires timely reassessment, as the child may soon require intervention). SPS is unique, as it provides direction regarding pain management. As children generally use simpler language than their parents (20), we hypothesized that SPS’s simple wording and recognizable ‘Stoplight Construct’ might improve their ability to self-report pain, better guiding their own analgesia requirements. The primary objective was to assess SPS’s ability to accurately measure paediatric pain. We hypothesized SPS would demonstrate good agreement and better congruency between child and caregiver scores, when compared with FPS-R. We further hypothesized SPS would be responsive to changes in perceived pain, and that patients and caregivers would prefer SPS over FPS-R.

PATIENTS AND METHODS

Study design

This prospective observational cohort study was approved by the University of Alberta’s Health Research Ethics Board.

Participants

Study recruitment occurred at the Stollery Children’s Hospital paediatric emergency department (ED) (Edmonton, Alberta) from November 2014 to February 2017. Children aged 3 to 12 years old, presenting to the ED with an acutely painful injury (<72 hours old) were screened by a trained research assistant (RA). Patients requiring analgesia at the time of enrollment were included in the study. Patients were excluded if the child had emergent medical needs, a chronic painful condition, an absent legal guardian, or if the patient or caregiver was not fluent in English.

Measures

We aimed to validate SPS, as compared to FPS-R (Figure 1A and 1B). FPS-R is widely accepted, validated, and recommended for use in research and practice for children aged 4 to 12 years (12,15,18,19). FPS-R consists of six faces (scored 0, 2, 4, 6, 8, and 10) (15). The Baxter Retching Faces (BARF) scale is a validated six face scale (scored 0, 2, 4, 6, 8, and 10) used to assess nausea in children; we utilized the BARF scale to test discriminant validity (21).

Figure 1.

(A) Stoplight Pain Scale. Reproduced with permission. © 2014 Booster Shot Media, Inc. and Amy L. Drendel, DO. (B) Faces Pain Scale-Revised. Reproduced with permission. FPS-R © 2001, International Association for the Study of Pain. All rights reserved.

Procedures

Participants were recruited as a convenience sample from 3 pm to 11 pm, 7 days per week. Informed consent, and assent for children age ≥ 7 years, was obtained before enrolment. Caregivers provided information regarding demographics, past medical history, previous ED visits, previous hospitalizations, and pre-hospital analgesia. Both the child and their caregiver were asked to independently provide scores using three separate tools: FPS-R, SPS, and BARF. Pain assessment occurred at the following specific intervals: before analgesia administration in the ED (i.e., initial measurement), 30 ± 10 minutes after analgesia administration, and for painful procedures. Before discharge, patients and caregivers were individually asked to indicate their preferred pain assessment tool for use at home. Every effort was made to elicit the child and their caregiver’s pain and preference responses independent of one another; caregivers were asked to refrain from helping their child answer questions. Data were collected and stored on Research Electronic Data Capture (REDCap), a secure electronic data capture tool (22). Enrolled participants were offered a $5 gift card.

Analysis

We determined the validity of SPS by evaluating convergent validity, discriminative validity, responsivity, and reliability. Subgroups for post hoc analyses included children aged 3 to 5 years old versus 6 to 12 years old and previous hospitalization versus no previous hospitalization. We removed patients for whom missing data precluded the determination of validity measures.

Convergent validity refers to the degree to which two different tools that are supposed to measure the same thing (in this case, pain) produce similar results. FPS-R was used as the standard pain assessment tool against which SPS was compared to for convergent validity. Discriminative validity is the degree to which a tool (i.e., SPS) is actually measuring the construct it is meant to measure (i.e., pain), and not something else (i.e., nausea). By comparing it to the BARF scale, we tested if SPS was specifically measuring pain. Construct validity was assessed by measuring responsivity to pain-relieving (i.e., provision of analgesia) or pain-producing events (i.e., painful medical procedures). We assessed this after analgesia administration and for painful procedures. To compare the three-faced SPS pain assessment tool to the six-faced FPS-R and BARF, it was predetermined that scores of 0, 2, and 4 (none to mild) for FPS-R should correspond to ‘Green’, 6 (moderate) to ‘Yellow’, and 8 and 10 (severe) ‘Red’ SPS scores (23).

To assess convergent, discriminative, and construct validities, we used Cohen’s κ as a measure of agreement between two scales. Point estimate for κ and 95% confidence intervals were calculated for patients and caregivers separately at the three specified assessment points.

To assess directionality of SPS, ensuring sensitivity to change in perceived pain, we compared SPS scores at each of the three assessment points. We expected the percentage of patients and caregivers who selected ‘Red’ to decrease after the analgesia administration and increase immediately after the painful procedure. All statistical analyses were performed using SAS Ver. 9.4 (SAS Institute Inc., Cary, NC). Agreement, as measured by the κ statistic was categorized as follows: 0.0 to 0.20 was considered poor agreement, 0.21 to 0.40 fair agreement, 0.41 to 0.60 moderate, 0.61 to 0.80 good, and 0.81 to 1.00 very good agreement (24,25).

RESULTS

Demographic characteristics

Two hundred and sixty-one patients were screened for enrollment; 2 withdrew from the study, 9 had incomplete data, and 23 ultimately did not receive analgesia. Therefore, 227 children were enrolled in the study (Table 1). In our study, 16.4% (10/61) of 3 to 5-year olds and 14.5% (24/166) of 6- to 12-year olds received analgesia before ED registration. Pre-hospital analgesia was administered by parents (27/34), Emergency Medical Services personnel (1/34), the patient (1/34), or others (5/34). In total, 55.9% (19/34) of children received ibuprofen, 41.2% (14/34) acetaminophen, and 2.9% (1/34) inhaled nitrous oxide. Five children received a second dose of analgesia before ED arrival; in all cases, it was ibuprofen or acetaminophen. Of the 138 children providing pain measures for medical procedures, reported procedures included x-ray (n=98), suturing (n=22), medical examination (n=8), backslab/cast application (n=4), and other (n=6).

Table 1.

Demographics characteristics of children and their caregivers.

	3–5 years old (n = 61)	6–12 years old (n = 166)	All children (n=227)
Child’s age (years)
Mean ± SD	4.4 ± 0.9	9.5 ± 2.1	8.1 ± 2.9
Child’s sex
Female n(%)	27 (44.3)	87 (52.4)	114 (50.2)
Child’s previous hospital experience n(%)
Prior ED visit	35 (57.4)	136 (81.9)	201 (88.6)
Prior admission	9 (14.8)	44 (26.5)	53 (23.4)
Prior admission, painful complaint	7 (11.5)	21 (12.7)	28 (12.3)
Presenting complaint
Upper limb injury	25 (41.0)	77 (46.4)	102 (44.9)
Lower limb injury	12 (19.7)	43 (25.9)	55 (24.3)
Laceration	16 (26.2)	19 (11.4)	35 (15.4)
Head injury/pain	3 (4.9)	15 (9.0)	18 (7.9)
Other MSK injury*	5 (8.2)	6 (3.6)	11 (4.8)
Facial trauma	0 (0)	5 (3.0)	5 (2.2)
Burn	0 (0)	1 (0.6)	1 (0.4)
Caregiver’s age (years)**
Mean ± SD	35.1 ± 4.6	39.9 ± 5.9	38.5 ± 6.0
Caregiver’s sex
Female n(%)	41 (68.9)	126 (75.9)	168 (74.0)

ED Emergency department; SD Standard deviation.

*MSK = musculoskeletal.

**Caregiver’s Age: 3–5 years old n=61, 6–12 years old n=134, All children n=187, n= 32 chose not to disclose their age.

Convergent validity

Children’s self-reported pain before analgesia administration demonstrated fair agreement (κ=0.28, 95% CI 0.20 to 0.36) comparing SPS and FPS-R; caregivers demonstrated poor agreement (κ=0.14, 95% CI 0.07 to 0.21). When rating pain 30 minutes after analgesia, children’s self-reported pain showed fair agreement (κ=0.25, 95% CI 0.16 to 0.34) between SPS and FPS-R; caregivers demonstrated poor agreement (κ=0.16, 95% CI 0.09 to 0.23). Agreement of reported pain was fair directly following a painful procedure for both children (κ=0.26, 95% CI 0.15 to 0.36) and caregivers (κ=0.23, 95% CI 0.13 to 0.33).

Table 2 presents caregiver and child agreement when using SPS and FPS-R. When initially measured (before analgesia), child and caregiver assessment utilizing SPS showed fair agreement (κ=0.28, 95% CI 0.17 to 0.38). Child and caregiver scores utilizing FPS-R showed fair agreement (κ=0.22, 95% CI 0.14 to 0.29).

Table 2.

Child and Caregiver Agreement between each pain assessment tool used, at various assessment points

	Simple κ value	ASE	95% confidence interval	Interpretation
At initial presentation to the ED
SPS	0.28	0.05	0.17–0.38	FAIR
FPS-R	0.22	0.04	0.14–0.29	FAIR
BARF	0.19	0.04	0.11–0.27	POOR-TO-FAIR
30 minutes following analgesia administration
SPS	0.38	0.05	0.28–0.48	FAIR-TO-MODERATE
FPS-R	0.21	0.04	0.13–0.29	POOR-TO-FAIR
BARF	0.29	0.05	0.19–0.38	FAIR
Directly following a painful procedure
SPS	0.46	0.06	0.34–0.59	FAIR-TO-MODERATE
FPS-R	0.28	0.05	0.18–0.39	POOR-TO-FAIR
BARF	0.24	0.06	0.11–0.36	POOR-TO-FAIR

ASE Asymptomatic Standard of Error; BARF Baxter Retching Faces Scale; ED Emergency department; FPS-R Faces Pain Scale- Revised; SPS Stoplight Pain Scale

Discriminative validity

Thirty minutes after analgesia administration, caregivers and children had fair agreement between their SPS scores (κ=0.38, 95% CI 0.28 to 0.48); agreement was higher directly following painful procedures (κ=0.46, 95% CI 0.34 to 0.59). Agreement between child and caregiver scores for FPS-R 30 minutes after analgesia administration (κ=0.21, 95% CI 0.13 to 0.29) and directly after painful procedures (κ=0.28, 95% CI 0.18 to 0.39) remained the same.

Discriminative validity was assessed by comparing SPS to BARF scores. When initially assessed before analgesia, children (κ=0.05, 95% CI 0.00 to 0.11) and caregivers (0.00, 95% CI −0.04 to 0.04) were found to have no agreement between SPS and BARF. After analgesia administration, there was poor agreement between SPS and BARF for both children (κ=0.12, 95% CI 0.05 to 0.20) and caregivers (κ=0.08, 95% CI 0.03 to 0.14). Scores measured directly after painful procedures also showed poor agreement between SPS and BARF, both for children (κ=0.07, 95% CI 0.01 to 0.13) and caregivers (κ=0.08, 95% CI 0.01 to 0.15).

Responsivity

At initial presentation to the ED, 41.2% (93/226) of children and 53.7% (122/227) of caregivers selected Red on SPS, while 43.8% (99/226) of children and 37.4% (85/227) of caregivers selected Yellow. Thirty minutes after receiving analgesia, 24.7% (54/219) of children and 19.7% (44/223) of caregivers selected Red using SPS, representing an overall drop from pre-analgesia. Immediately after painful procedures, 23.5% (31/132) of children and 23.5% (31/132) of caregivers selected Red. A similar pattern was found for the pain reported using FPS-R scale. At initial presentation to the ED, the mean score using FPS-R was 5.1 ± 3.0 for children and 5.2 ± 2.5 for caregivers. The mean FPS-R score dropped 30 minutes after receiving analgesia to 3.4 ± 2.8 for children and 3.1 ± 2.3 for caregivers. For painful procedures, it was a mean of 3.3 ± 3.1 for children and 3.5 ± 2.8 for caregivers. At initial presentation, of the 117 caregivers who chose moderate–severe pain scores with FPS-R, 98.2% (n=115) chose Yellow/Red pain SPS scores. Interestingly, of the 227 caregivers who chose mild pain scores, 40.5% (n=92) still chose Yellow/Red SPS scores. Thirty minutes after analgesia, of the 47 caregivers who chose moderate–severe pain scores with FPS-R, 100% chose Yellow/Red pain SPS scores. Of the 176 caregivers who chose mild pain scores with FPS-R, 56.3% (n=99) still chose Yellow or red SPS scores.

The subgroup scores for both children and caregivers are presented in Supplementary Figures 2–4. Figure 2A and 2B suggests that caregiver assessments using the FPS-R showed fairly frequent under-estimation of the children’s pain. In contrast, when using the SPS, caregiver report was sometimes higher or lower, depending on age and prior hospitalization status of the child (Figure 3A and 3B).

Caregiver and child scale preference

76.6% (36/47) of children aged 3 to 5 years and 71% (103/145) of children aged 6 to 12 years preferred SPS. Further, 72.6% (33/46) of children who had not been previously hospitalized and 71.7% (106/146) of children who had been previously hospitalized preferred SPS.

Sixty per cent (118/196) of caregivers preferred SPS over FPS-R when measuring their child’s pain. This preference varied with patient age: 77.8% (12/44) for 3 to 5 years old and 53.5% (76/142) for 6 to 12 years old. Further, 59.6% (28/47) of caregivers preferred SPS over FPS-R when their child had been previous hospitalized, and 60.4% (90/149) preferred SPS when their child had never been hospitalized.

DISCUSSION

SPS demonstrated fair convergent validity and excellent discriminative validity with children. SPS demonstrated fair agreement between children and caregivers’ scores, with improved agreement after repeated use of SPS. Further, SPS was preferred over FPS-R by both children and caregivers.

Pain assessment tools should be reliable, valid, and age-appropriate (15,16,21). They should also be simple and practical for both children and caregivers to use (15,16,21). While there are a number of tools available to assess paediatric pain (10,16,18), conventional pain scoring tools do not directly guide pain management (26). Pain is complex and multifactorial, and using an assessment tool designed for research might fail to capture a child’s desired response to their pain. SPS’s simplification of pain measurement to three action-associated scores has the potential to empower children and caregivers to advocate for appropriate pain management (15). Caregivers in our study ‘erred on the side of treatment’ (i.e., Yellow/Red scores with SPS) even for mild FPS-R pain scores, suggesting a potential disconnect between how mild pain is defined in clinical practice and caregiver goals of care for their child.

SPS aims to bridge the gap between measuring perceived pain and determining how the child wishes to have their symptoms managed. As SPS measures the pain experience in an action-oriented context, unlike traditional pain assessment tools, it can be difficult to compare SPS to other tools. This may explain why our study showed only fair convergent validity between SPS and FPS-R for children, despite child and caregiver preference for SPS.

When a child is unable to communicate their personal experience with pain, the caregiver’s assessment of their child’s pain is often used as a surrogate marker of pain recognition and quantification. However, caregivers may not always accurately assess their child’s pain (2,7). This is especially true for younger children (1,5), where currently used pain scales are known to be less accurate (19,27). Children and their caregivers also use different language and life experiences, including prior hospitalizations, to describe pain-related constructs (20). The simple language and recognizable ‘stoplight construct’ of SPS may help overcome language-based challenges and may explain why children and caregivers showed better agreement and less variability between their scores when using SPS. With repeat use, child and caregiver SPS score agreement improved from fair to moderate. This further suggests that SPS has the potential for greater accuracy between child and caregiver scores after users are provided with brief training and a small amount of practice. Further research is needed to explore the utility of SPS in other clinical settings and at-home, to confirm these hypotheses.

The use of a pain tool is known to increase frequency of pain measurements, guide treatment (1,5), and improve satisfaction (14). Despite this, pain documentation is poor and likely contributes to poor pain management for children in the ED (1,5). Using a simplified, preferred pain assessment tool might promote more frequent assessments of children’s pain, thereby enhancing overall pain management and experience. At-home pain management could also be improved by discharging children home with a simple pain tool.

Limitations

When asking which pain assessment tool was preferred, the original script failed to explicitly limit preference to SPS and FPS-R; some early respondents listed BARF as their preferred tool and were removed from these analyses. Patients were only exposed to the pain assessment tools for a short duration; extended analysis of the tool’s responsiveness would have been preferred. The study was limited to a single setting, in one centre, and with English-speaking patients.

CONCLUSION

SPS demonstrates fair agreement with FPS-R for children and fair to moderate agreement between children and caregivers; agreement improved with repeat use. SPS is family-friendly, action oriented and simple to use, and was preferred over FPS-R by children and caregivers; it may have a role in empowering direct child and family involvement in pain management. SPS has the potential to enhance pain management at home, pre-hospital, and in the hospital, but requires further study at this time.

Presentations: This work was presented at the Canadian Association of Emergency Physicians Annual Meeting (Calgary, Alberta) on 29 May 2018 as a poster presentation.