Development and psychometric properties of the Swedish ALPS-Neo pain and stress assessment scale for newborn infants

Abstract

Aim

To validate and evaluate the psychometric properties of the ALPS-Neo, a new pain assessment scale created for the continuous evaluation of pain and stress in preterm and sick term infants.

Methods

A unidimensional scale for continuous pain, Astrid Lindgren Children's Hospital Pain Scale (ALPS 1), was developed further to assess continuous pain and stress in infants treated in the neonatal intensive care unit (NICU). The pain scale includes observations of five behaviours. A manual was created, clarifying the scoring criteria. An internal and an external panel assessed face validity. Psychometric properties were evaluated in three different steps. Inter-rater reliability was estimated from video-based assessments (n = 625) using weighted kappa statistics (test I). Inter-rater reliability was further evaluated in test II (n = 125) and test III (n = 96) by real-time assessments using the intraclass correlation coefficient (ICC) and Cronbach's alpha.

Results

The final inter-rater reliability (test III) was assessed as good with ICC 0.91 for the total score and 0.62–0.81 for the five items. Cronbach's alpha showed 0.95 for the total score.

Conclusion

ALPS-Neo is a new assessment tool for optimising the management of pain and stress in newborn infants in the NICU. It has proved easy to implement and user-friendly, permitting fast, reliable observations with high inter-rater reliability.

Key notes

• Systematic and repeated assessment of pain and stress are the basis of adequate pain management in the NICU and objective assessment methods, validated and psychometrically tested for the specific population and type of pain, are needed.

• ALPS-Neo is a unidimensional pain and stress assessment scale, designed to be used with physiological parameters.

• We found the ALPS-Neo easy to implement and user-friendly, permitting fast, reliable observations with high inter-rater reliability.

Introduction

Pain assessment in preterm infants is complex and difficult due to their limited capacity to express pain and stress, which is related to their physiological and neurobiological immaturity. Furthermore, preterm infants cannot maintain their physiological and behavioural activation if the pain becomes persistent (1). Although more immature in their pain responses, preterm infants are also more sensitive to the adverse effects of pain than older infants (2,3). It is therefore important that healthcare professionals have access to validated pain assessment instruments that have been developed for the specific population they intend to observe and which allow them to continuously observe and interpret infants' subtle cues to pain and stress so that they can respond effectively.

Newborn infants admitted to the neonatal intensive care unit (NICU) are often exposed to repeated painful procedures as well as prolonged pain and stress (4). Not only can distressing and painful procedures cause acute pain and stress, which may have a negative impact on both the infants' clinical condition and likelihood of improvement, but repeated pain and stress may also alter brain development (5,6) and pain responses later in life (2). As the evidence of cortical activation from procedural pain in preterm infants was demonstrated (7–10), it has become even more urgent that reliable and user-friendly pain assessment instruments are developed so that these patients can be provided with adequate pain treatment.

A considerable number of pain scales for assessing pain in preterm and term newborn infants are available. Many of them have primarily been designed and validated for the assessment of procedural pain, for example the Premature Infant Pain Profile (PIPP) (11) and the Behavioural Indicators of Infant Pain (BIIP) (12). Only a few scales have been designed and validated for the assessment of prolonged pain, including the Échelle Douleur Inconfort Nouveau-Né (EDIN) (13) and the Comfortneo scale (14).

Despite the fact that systematic pain assessment is now considered to be the basis of adequate pain management, little progress has been made on incorporating pain assessment as a clinical routine in neonatal intensive care units in Sweden (15). To meet the needs for adequate pain and stress management in infants treated in the NICU and to overcome the difficulties of implementing pain and stress assessment in the clinical NICU setting, an instrument designed for easy and accurate assessment of prolonged pain and stress in this vulnerable population is required. Ideally, this would be a scale that can be used as an additional vital parameter to safely, quickly and repeatedly monitor pain and stress to optimise the treatment of these infants' pain.

Aim

The aim of the present study was to validate and evaluate the psychometric properties of a new pain assessment scale created for continuous pain and stress evaluation in both preterm and sick term newborn infants.

Ethical Consideration

The study was conducted in accordance with the Helsinki Declaration (2013) and approved by the Regional Ethical Review Board in Lund (Ref. no. 389/2004). All parents provided their written or verbal consent for their infants to participate in the study.

Method and Result

Design

The study had an explorative and correlational design and included several steps (see below). A pre-existing, but not validated, pain assessment scale for term newborn infants, the Astrid Lindgren Children's Hospital Pain Scale (ALPS 1), was further developed to better match the responses to pain and stress of newborn infants treated on the NICU. The new pain and stress assessment scale, the Astrid Lindgren and Lund Children's Hospitals Pain and Stress Assessment Scale for Preterm and sick Newborn Infants (ALPS-Neo) (Fig.1), was evaluated to determine its psychometric properties. To describe the process of revising ALPS 1 to create ALPS-Neo and the following psychometric evaluations, we have chosen to combine the method and results as this will provide a chronological view of the steps involved.

Figure 1.

Astrid Lindgren and Lund Children's Hospitals Pain and Stress Assessment Scale for Preterm and sick Newborn Infants ALPS-Neo.

Astrid Lindgren Children's Hospital Pain Assessment Scale for term neonates (ALPS 1)

Astrid Lindgren Children's Hospital Pain Scale (ALPS 1) is a five-item unidimensional scale for the continuous evaluation of prolonged pain and includes scores for facial expression, breathing pattern, tone of extremities, hand and foot activity and level of activity. The score awarded to each item ranges from zero to two, with a total sum varying between zero and ten. ALPS 1 has been used in many hospitals in Sweden over the last decade and is popular because of its simplicity, but it has not been psychometrically tested (personal communication with the originator, Larsson B).

Step 1: Adapting ALPS 1 to ALPS-Neo

ALPS-Neo consists of the same five items as ALPS 1. Each scoring criteria in ALPS 1 was revised and adjusted to also take into consideration preterm infants' behavioural responses to pain and stress, as based on the Newborn Individualized Development Care and Assessment Program (NIDCAP) (16–18). Holsti et al. (18) found eight NIDCAP behavioural responses that increased significantly during a painful procedure. These behaviours have been integrated in the scoring criteria of ALPS-Neo. A written manual was developed for ALPS-Neo (Appendix S1) defining and clarifying the scoring criteria for guidance in the clinical assessment situation (16,17).

Step 2: Face validity of ALPS-Neo

Internal panel of experts

To assess clinical face validity, an internal panel of experts was used, consisting of 25 registered nurses, specialised in paediatric and/or intensive care nursing with experience in neonatal care, and one senior neonatologist. The participants in the panel all worked at the same level III NICU in Sweden (NICU A). The nurses' overall NICU experience ranged between four and 31 years (median 13). They all had extensive experience in assessing preterm and sick term newborn infants' behaviour and responses as the NIDCAP model had been implemented in NICU A in the middle of the 1990s.

ALPS-Neo and the manual were presented to the panel members who were asked to individually assess whether the description of each scoring criterion for the five items was related to pain and stress in infants treated in the NICU and whether they considered the descriptions to be clear and easy to understand. The two first authors (AK and PL) discussed the comments from the panel with an experienced paediatric nurse who was responsible for clinical pain and stress management in NICU A (HA), which resulted in minor revisions to both the scale and the manual.

External panel of experts

An external panel, consisting of three registered nurses, each with a PhD in nursing and extensive knowledge of neonatal pain and pain assessment in infants, was used to assess research-based face validity. The first and second versions of ALPS-Neo and the manual were presented to the panel members, who were asked to perform the same evaluation as the internal panel. Minor suggestions regarding the wording, the language or definitions, the scoring criteria and the manual were provided by the external panel and taken into consideration.

Step 3: Inter-rater reliability test I – video-based assessments in NICU A

We used 25 video-recorded sequences (each lasting 1–2 min) of 18 different preterm infants as the basis for assessing the inter-rater reliability. The characteristics of these infants are described in Table1.

Table 1.

Infant characteristics in inter-rater reliability tests I–III. Figures are ranges and numbers

	Test I NICU A (n = 18)	Test II NICU A (n = 28)	Test III NICU B (n = 40)
Gestational age at birth, weeks and days	23 0/7–31 1/7	23 2/7–39 6/7	25 3/7–41 5/7
Postmenstrual age at assessment, weeks and days		24 6/7–43 2/7	29 2/7–45 3/7
Term infants, n	0	3	11
Preterm infants, n	0	10	22
Very preterm infants (GA < 32 weeks), n	4	6	5
Extremely preterm infants (GA < 28 weeks), n	14	9	2
Small for gestation age (SGA), n	4	2	5
Treated with CPAP, n	8	5	9
Treated with ventilator, n	10	6	0
Analgesic treatment	3	4	0
Postoperative care	1*	4†	0

^*Ductus arteriousus ligation.

^†Ductus arteriousus ligation (n = 2), abdominal surgery (n = 2).

Twenty-five nurses with a median NICU experience of 3 years (range 0.5–33) were randomly recruited from NICU A for these assessments. A few days before the video assessments, they were given an introduction to ALPS-Neo and the manual by one of the first authors (PL). As part of the preparation for the video assessments, they were again instructed in the use of ALPS-Neo by HA. They observed the video sequences and made the assessments individually according to ALPS-Neo. An independent nurse was present during the video demonstration in case of any technical problems, but she was not involved in any discussions about the outcome of any of the sequences assessed.

The inter-rater reliability for each video sequence assessment was calculated using weighted kappa (k_w) statistics, using the software program VassarStats: Website for Statistical Computation (http://vassarstats.net/). The value of k_w can be interpreted as follows: <0.2, poor agreement; 0.21–0.40, fair agreement; 0.41–0.60, moderate agreement; 0.61–0.80, good agreement; and 0.81–1.00, very good agreement (19).

The results demonstrated a fair inter-rater reliability for most of the items, as well as for the total score (k_w between 0.15 and 0.62). As the inter-rater reliability was judged to be unsatisfactory, a new face validity test with a different approach was conducted, see step 4.

Step 4: Face validity II

Five experienced neonatal nurses from NICU A were recruited for the second face validity test. Two of them had been involved in the first face validity procedure. Their NICU experience ranged between six and 30 years (median 29). They were asked to individually assess five of the 25 video sequences (selected by AK and PL), chosen to represent a wide range of the NICU population, and to describe on a score sheet how the observed infants' behaviour influenced their scoring. The nurses were allowed to look at the sequences repeatedly and to use the manual continuously.

Thereafter, the first authors scrutinised the observers' comments and assessments in relation to each video sequence. All items were declared to be relevant, but the scoring criteria for each item were not clearly differentiated. Minor changes were made, mainly by reducing the number of words for some of the scoring criteria.

Step 5: Inter-rater reliability test II – real-time assessment in NICU A

Twenty nurses, with two- to 35-year NICU experience (median 15), worked in ten pairs in NICU A, where they conducted real-time assessments of 28 infants on a total of 125 occasions. The infants' characteristics are presented in Table1. Before starting, the nurses were individually introduced to the adjusted ALPS-Neo and asked to consider the score in relation to the infants' other vital signs. An independent nurse guided the observers to various infants in the NICU.

The intraclass correlation coefficient (ICC) was used to determine inter-rater reliability. An ICC > 0.8 was considered to be very good; 0.65–0.80 was assessed as good, and 0.35 to 0.65 as moderate. The internal consistency was calculated using Cronbach's alpha, and a coefficient >0.80 was interpreted as very good (20). The software program SPSS™ (version 18; IBM Corporation, Armonk, NY, USA) was used when analysing both ICC and Cronbach's alpha. The result showed increasing inter-rater agreement but was still too low to be acceptable (Table2).

Table 2.

Intraclass correlation coefficient (ICC) and Cronbach's alpha in reliability test II and III

Item	Inter-rater reliability II			Inter-rater reliability III
	ICC	95% CI	Cronbach's alpha	ICC	95% CI	Cronbach's alpha
Facial expression	0.402	0.244–0.539		0.615	0.472–0.726
Breathing pattern	0.401	0.244–0.539		0.668	0.540–0.766
Arm/leg muscle tone	0.367	0.204–0.511		0.755	0.653–0.831
Hand/foot activity	0.452	0.299–0.583		0.783	0.690–0.850
Level of activity	0.467	0.318–0.593		0.812	0.730–0.871
Total score	0.610	0.487–0.709	0.758	0.908	0.864–0.938	0.953

The manual was reviewed to further clarify the main items' scoring criteria. It was concluded that another evaluation of ALPS-Neo should be performed at a NICU in which the instrument had not been used before as there was a risk that the staff at NICU A were by now too familiar with the instrument and would therefore not recognise the changes that had been made.

Step 6: Inter-rater reliability III – real-time assessment in NICU B

Twelve pairs of neonatal nurses (n = 24) with one- to 40-year (median 11) NICU experience were recruited from another Swedish level II NICU (NICU B). In NICU B, pain assessment was not part of the ordinary care procedures; however, the concept of NIDCAP had been implemented in this NICU, and NIDCAP observations were performed on a more regular basis than in NICU A.

Before starting the assessments, the nurses received a 30-min introduction to general pain evaluation and the details of ALPS-Neo, corresponding to step five. They performed 96 assessments on 40 infants, 75 on preterm infants and 21 on sick term newborn infants. Infant characteristics are provided in Table1. One of the first authors (AK) observed the assessors during the scoring procedures but was not available to answer questions.

Intraclass correlation coefficient (ICC) was used to determine inter-rater reliability, and the internal consistency was calculated using Cronbach's alpha (20). The inter-rater reliability was assessed as good. It varied between 0.62 and 0.81 for the five items, and the total score was 0.91. Cronbach's alpha was 0.95 for the total score (Table2).

Discussion

ALPS-Neo demonstrates high inter-rater reliability and appears to be a valid, reliable unidimensional scale that can be used for the bedside assessment of pain and stress as an additional vital parameter for infants treated in the NICU. With ALPS-Neo, symptoms of pain and stress are monitored and registered simultaneously with other physiological parameters, and this makes it possible to continuously evaluate pain and the need for analgesics or comfort measures as part of an overall clinical context.

Clinical application

Implementation of pain and stress assessments in the NICU is still a challenge (21), and there may be several reasons for this. Some scales are not user-friendly enough and are therefore not suitable for frequent and repeated use in day-to-day clinical practice. This may be one explanation for the slow progress in implementing pain assessment as a clinical routine in Swedish NICUs, with 50% of NICUs still not using any structured method of pain assessment (15). When used together with the manual that clarifies the scoring criteria, ALPS-Neo is, according to our experience, considered user-friendly and easy to score and easy to incorporate into the daily care routine. ALPS-Neo is a scale that enables caregivers to respond promptly and repeatedly to an infant's needs and will reduce the time an infant experiences pain and/or stress and thereby increases the infant's well-being. The assessment of muscle tone as high or low is included in the highest scores of three of the five items in ALPS-Neo. Most pain assessment scales only take high muscle tone in consideration; however, preterm and sick newborn infants might have difficulty maintaining muscle tone when they experience continuous distress and/or pain. Muscle tone can be difficult to evaluate as it may be either increased or decreased, and there is a risk that low muscle tone is misinterpreted as indicating well-being, despite the infant actually suffering from illness, pain or stress.

New knowledge about the long- and short-term negative effects of pain and stress (22), together with the risks of the neuro-apoptotic effects of anaesthetic and sedative drugs on the vulnerable immature brain (23), has placed even more emphasis on the need for optimal pain and stress management, including supportive care. Individualized developmentally supportive care in accordance with NIDCAP has proven to be an effective means of providing the best possible protection and optimal developmental conditions for the immature newborn brain during care in the NICU (24). It has also been shown to reduce the stress associated with acute procedures (25,26). The foundation of this model emphasises the important role of the caregiver in being sensitively attuned to the signs of the infant's well-being, as well as their pain and stress, which matches well with purpose of continuous pain and stress assessment.

As there is a lack of both a gold standard of objective pain assessment and validated multimodal analgesia, a balanced approach is currently recommended. Nonpharmacological intervention should be performed initially, and if needed and based on systematic assessments, this should be followed by titrated administration of sedatives and analgesics with reassessment to adjust treatment (21,27). The implementation of an algorithm that aims to provide an even more structured pain treatment would be in line with current pain treatment policy (28) but has not been validated for ALPS-Neo and, in our unit, has so far only been used within research projects (29).

Psychometric evaluation – strengths and limitations

In the present study, we used established methods, such as expert panels, to develop the scale and also appropriate methods for the psychometric evaluation of the new scale for assessing neonatal pain and stress (20). To overcome the risk that a unidimensional pain and stress scale based on infants' behavioural signs might allow for different interpretations, a manual was developed to further guide the assessments. Video-recorded sequences of preterm infants were used as a basis for the first evaluation of the inter-rater reliability. This was primarily related to an ethical issue, to minimise the influence the assessments might have on the families and the healthcare professionals. Another reason for the use of video sequences was that it provided the opportunity for many different nurses to assess the same sequences. However, the video sequences focused only on the behaviour of the infants, and assessing video sequences is difficult if you cannot simultaneously display the physiological parameters and consider the environmental aspects. This may be one reason for the unsatisfactory result of the first reliability test. As ALPS-Neo is to be regarded as an additional vital parameter, the subsequent reliability tests were conducted on real-time assessments. Another reason for the unsatisfactory result may be that the descriptions were not sufficiently distinct and therefore invited variable assessments. This consideration led to further development of both the scale and the manual. The ALPS-Neo manual increased the inter-rater reliability and was perceived to be a useful document during the introduction, implementation and application of the scale.

The initial validation of ALPS-Neo indicates that this scale is sufficiently sensitive to be used for the repeated evaluation of pain and stress in preterm and term sick newborn infants cared for in the NICU. In contrast to term infants cared for postoperatively in the PICU where ALPS-1 is used, term infants in the NICU are often cared for during the initial post-natal period of respiratory and circulatory transition. Similar to preterm infants, these infants may have a limited ability to express that they are in a state of stress or pain due to severe illness or physiological instability. However, the sample size of term infants was low, which must be considered a weakness of this study, and further investigation in this population is needed.

Another limitation of our study is that we did not assess the changes in pain response before and after a painful procedure, that is construct validity. In Sweden, blood samples are commonly drawn from central lines and venous puncture to reduce the adverse effects of pain and stress in preterm infants (2,3). From an ethical standpoint, we found it impossible to conduct construct validity using capillary puncture. However, it may be possible to use other painful procedures such as endotracheal tube suctioning to assess construct validity.

Furthermore, we did not compare ALPS-Neo with another validated pain scale, that is concurrent validity. The EDIN scale (13) is now the most well-known scale for the assessment of prolonged pain in preterm infants. However, when the work to validate ALPS-Neo began, there was no scale for the assessment of prolonged pain available for a Swedish setting. The EDIN is a unidimensional scale, but in contrast to ALPS-Neo, it retrospectively summarises the infant's behaviour over a period of several hours. With such a strategy, there is a risk that the evaluation of infant pain and stress may not be very accurate as the infant's condition may change rapidly during intensive care. Continuous assessment and reassessments are crucial to be able to give adequate nonpharmacological and pharmacological interventions when needed. In addition, some behavioural items in the EDIN scale (13) such as quality of contact appeared not to be relevant for preterm and/or sedated infants, further motivating the development of ALPS-Neo. The EDIN item consolability is not included as an individual item in ALPS-Neo. However, this item is judged indirectly as ALPS-Neo is scored repeatedly to follow-up infants' pain and stress. If the infant is responding to a nonpharmacological intervention, as indicated by lowering of the ALPS-Neo score, this can be interpreted as a sign of consolability by the intervention. If the infant still has a high total score, repeated assessments and reassessments are performed, and pharmacological intervention is considered. We used the Swedish version of EDIN concomitantly with ALPS-Neo in a randomised controlled trial evaluating premedication for endotracheal intubation in 34 preterm infants (29). During a 24-h observation period following the intubation, EDIN and ALPS-Neo were scored repeatedly, ALPS-Neo every 30 min and EDIN every fourth hour, six common time points. No significant scoring differences were found (unpublished data) (29).

Conclusion

ALPS-Neo is a user-friendly and easily implemented pain and stress assessment scale with high inter-rater reliability. It permits fast, repeated and reliable behavioural observations of both preterm and term sick newborn infants. Together with other vital parameters, ALPS-Neo may prove to be valuable in optimising pain treatment for seriously ill infants in the NICU.

Further Research

ALPS-Neo needs to be tested in larger cohorts of preterm and term infants. To assess concurrent validity, ALPS-Neo should be used simultaneously with another pain scale for assessing prolonged pain that has been validated in the same patient population. To add further validity, construct validity also needs to be performed, preferably in conjunction with a common painful intervention such as tracheal suction. To further improve pain and stress management in sick newborn infants, an algorithm that aims to support structured pain assessment and treatment needs to be implemented and tested.

Supporting Information

Additional Supporting Information may be found in the online version of this article:

Appendix S1

Manual for ALPS-Neo pain and stress assessment scale.