Diagnostic Values of the Critical Care Pain Observation Tool and the Behavioral Pain Scale for Pain Assessment among Unconscious Patients: A Comparative Study

Abstract

Background

Pain assessment in unconscious patients is a major challenge for healthcare providers. This study aims to compare the diagnostic value of the critical-care pain observation tool (CPOT) and the behavioral pain scale (BPS) for pain assessment among unconscious patients.

Materials and methods

This cross-sectional study was conducted in 2019. Forty-five unconscious patients were selected randomly from four general intensive care units (ICUs) in the north of Iran. The discriminant validity of CPOT and BPS were evaluated for pain during a nociceptive and a nonnociceptive procedure. For reliability assessment, interrater agreement was obtained using Lin's concordance correlation coefficient and weighted kappa coefficient.

Results

Patients who had been hospitalized in ICU due to surgery or trauma (57.70%) or medical problems (42.30%) were studied. During the nociceptive procedure, the mean scores of CPOT and BPS and all their dimensions, except for the compliance with ventilator dimension, were significantly greater than the nonnociceptive procedure (p <0.05) although the effect size of both instruments was small (0.32 vs 0.18). The Lin's concordance correlation coefficient in nonnociceptive and nociceptive procedures was respectively 0.67 and 0.62 for CPOT and 0.74 and 0.88 for BPS.

Conclusion

CPOT and BPS have acceptable discriminant validity in differentiating nonnociceptive and nociceptive procedural pain although the effect size of CPOT is larger than that of BPS. Although both instruments have low reliability, the reliability of BPS is better.

How to cite this article

Nazari R, Froelicher ES, Nia HS, Hajihosseini F, Mousazadeh N. Diagnostic Values of the Critical Care Pain Observation Tool and the Behavioral Pain Scale for Pain Assessment among Unconscious Patients: A Comparative Study. Indian J Crit Care Med 2022;26(4):472–476.

Highlights

• Valid and reliable behavioral pain assessment instruments for unconsciousness patients is a necessity.

• The Persian version of the critical-care pain observation tool (CPOT) and behavioral pain scale (BPS) have acceptable discriminant validity, though the effect size of CPOT is larger.

• Both instruments have low reliability; however, the reliability of BPS is better.

Background

Pain is a common problem among patients in ICUs and suggests the necessity of preventive measures.^1,2 Studies show that most patients in ICU suffer from pain.^3,4 The most common causes of pain in patients in ICU are surgical interventions, posttrauma pain, and pain associated with procedures such as arterial line placement, chest tube removal,⁵ airway suctioning, and during wound care.⁶ Besides, patients in ICU may experience pain during usual nonnociceptive care and even at rest.⁷

Undiagnosed and unmanaged pain can result in complications and seriously affect patients’ condition.¹ For instance, unmanaged pain can cause tachycardia, altered immune responses, excessive release of catecholamine, and increased oxygen consumption.⁸ Moreover, it can increase the duration of mechanical ventilation, prolong ICU stay, and thereby increase mortality rates.^5,9 Unmanaged pain can also increase the risk of posttraumatic stress disorder and reduce quality of life.¹⁰

The most basic step to effective pain management is accurate pain assessment using appropriate instruments.¹ The gold standard for pain assessment is patients’ self-report. However, most patients in ICU cannot report their pain due to altered consciousness, mechanical ventilation, or sedation.^1,11 Despite great efforts to accurately assess pain in patients in the ICU, their pain is still underestimated or remains undiagnosed and unmanaged.¹² Significant reasons for pain underestimation and ineffective management are the subjectivity of pain and the differences among different individuals in pain experience, pain definition, and pain expression, which make accurate pain assessment and diagnosis a challenge for healthcare providers.¹

Using reliable and valid behavioral pain assessment instruments is one method for accurate pain assessment in patients with an inability to report their pain.⁷ These instruments were developed based on changes in patients’ behaviors in response to nociceptive stimuli.¹³ One of the behavioral pain assessment instruments is the critical-care pain observation tool (CPOT) that was developed by Gélinas et al.¹⁴ One study reported that CPOT is a good instrument with acceptable psychometric properties for pain assessment on agitated patients in ICU but noted that other behavioral criteria of pain should also be considered in pain assessment in these patients.¹⁵ Heidarzadeh et al. studied patients who were unconscious and confirmed that the psychometric properties of the Persian CPOT were acceptable.¹⁶ The behavioral pain scale (BPS) is another commonly used pain assessment instrument. This scale was developed by Payen et al.¹⁷ The psychometric properties of the Persian BPS were examined among patients who were on mechanical ventilation in an ICU. However, these patients were also able to report their pain using head movements.¹⁸

Although both CPOT and BPS have been used in different studies for pain assessment, there are no data comparing their psychometric properties in the same sample of patients. Moreover, only limited data are available about the psychometric properties of these instruments among patients with altered consciousness. Therefore, this study was conducted to compare the diagnostic value of CPOT and BPS for pain assessment in the same group of unconscious patients.

Materials and Methods

This cross-sectional study was conducted in four general ICUs of two hospitals in the north of Iran in 2019. The discriminant validity and reliability of two common pain assessment tools were examined simultaneously. The following assessment tools were used: The CPOT and BPS. Both are instruments for pain assessment in patients with critical conditions. CPOT has four main dimensions, namely, facial expression, body movement, compliance with ventilator, and muscle tension. Each dimension is scored between zero and 2, resulting in a possible total score between zero and 8. A score of zero means no pain whereas scores >2 show pain and the necessity for pain management. An earlier psychometric evaluation of CPOT showed that 91% of a panel of experts confirmed the validity of the instrument and the intraclass correlation coefficients of the instrument and its dimensions were between 0.73 and 0.99. In that study, the scores of CPOT significantly increased during position change and did not change during noninvasive blood pressure measurement, which confirmed the discriminant validity of the instrument.¹⁶

The BPS was developed by Payen et al. for pain assessment among patients in ICU. It has three main dimensions, namely, facial expression, upper limb movements, and compliance with mechanical ventilation. Each dimension is scored from 1 to 4, resulting in a possible total score of 3 (“no pain”) to 12 (“maximum pain”).¹⁷ A study in Iran reported that Cronbach's alpha of the scale was 0.75 and its interrater correlation coefficients were between 0.78 and 0.94 and noted that the scale has acceptable discriminant and criterion validity.¹⁸

Sample: The sample consisted of 45 unconscious patients for discriminant validity assessment: Convenience sampling was based on the following eligibility criteria: age of 18 years or greater, inability to report pain (determined by a Glasgow Coma Scale score of less than 14), hospitalization in ICU for at least 24 hours. Exclusion criteria were postoperative complication, unstable hemodynamic status, cognitive or psychiatric disorders, history of epilepsy, and intake of neuromuscular blocking agents; as well as a diagnosis of quadriplegia. The sample size was calculated assuming an effect size of 0.60, a power of 0.80, and a confidence level of 0.95. The G-power software (v. 3.03.10) for sample size calculation showed that with the abovementioned parameters, 45 participants were needed.

Measurements

Demographics and clinical information (including the Richmond agitation-sedation scale and Glasgow coma scale) were abstracted from the medical record. Initially, eligible patients were identified from the patient's medical record. An experienced ICU nurse who simultaneously used CPOT and BPS for pain assessment during a nociceptive procedure (position change) and a nonnociceptive procedure (noninvasive blood pressure measurement) did the discriminant validity assessment. Patient examination in both nociceptive and nonnociceptive procedures was performed to see to what extent each instrument was able to differentiate pain from nonpain.

Interrater agreement: For assessment of interrater agreement, two experienced nurses simultaneously and independently assessed 21 eligible patients using CPOT and BPS during a nociceptive and a nonnociceptive procedure. These nurses had undergone training in pain management and the use of pain assessment tools.

The SPSS (version 24) software was used for data analysis. The discriminant validity of each instrument was assessed by comparing its score in nociceptive and no-nociceptive procedures through the Mann–Whitney U test. The discriminant validity of each instrument was calculated separately. For reliability assessment, interrater agreement was assessed using the Medcal software by calculating Lin's concordance correlation coefficient (CCC) for each instrument and calculating the weighted kappa coefficient for dimensions.

The ethics committee of the Mazandaran University of Medical Sciences, approved this study (code: IR.MAZUMS.REC.1398.4824). The family members of eligible patients provided written informed consent for the study. In order not to interfere with patient care, pain assessments were performed during their routine care. This practice prevented the imposition of any undue movement or burden on the patient.

Results

In total, 45 unconscious patients hospitalized in ICU were assessed in this study. The mean age of the patients was 56.64 (±14.75) years and the majority were males (62.20%). They had been hospitalized in ICU due to surgery or trauma (to the thorax, abdomen, nervous system, or limbs) (57.70%) or medical problems such as cardiovascular, gastrointestinal, neurological, or respiratory disorders (42.30%). In terms of breathing, 60% of patients were under mechanical ventilation and 40% were not (intubated with no mechanical ventilation [11.10%] and nonintubated [28.90%]). Their scores for the Richmond agitation-sedation scale were between −4 and +3, and their level of consciousness based on the Glasgow coma scale was between 4 and 13.

The discriminant validity of CPOT and BPS was assessed by comparing their scores during nociceptive and nonnociceptive procedures. The Mann–Whitney U test revealed that the mean scores of CPOT and its facial expression, body movement, and muscle tension dimensions during the nociceptive procedure were significantly greater than that of the nonnociceptive procedure (p <0.05; Table 1), and there were no significant differences between nociceptive and nonnociceptive procedures in the mean scores for the compliance with ventilator dimension (p = 0.17). Moreover, the mean scores of BPS and its facial expression and upper limb movement dimensions during the nociceptive procedure were significantly greater than for the nonnociceptive procedure (p <0.05) whereas the difference between these two procedures due to the mean score of the compliance with mechanical ventilation dimension of BPS was not statistically significant (p = 0.08; Table 1). Both CPOT and BPS differentiated between nociceptive and nonnociceptive procedures although the effect size for all significant differences between the two procedures was rather small (p <0.05; Table 1).

Table 1.

Difference in mean scores of CPOT and BPS and their dimensions between nociceptive and nonnociceptive procedures (n = 45)

Instrument	Procedure\Dimensions	Nonnociceptive (n = 45) Mean (±SD)	Nociceptive (n = 45) Mean(±SD)	p value	Effect size
CPOT	Facial expression	0.35 ± 0.08	0.80 ± 0.11	0.00	0.18
	Body movement	0.15 ± 0.05	0.49 ± 0.08	0.00	0.04
	Compliance with ventilator	0.13 ± 0.06	0.24 ± 0.07	0.17	0.20
	Muscle tension	0.31 ± 0.08	0.69 ± 0.10	0.00	0.21
	Total	0.95 ± 0.19	2.22 ± 0.28	0.00	0.32
BPS	Facial expression	1.38 ± 0.61	2.00 ± 0.91	0.00	0.19
	Upper limb movements	1.11 ± 0.32	1.40 ± 0.65	0.02	0.12
	Compliance with mechanical ventilation	1.06 ± 0.33	1.29 ± 0.70	0.08	0.06
	Total	3.56 ± 0.14	4.70 ± 0.23	0.00	0.18

SD, standard deviation; CPOT, critical-care pain observation tool; BPS, behavioral pain scale

The interrater Lin's CCC values during nonnociceptive and nociceptive procedures were, respectively, 0.67 and 0.62 for CPOT and 0.74 and 0.88 for BPS (Table 2). Moreover, the weighted kappa values of CPOT and BPS and their dimensions were from 0.24 to 0.70 and from 0.43 to 0.82, respectively (Table 3).

Table 2.

Lin's interrater CCCs for CPOT and BPS during nociceptive and nonnociceptive procedures (n = 21)

Procedure\Instrument	Nonnociceptive	95% CI	Nociceptive	95% CI
CPOT	0.67	0.37,0.84	0.62	0.3,–0.81
BPS	0.74	0.49,0.88	0.88	0.58,0.91

CPOT, critical-care pain observation tool; BPS, behavioral pain scale

Table 3.

Interrater kappa agreement coefficients of the CPOT and BPS dimensions during nociceptive and nonnociceptive procedures (n = 21)

Instrument	Procedure\Dimension	Nonnociceptive	Nociceptive
CPOT	Facial expression	0.46	0.35
	Body movement	0.64	0.38
	Compliance with ventilator	0.60	0.24
	Muscle tension	0.70	0.50
BPS	Facial expression	0.63	0.61
	Upper limb movements	0.64	0.60
	Compliance with mechanical ventilation	0.83	0.43

CPOT, critical-care pain observation tool; BPS, behavioral pain scale

Discussion

This study evaluated the discriminant validity and the interrater reliability of CPOT and BPS in the same group of unconscious patients in ICU. Findings showed a statistically significant difference between the mean scores of CPOT during nociceptive and nonnociceptive procedures, confirming the acceptable discriminant validity of the instrument. This finding is consistent with several earlier studies that the Persian CPOT has acceptable discriminant validity.^15,16,19 Our study findings also showed that the effect size of CPOT in differentiating nociceptive and nonnociceptive procedures was rather small, denoting that the discriminant validity of the instrument was low. We did not find any study about the effect size of the Persian CPOT for the purpose of comparison. However, in line with our findings, a systematic review and meta-analysis showed that the discriminant validity of CPOT was fairly low.²⁰

We also found a significant difference between the mean scores of BPS during nociceptive and nonnociceptive procedures that confirms the acceptable discriminant validity of the scale. However, the effect size of the scale in differentiating these two procedures was rather small. In line with our findings, the study by Heidarzadeh et al. (2017) reported an acceptable validity of the BPS but provided no information about its effect size.¹⁸ The small effect size of BPS in the present study denotes that the discriminant validity of the scale is low. Despite the small effect size of both CPOT and BPS, the effect size of BPS was greater than that of CPOT denoting that BPS differentiates between nociceptive and nonnociceptive procedures better than CPOT. Gélinas et al. (2019) reported that among pain measurement instruments for patients in ICU, the CPOT and BPS are the best instruments. However, that study did not provide any information about the superiority of one instrument over the other.¹³ However, one study showed that among eight pain measurement instruments, CPOT had the best validity.²⁰ These discrepancies among different studies regarding the validity of CPOT and BPS are attributable to differences among studies regarding pain measurement time points, type of nociceptive and non-nociceptive procedures, sample size, and language used for the instrument.

Additionally, all dimensions of CPOT and BPS (except the “compliance with ventilator”) differentiated nociceptive and nonnociceptive procedures. The inability of these instruments in differentiating nociceptive and nonnociceptive procedures regarding patients’ compliance with the ventilator may be due to the fact that the patients were sedated and were receiving muscle relaxants and therefore tolerated mechanical ventilation better. In contrast to our findings, a prior study reported that the Persian CPOT differentiated nociceptive and nonnociceptive procedures regarding compliance with the ventilator in nonagitated patients.¹⁵

Our findings also showed that the coefficient of interrater agreement was less than 0.90 for both CPOT and BPS and the agreement coefficient of the CPOT was less than BPS. Moreover, the interrater agreement coefficients of both instruments during nociceptive procedure were less than for nonnociceptive procedure. Agreement coefficients less than 0.90 show weak interrater agreement. A systematic review reported that the interrater agreement coefficient of BPS in 18 studies was more than 0.60, and the authors considered this value acceptable.¹³ Another study showed that the interrater agreement of coefficients of CPOT were between 0.95 and 0.96 among intubated patients and between 0.96 and 0.98 among nonintubated patients.²¹ These contradictions across different studies are attributable to the difference between the characteristics of their samples. For example, Rafiei et al.'s (2016) study was conducted in conscious patients in surgical wards¹⁶ whereas our patients were unconscious patients in ICUs.

Conclusion

The salient findings of this study are that both CPOT and BPS have acceptable discriminant validity in differentiating nociceptive and nonnociceptive procedures among unconscious patients in ICU. Although both CPOT and BPS have relatively low reliability, the reliability of BPS was better than that of CPOT. The present study provides further evidence regarding the discriminant validity and interrater reliability of CPOT and BPS. Nurses need to also pay careful attention to nonverbal signs of pain while using CPOT and BPS for pain assessment in unconscious patients.

Orcid

Roghieh Nazari https://orcid.org/0000-0002-7041-9101

Erika Sivarjan Froelicher https://orcid.org/0000-0003-1852-8922

Hamid Sharif Nia https://orcid.org/0000-0002-5570-3710

Fatemeh Hajihosseini https://orcid.org/0000-0002-0084-9827

Noushin Mousazadeh https://orcid.org/0000-0002-0516-4325