Validation and evaluation of two observational pain assessment tools in a trauma and neurosurgical intensive care unit

Abstract

BACKGROUND:

Studies have demonstrated that patients in the intensive care unit experience high levels of pain. While many of these patients are nonverbal at some point during their stay, there are few valid tools available to assess pain in this group.

OBJECTIVES:

To evaluate the validity and clinical utility of two pain assessment tools, the revised Adult Non-Verbal Pain Scale (NVPS-R) and the Critical Care Pain Observation Tool (CPOT), in a trauma and neurosurgical patient population.

METHODS:

Patients were assessed using the NVPS-R and CPOT by trained intensive care unit nurses (n=23) and research assistants before, during and after two procedures: turning of the patient (nociceptive procedure) and noninvasive blood pressure cuff inflation (non-nociceptive procedure). Communicative patients were also asked to report their level of pain during each assessment.

RESULTS:

A total of 66 patients (34 communicative, 32 noncommunicative) were included in the study. CPOT and NVPS-R scores increased significantly when participants were exposed to turning, but not during noninvasive blood pressure measurement (repeated measures ANOVA: CPOT, F=5.81, P=0.019; NVPS-R, F=5.32, P=0.025) supporting discriminant validity. CPOT and NVPS-R scores were significantly higher during the turning procedure for patients who had indicated that they were in pain versus those who were not, indicating criterion validity. Inter-rater reliability was generally higher for the CPOT than NVPS-R. Nurses rated the feasibility of the two tools as comparable but provided higher ratings of acceptability for the CPOT.

CONCLUSIONS:

While the present study supports the use of the CPOT and the NVPS-R with critically ill trauma and neurosurgical patients, further research should explore the role of vital signs in pain.

Pain is frequently experienced by patients in the intensive care unit (ICU) for a variety of reasons including traumatic injuries, infection, immobility and procedures (1). In a 2008 survey of 868 verbal ICU patients in France, pain was identified as one of the greatest discomforts experienced during their ICU stay (2). Poor pain management has been associated with a number of adverse outcomes including thromboembolic events, pulmonary complications, increased length of ICU stay, chronic pain and post-traumatic stress disorder (3,4).

Improved pain management has been shown to reduce morbidity and improve long-term outcomes, as well as increase patient comfort. Regular pain assessments in nonverbal patients in the ICU have been associated with reduced duration of mechanical ventilation, reduced length of ICU stay and decreased rates of nosocomial infections (1,4,5).

Although the patient’s self-report is the most valid measure for pain, many ICU patients are unable to provide it due to various factors including sedation, mechanical ventilation and physiological effects of their illness or injury (6). Recognizing pain in patients unable to communicate is particularly important because they are at the highest risk for undertreated pain (7). In such situations, pain assessment methods need to be adapted to the communication capabilities of the patient. In nonverbal patients unable to self-report, the use of behavioural indicators are strongly recommended for the detection of pain, and physiological indicators (ie, vital signs) are suggested as cues for further assessment of pain (8,9).

Clearly, there is a need for pain assessment tools for nonverbal critically ill patients (7). However, there are few valid tools available for the assessment of pain in this group, and those that have been validated for use in adult neurosurgical and trauma patients are rare (10). The adult Non-Verbal Pain Scale (NVPS) was developed by Odhner et al (11) for patients in a critical care burn unit. It was developed in response to the lack of pain assessment tools for patients who were unable to rate and describe their pain. This includes patients who were mechanically ventilated and/or sedated (11). The original NVPS was comprised of five categories: facial movements; body movements; guarding; physiological I (blood pressure, heart rate, respiratory rate); and physiological II (dilated pupils, flushing, diaphoresis, pallor). A revision of the NVPS replaced the physiological II category, which exhibited poor reliability and validity, with a respiratory component (ie, respiratory rate, pulse oximetry and ventilator compliance) (12). Each category is rated on a scale of descriptors from zero to two, resulting in a total score ranging from zero to 10. The revised version (NVPS-R) was used in the current study.

The Critical Care Pain Observation Tool (CPOT) was developed by Gélinas et al (13) to measure behavioural indicators of pain in critically ill patients; specifically, patients unable to communicate their pain. The tool comprises four behavioural categories: facial expression; body movements; muscle tension; and compliance with the ventilator for mechanically ventilated patients, or vocalization for extubated patients. Each category is scored on a scale of zero to two, with a possible total score ranging from zero to eight. The CPOT has been validated in various adult groups in the ICU including postoperative, medical and trauma patients (13–16). The CPOT has also been positively evaluated for feasibility and clinical utility (17).

The clinical impact of implementation of both the original NVPS and the CPOT have been evaluated with trauma patients. A study published in 2010 (18) found that the use of the NVPS in a trauma-neurosurgical ICU was associated with improved patient ratings of their experience, improved documentation by nurses and increased nurses’ confidence in assessing pain in nonverbal patients. The implementation of the CPOT tool in critical care units has been associated with more frequent pain assessments, a lower number of complications, and enhanced administration of analgesics and sedatives (19,20). The validity has not been explicitly studied in a trauma and neurosurgical ICU for either tool. The objectives of the present study were: to examine discriminant validation of the CPOT and NVPS-R scores by using each tool during a nociceptive procedure (turning) and a non-nociceptive procedure (noninvasive blood pressure [NIBP] cuff inflation); to examine criterion validation of the CPOT and the NVPS-R scores using self-reports of pain in a subgroup of trauma and neurosurgical patients; to examine inter-rater reliability of the NVPS-R and the CPOT scores between two independent raters; and to describe the nurses’ perceptions of the feasibility and acceptability of the use of these pain assessment tools.

METHODS

The present prospective repeated-measures descriptive study of patients with traumatic injuries or neurosurgical indications was conducted in a 19-bed ICU of an urban teaching hospital (St Michael’s Hospital) in Toronto, Ontario. The study protocol was reviewed and approved by the institutional research ethics board. Written consent was obtained from the nurse participants and a waiver of written consent was granted for the patient participants because routine pain assessment is a standard of care within the hospital, and participating in the study posed minimal risk to the patients. A letter of information was provided to the study patients and/or their families.

Study samples

The study participants included both the nurses and the patients in the ICU. Twenty-three nurses consented to participate in the research, of whom 12 were trained to use the CPOT and 11 the NVPS-R. The randomly assigned nurses participated in a 1 h training session that included a PowerPoint (Microsoft Corporation, USA) presentation outlining the importance of pain assessment on patient outcome, a detailed description of the use and scoring of either the CPOT or the NVPS-R, and an opportunity to practice using their assigned tool by viewing videos (20) of four representative ICU patients created for educational purposes.

Seventy patients (36 communicative and 34 noncommunicative) were recruited for the study. Data from four participants (two communicative and two noncommunicative) were subsequently excluded due to incompleteness. Patients were included if they were admitted to the ICU and were ≥16 years of age. Patients who were quadriplegic, receiving neuromuscular blocking agents or were being investigated for brain death were excluded from the study.

Study variables and instruments

The following data were collected for the purposes of the study:

Patient data:

Sex, age, primary diagnosis, Glasgow Coma Scale (GCS) score on ICU admission, ventilation status and analgesic regimen (none, analgesia only, sedation only, sedation and analgesia).

NVPS-R score:

The NVPS-R has been tested with 64 trauma and surgical patients (21) and 24 nonverbal, critically ill patients in cardiac postanesthesia care unit (22). Moderate inter-rater reliability (intraclass correlation coefficients [ICC] 0.62 to 0.68), low to moderate internal consistency coefficients (Cronbach’s α = 0.36 to 0.75) and discriminant validation (significantly higher NVPS-R scores during procedural pain compared with rest) were demonstrated (6,22). Criterion validation has not been reported for the NVPS-R.

CPOT score:

Initially developed in French, the CPOT was translated into English using a back-to-back translation method. The use of the CPOT has been tested in 524 critically ill adults with various diagnoses (postoperative, medical and trauma) (13,14,23–28). Good discriminant validation was found with higher CPOT scores during nociceptive procedures (ie, turning, endotracheal suctioning) compared with rest or a non-nociceptive procedure (ie, NIBP cuff inflation, arm wash) (P<0.001). Criterion validation was supported with moderate correlations of 0.59 and 0.71 (P<0.05) between CPOT scores and self-reports of pain intensity of ICU patients during turning (13,14). In addition, the CPOT was reported to have high specificity (78%) and sensitivity (86%) with a cut-off score >2 during nociceptive exposure in postoperative ICU patients (29). In other studies, the CPOT has also demonstrated moderate to high inter-rater reliability, with kappa >0.60 or ICC >0.80 (13,14,26,28).

Study procedures

To examine discriminant validation, assessments using the NVPS-R and the CPOT were completed during two procedures: a nociceptive procedure known to be painful (turning of the patient) (30); and a non-nociceptive or nonpainful procedure (NIBP cuff inflation) (14). Assessments were completed at rest 5 min preprocedure, during the procedure (nociceptive and non-nociceptive) and 20 min postprocedure, for a total of six assessments per patient. The assessment periods have been designated as N1 (non-nociceptive preprocedure), N2 (during the non-nociceptive procedure) and N3 (non-nociceptive post-procedure), and P1 (nociceptive preprocedure), P2 (during the nociceptive procedure) and P3 (nociceptive postprocedure), respectively.

Each day, a study coordinator screened the ICU for eligible patients. When a patient was identified, the study coordinator asked the bedside nurse when they planned to next turn the patient or measure the patient’s blood pressure. The study coordinator then recruited two of the nurses (one trained on each tool) at the appropriate time to conduct the assessments. Each nurse independently completed the assessments. The study coordinator concurrently completed one of the tools (either NVPS-R or CPOT in an alternating manner per patient) to examine inter-rater reliability. Thus, inter-rater reliability data was only available for a subset of the participants: for the communicative group, paired assessments were conducted for 17 patients using the CPOT, and 18 patients using the NVPS-R; for the noncommunicative group, paired assessments were conducted for 17 patients using the CPOT and 22 patients using the NVPS-R. The assessors were blinded to one another’s scores. Given the small sample sizes, data were pooled across groups (communicative or noncommunicative) for the inter-rater agreement analyses.

Criterion validation was examined at the same time as discriminant validation. Criterion validation relates to the ability of the scale to correlate with the gold standard measure (in this case, the patient’s self-report of pain). Communicative patients were asked by the study coordinator to report their pain levels following each of the six assessments as described above. The patient was first asked whether they were in pain (yes/no) during the assessment period and, if able, to rate their pain using the 0 to 10 Faces Pain Thermometer (31) to determine whether there was congruence between the patient’s self-report and the NVPS-R and/or CPOT scores.

At the end of the study, each ICU nurse who participated as a rater was asked to complete a structured questionnaire including a four-point Likert scale (1 = totally disagree to 4 = totally agree) to obtain feedback about the feasibility and acceptability (eg, training, directives of use, easy to complete, quick to use) of using the CPOT or NVPS-R (17).

Data analyses

Descriptive analyses for all variables collected were completed: means and SDs for continuous variables (eg, age, years of nursing, years in the ICU) were calculated and were compared between communicative and noncommunicative patients, or nurses trained using either tool, using t tests. For categorical data (eg, sex, injury type, analgesia regimen), frequencies and percentages were calculated and compared between communicative and noncommunicative patients using the χ² test. A nonparametric independent samples median test was used to compare GCS scores (ordinal data) between the groups.

For discriminant validation, NVPS-R and CPOT scores obtained during the nociceptive and non-nociceptive procedures were compared with scores obtained at rest and postprocedure using repeated-measures ANOVA (RM-ANOVA). For criterion validation, the association between CPOT/NVPS-R scores and patients’ self-reports of pain (presence or absence, 0 to 10 Faces Pain Thermometer score) were evaluated. Linear mixed methods were used to compare CPOT and NVPS-R scores between patients in pain (yes) and pain-free patients (no) and across procedures (pre-, during and post-turning, and NIBP cuff inflation). The linear mixed-methods approach is preferable to RM-ANOVA when sample sizes across groups vary substantially. Spearman correlations between the CPOT and NVPS-R scores and patients’ self-report scores were calculated. Two-way mixed, type A (absolute agreement) ICCs were calculated to determine the inter-rater (nurse and study coordinator) reliability of the scale scores. ICCs should be >0.60 and, ideally, >0.75 (32).

RESULTS

Study participants

The mean age of the 66 patients for which data were analyzed was 50.5 years, and 54.5% were male (Table 1). Participants were categorized as communicative (n=34) or noncommunicative (n=32). The groups were comparable with regard to demographic, injury-related and treatment characteristics, with the exception that noncommunicative patients had a lower GCS score on ICU admission and were more likely to have been intubated than communicative patients.

TABLE 1.

Description of patient participants (n=66)

Variable	Communicative (n=34)	Noncommunicative (n=32)	Total (n=66)	Test
Sex
Male	15 (44.1)	21 (65.6)	36 (54.5)	χ2=3.08
Female	19 (55.9)	11 (34.4)	30 (45.5)
Age, years, mean ± SD	50.9±17.9	50.0±17.7	50.5±17.7	t=0.20
Glasgow Coma Scale score, median (IQR)	14.5 (13–15)	7.5 (5.25–13)	13 (6–15)	P<0.001*
Primary diagnosis
Traumatic				χ2=0.29
Traumatic brain injury	4 (11.8)	9 (28.1)	13 (20.0)
Other, nontraumatic brain injury	8 (23.5)	7 (21.8)	15 (22.7)
Neurosurgical
Subarachnoid hemorrhage	12 (35.3)	7 (21.9)	19 (28.8)
Spinal	3 (8.8)	–	3 (4.5)
Oncology	3 (8.8)	3 (9.4)	6 (9.1)
Other	4 (11.8)	5 (15.6)	9 (13.6)
Medical	–	1 (3.1)	1 (15.2)
Ventilation status
None	13 (38.2)	3 (9.4)	16 (24.2)	χ2=26.71*
Face mask/nasal prongs	15 (44.1)	3 (9.4)	18 (27.3)
Intubated	6 (17.6)	26 (81.3)	32 (48.5)
Analgesia regimen
No drips	13 (38.2)	13 (40.6)	26 (39.4)	χ2=5.40
Analgesia	17 (51.5)	9 (28.1)	26 (39.4)
Sedation	1 (3.0)	1 (3.1)	2 (3.0)
Sedation and analgesia	3 (9.1)	9 (28.1)	12 (18.2)

Data presented as n (%) unless otherwise indicated;

^*P<0.05. IQR Interquartile range

Twelve ICU nurses were trained to use the CPOT, and 11 to use the NVPS-R (Table 2). Nurses were primarily female, working full-time in the ICU and had a similar number of years of experience in both groups.

TABLE 2.

Description of nurse participants (n=23)

Variable	CPOT nurses (n=12)	NVPS-R nurses (n=11)
Sex
Male	2 (16.7)	0
Female	10 (83.3)	11 (100)
Age, years, mean ± SD	34.9±8.0*	32.2±6.5*
Years as nurse, mean ± SD	13.1±7.0*	12.9±8.7*
Years in ICU, mean ± SD	8.9±5.2*	11.7±7.5*
Employment status
Full-time	8 (66.7)*	8 (72.7)*
Part-time	2 (16.7)	2 (18.28)

Data presented as n (%) unless otherwise indicated.

^*n=10 (percentages have been adjusted for missing data). CPOT Critical Care Pain Observation Tool; ICU intensive care unit; NVPS-R Non-Verbal Pain Scale – Revised

Descriptive statistics for the CPOT and NVPS-R scores

Descriptive statistics for the CPOT and NVPS-R scores over time and according to group (communicative and uncommunicative) are provided in Figure 1. Scores on both the CPOT and NVPS-R increased during the nociceptive procedure (P2) compared with preprocedure (P1). The mean scores during turning (P1 to P3) were higher for noncommunicative compared with communicative patients, using both tools.

Figure 1).

Estimated marginal means of the Critical Care Pain Observation Tool (CPOT) and the Non-Verbal Pain Scale – Revised (NVPS-R) scores for pre-, during and postassessments of the nociceptive (black line; P1, P2 and P3, respectively) and non-nociceptive (grey line; N1, N2 and N3, respectively) procedures for communicative (solid lines) and noncommunicative (dashed lines) patients. Note that the y axes are scaled to one-half of the full scale score (CPOT = 8, NVPS-R = 10) to demonstrate relative changes in scores

Discriminant validation of the CPOT and NVPS-R scores

Significant interaction effects (time × procedure) were found for both the CPOT and the NVPS-R scores (Table 3). Descriptive statistics revealed that both the CPOT and the NVPS-R scores increased when participants were exposed to turning but not during NIBP cuff inflation (Figure 1). Moreover, mean scores with turning (pre-, during and post-turning) were higher for noncommunicative compared with communicative patients using both tools (RM-ANOVA: CPOT, F=5.81, P=0.019; NVPS-R, F=5.32, P=0.025). This difference was not observed during NIBP cuff inflation (RM-ANOVA: CPOT, F=0.02, P=0.879; NVPS-R, F=0.10, P=0.756).

TABLE 3.

Repeated-measures ANOVA comparing Critical Care Pain Observation Tool (CPOT) and Non-Verbal Pain Scale – Revised (NVPS-R) scores over time (pre-, during and postprocedure) and according to procedure (nociceptive versus non-nociceptive) for communicative (n=32) and noncommunicative (n=34) patients

	Repeated-measures ANOVA (F), effect
	CPOT			NVPS-R
	Time	Procedure	Time × procedure	Time	Procedure	Time × procedure
Communicative	13.56**	1.52	9.06**	17.73**	3.29	10.29**
Noncommunicative	30.01**	18.92**	19.93**	31.60**	12.86**	16.04**

^*P<0.05;

^**P<0.001

Criterion validation of the CPOT and NVPS-R scores

Two of the 34 communicative patients were unable to self-report their level of pain (neither yes/no nor provide a numerical pain score) due to intermittent drowsiness. Of the remaining patients, between 37.5% and 43.8% reported being in pain (yes) at rest and postprocedure (N1 to N3, P1, P3) versus 62.5% during the nociceptive procedure (P2; Table 4). CPOT and NVPS-R scores were significantly higher during turning (P2) for patients who had indicated that they were in pain versus those who were not (Figure 2), with significant interaction effects (time × procedure) for the assessments in which patients reported experiencing pain (Table 5).

TABLE 4.

Descriptive statistics for pre-, during and postprocedure of the nociceptive and non-nociceptive procedures for communicative patients (n=32) reporting being in pain versus those denying being in pain

Assessment	Patient-reported pain
	Yes		No
	n (%)	Score, mean ± SD	n (%)	Score, mean ± SD
N1	14 (43.8)	4.86±2.35	18 (56.3)	1.61±1.46
N2	12 (37.5)	5.17±2.67	20 (62.5)	1.60±1.64
N3	13 (40.6)	5.00±2.24	19 (59.4)	1.42±1.50
P1*	12 (40.0)	5.08±2.61	18 (60)	1.33±1.68
P2	20 (62.5)	6.37±1.90	12 (37.5)	1.33±1.50
P3	13 (40.6)	4.92±2.57	19 (59.4)	1.42±1.47

^*n=30. N1 Before the non-nociceptive procedure; N2 During the non-nociceptive procedure; N3 After the non-nociceptive procedure; P1 Before the nociceptive procedure; P2 During the nociceptive procedure; P3 After the nociceptive procedure

Figure 2).

Estimated marginal means of Critical Care Pain Observation Tool (CPOT) and the Non-Verbal Pain Scale – Revised (NVPS-R) scores for pre-, during and postassessments of the nociceptive (black) and non-nociceptive (grey) procedures for patients reporting being in pain (solid lines) versus those denying being in pain (dashed lines)

TABLE 5.

Linear mixed methods comparing Critical Care Pain Observation Tool (CPOT) and Non-Verbal Pain Scale – Revised (NVPS-R) scores over time (pre-, during and postprocedure) and according to procedure (nociceptive versus non-nociceptive) for communicative patients (n=30) reporting being in pain (yes) versus those denying pain (no)

	Effect, linear mixed methods (F)
	CPOT			NVPS-R
	Time	Procedure	Time × procedure	Time	Procedure	Time × procedure
Patient pain = Yes	8.06*	2.43	5.80*	10.9**	3.29	5.70*
Patient pain = No	0.31	0.35	0.52	0.90	0.35	0.28

^*P<0.01;

^**P<0.001

A nonparametric correlation analysis was conducted to evaluate the correlation between patient’s self-reported pain score and the nurses’ score for communicative patients across all available assessments (CPOT, n=190; NVPS-R, n=178). Moderate correlations were found for both the NVPS-R (Spearman’s σ=0.313; P<0.001 [two-tailed]) and the CPOT (Spearman’s σ=0.435; P<0.001 [two-tailed]).

Inter-rater reliability of the CPOT and NVPS-R scores

ICCs were calculated to compare the paired assessments completed by the study coordinator and nurse assessor. Generally acceptable ICCs were observed for the CPOT (0.60 to 0.97) and generally lower ICCs were observed for the NVPS-R (0.34 to 0.92; Table 6).

TABLE 6.

Intraclass correlation coefficients (ICCs) between two raters of the Critical Care Pain Observation Tool (CPOT) and Non-Verbal Pain Scale – Revised (NVPS-R) scores for pre, during and post-assessments of the nociceptive and non-nociceptive procedures (communicative and noncommunicative patients combined)

	CPOT (n=34)						NVPS-R (n=40)
	Preassessment		During assessment		Postassesment		Preassessment		During assessment		Postassesment
	ICC	95% CI	ICC	95% CI	ICC	95% CI	ICC	95% CI	ICC	95% CI	ICC	95% CI
Non-nociceptive	0.79	0.52–0.92	0.84	0.61–0.94	0.97	0.92–0.99	0.67	0.43–0.80	0.55	0.29–0.73	0.34	0.17–0.56
Nociceptive	0.84	0.61–0.94	0.82	0.58–0.73	0.60*	0.18–0.84	0.42	0.13–0.65	0.92	0.86–0.96	0.49	0.21–0.69

^*n=33

Ratings of feasibility and acceptability of the use of the CPOT and NVPS-R by ICU nurses

Three nurses (NVPS-R group, n=2; CPOT group, n=1) did not complete the survey, resulting in final sample sizes of n=10 for each group. Overall, the feasibility of the CPOT and the NVPS-R was positively evaluated by the nurse participants (Table 7). The two assessment tools were considered quick to use, simple to understand and easy to complete. Regarding acceptability, more nurses appeared to be in favour of the CPOT compared with the NVPS-R, with nurses in the CPOT group being slightly more likely to indicate that they would recommend using it routinely, that it is helpful for practice and that it influenced their practice with respect to pain assessment (Table 7).

TABLE 7.

Nurse participants’ responses to the feasibility and acceptability survey

Question	Tool	Frequency (n)				Percent who answered 3 or 4
		1 – Not at all	2 – A little	3 – Sufficiently	4 – Very
Feasibility
1. Was the length of time sufficient to train to use the tool accurately?	CPOT	0	0	4	6	100
	NPVS	0	0	3	7	100
2. Were the directives about the use of the tool clear?	CPOT	0	0	5	5	100
	NPVS	0	1	3	6	90
3. Is the tool quick to use?	CPOT	0	1	2	7	90
	NPVS	0	1	4	5	90
4. Is the tool simple to understand?	CPOT	0	0	3	7	100
	NPVS	0	0	3	6	100
5. Is the tool easy to complete?	CPOT	0	0	4	6	100
	NPVS	0	0	2	8	100
Acceptability
6. I would recommend using the tool routinely.	CPOT	0	4	5	1	60
	NPVS	2	5	1	2	30
7. The tool is helpful for practice.	CPOT	0	4	5	1	60
	NPVS	2	6	2	0	20
8. Using the tool has influenced my practice in assessing the patient’s pain.	CPOT	1	3	4	2	60
	NPVS	4	4	1	1	20

CPOT Critical Care Pain Observation Tool; NVPS Non-Verbal Pain Scale – Revised

The nursing satisfaction survey also offered respondents an opportunity to comment in more detail on the tools. The comments reflected and augmented the numerical scores. Respondents commented that the CPOT was a useful tool for easily identifying and assessing cues for pain in nonverbal, sedated, unconscious and intubated patients, and that it acted as a reminder that pain may occur with different procedures. One user expressed concern that agitation or restlessness may make assessment using the CPOT more confusing because it was difficult to interpret the patient being tense or biting their endotrachial tube. A second respondent commented on their preference for the inclusion of vital signs in the assessment tool. Respondents also commented that the NVPS-R was a useful tool for assessing cues for pain in sedated nonverbal and/or unconscious patients. There were three comments expressing concern that the NVPS-R did not identify pain in their patients at all, and a number of additional comments (n=3) that respondents did not believe the tool had any influence on their practice.

DISCUSSION

Our findings support the discriminant and criterion validation of the CPOT and NVPS-R for use in assessing pain in communicative and noncommunicative patients in a trauma and neurosurgical intensive care unit. Inter-rater and concurrent validity were generally weaker for the NVPS-R than CPOT. Nurses rated both tools as feasible and generally acceptable for use with their patient population.

The recently published ‘Clinical Practice Guidelines for the Management of Pain, Agitation, and Delirium in Adult Patients in the Intensive Care Unit’ (33) recognized that procedural pain is common in adult ICU patients and recommended that pain be routinely monitored in these patients. Moreover, they indicated that the “Behavioral Pain Scale (BPS) and the CPOT are the most valid and reliable behavioral pain scales for monitoring pain in medical, postoperative, or trauma (except for brain injury) adult ICU patients who are unable to self-report and in whom motor function is intact and behaviors are observable.” The NVPS-R was rated as having very low psychometric evidence and more studies were deemed to be necessary to examine its psychometric properties (ie, reliability and validity).

In our study, we examined the validity of the use of the CPOT and NVPS-R side-by-side in the same study, using predetermined and previously tested nociceptive and non-nociceptive stimuli in a naturalistic setting. We instructed the ICU nurses themselves to conduct the assessments, the individuals for whom the tools were intended. Moreover, we recruited a comparatively large sample size (n=66) of a relatively unstudied population (trauma and neurosurgical patients). There are a few reports of the use of the CPOT with populations that included a subset of trauma and/or neurosurgical patients: Gélinas and Johnston (14) evaluated 55 patients in a general ICU (49% with trauma or neurosurgical diagnoses); Gélinas and Arbour (24) included 257 patients from four separate health centres with 20.2% being trauma patients (mainly with head injury); Lee et al (25) included 31 neurosurgical patients; and recently, Kwak and Oh (34) reported on the validation of the Korean translation of the CPOT with 202 patients, 60.9% with neurological injuries. The NVPS-R was originally developed in a burn trauma unit, and was subsequently evaluated in 64 trauma and surgical patients (the number of neurosurgical patients was not reported) (21), and 200 patients from 13 critical care units, 19% of which had trauma diagnoses (it was unclear how many with brain injury) (6).

In the present study, we intentionally examined the performance of the tools in both communicative versus noncommunicative patients because, in a clinical setting, it would be ideal to have one pain assessment tool that could be used to assess different patients. Moreover, by including patients who were communicative, we were able to assess criterion validation by comparing the patients’ self-reports with the pain scores assessed using the selected tools.

Our study was limited by the inclusion of only a single site. Moreover, even though our study was conducted in a single unit, the patient population included both brain-injured (eg, traumatic brain injury, subarachnoid hemorrhage, neuro-oncological) and nonbrain-injured patients (traumatic injuries, spinal surgery). Preliminary research suggests that patients with brain injuries may display different pain-related behaviours compared with their nonbrain-injured counterparts (24). Future studies may need to examine the validity of the tools in these patients separately.

Discriminant validation for both of the tools was supported by our findings, with both CPOT and NVPS-R scores increasing during the nociceptive procedure (turning) compared with pre- and post-procedure, and compared with the non-nociceptive procedure (NIBP cuff inflation). Discriminant validiation has been previously demonstrated for the CPOT (13,14,25,28,34–36) in various populations and in fewer reports for the NVPS-R (6,21). The mean CPOT scores (2.66±1.77 for noncommunicative patients; 1.88±1.93 for communicative patients) during the nociceptive procedure were comparable with what has been reported in previous studies: 2.42±1.31 to 3.26±1.84 during endotracheal suctioning in a sample of patients with brain injury (25); 3.37±2.41 during suctioning in a mixed sample of patients (60.9% with neurological injury) (34); and 1.93±1.41 during turning in 96 mechanically ventilated medical and surgical patients (28). Only two studies have examined the discriminant validation of the NVPS-R in critically ill patients: mean NVPS-R scores of 2.8±1.9 during turning and 0.87±1.2 before turning were reported in a mixed sample of 200 ICU patients (6); mean NVPS-R scores of approximately 2.4 were reported during nociceptive procedures versus 0.4 pre- and 0.8 postprocedure in 64 trauma and surgical patients (21). These scores are comparable with those of the non-communicative patients observed in our study: 2.73±2.24 during turning versus 0.57±0.97 pre- and 0.30±0.91 post-turning. There are no comparative data available in the literature for communicative patients assessed with the NVPS-R, either during nociceptive or non-nociceptive procedures.

Of interest, the mean score during the nociceptive procedure was significantly higher for noncommunicative patients (CPOT = 2.66±1.77; NVPS-R = 2.73±2.24) than communicative patients (CPOT = 1.88±1.93; NVPS-R = 1.69±1.82), yet was comparable during the non-nociceptive procedure for both tools. This finding is contrary to previous reports of postcardiac surgery patients in which CPOT scores during nociceptive procedures were higher in conscious (3.38±1.38 and 3.47±1.51) compared with unconscious patients (2.7±1.36 and 2.23±1.48) (13;29). Conversely, Vazquez et al (28) found no significant difference in CPOT scores between conscious and unconscious patients (n=96, primarily surgical patients). It is interesting to note that Keane (37), reporting on cardiac surgery patients, also found higher CPOT scores in unconscious compared with conscious patients. The difference in findings may be attributable to the different patient populations examined in these studies. For example, surgical patients have been shown to demonstrate higher CPOT scores than patients with head trauma (24). It is also likely that the communicative patients in our study had less pain overall than the noncommuni-cative patients, potentially as a result of more analgesia administered (51.5% of communicative patients versus 28.1% of noncommunicative patients in our study received analgesia).

Although the CPOT scores at preprocedure were slightly higher than the NVPS-R scores, the percentage increase in score during the nociceptive procedure was comparable across the tools. For example, for noncommunicative patients, the mean increase in CPOT score from pre- to during-procedure was 1.69 points (21.1%) versus 2.16 points (21.6%) for the NVPS-R. Thus, although the tools have different scales, the degree of change in pain score was comparable.

Patients who answered ‘yes’ when asked whether they were experiencing pain at each assessment had higher numerical pain scores than those reporting no pain. Also, although not statistically significant, patient’s self-reported pain scores were, on average, higher during the nociceptive procedure (6.38±1.90) versus the non-nociceptive/rest assessments (ranging from 4.86±2.35 to 5.17±2.67) for patients answering ‘yes’ to pain compared with those answering ‘no’ for which the scores remained relatively consistent across assessments (1.33±1.68 to 1.61±1.46). In addition, our findings indicated that scores on the CPOT and NVPS-R increased significantly during the nociceptive procedure for patients who answered ‘yes’ to experiencing pain, but not for those who answered ‘no’, or during the non-nociceptive procedure, supporting the criterion validation of both tools’ scores. Correlations with the patients’ self-reported pain scores were higher for the CPOT (Spearman σ=0.435) than for the NVPS-R (Spearman σ=0.313). These findings add to the results of previous studies that have demonstrated criterion validation of the CPOT scores with cardiac surgery patients (13), mixed ICU patients (29) and, to a lesser extent, in burn patients (36). The present study is the first to demonstrate the criterion validation of the NVPS-R scores.

In our study, we observed generally acceptable ICCs for the CPOT (0.79 to 0.97, except for 0.6 postnociceptive procedure) and generally lower ICCs for the NVPS-R (0.34 to 0.67, except for 0.92 during the nociceptive procedure). Previous studies of the CPOT conducted with postsurgery cardiac patients (13,22,37) have documented moderate to high inter-rater reliability for the CPOT. In populations similar to that included in the present study, Gélinas and Johnston (14) reported ICCs of 0.88 during a nociceptive procedure and 0.8 to 0.93 during rest periods in their sample of 55 patients with medical diagnoses and traumatic injuries, and Kwak and Oh (34) reported Kappa coefficients of 0.85 (at rest), 0.81 (during suctioning) and 0.88 (20 min postprocedure) in their study of 200 primarily neurologically injured patients. In a sample of 200 critically ill patients with varying diagnoses, Juarez et al (6) documented ICCs of 0.62 (at rest) and 0.68 (with turning) using the NVPS-R. As has been previously shown (6) our data suggest that the NVPS-R has generally lower inter-rater reliability than the CPOT. There are several possible reasons why the ICCs for both the CPOT and NVPS-R were lower and more variable in our study than has been shown in previous studies. In our study, 12 (CPOT) and 11 (NVPS-R) staff nurses were trained to conduct the assessments, unlike previous studies, in which dedicated research nurses were employed (13,22,37). Better ICCs may be related to the educational training or the number of times (eg, experience) the nurses were able to perform the assessments (6). Also, in previous studies that found high inter-rater reliability of the tools, the assessors trained until they reached a certain predetermined level of agreement (13,21,22,37). While these factors (same assessors, extensive training) are ideal for research purposes, they are not necessarily practical in the clinical scenario. While we may have obtained higher scores on inter-rater reliability with more extensive training, we aimed to keep the educational session comparable to that which the nurses would likely receive if the tool was to be implemented in clinical practice. Nurses in the study reported that they were satisfied with the training provided. Wibbenmeyer et al (36) indicated that their participating staff “were briefly educated on the use of scales” which may have contributed to the lower ICCs observed for both the CPOT and NVPS (original) in their study. It is also important to note that the study coordinator in our study was not a clinician, which may have influenced the ratings because it has been suggested that the assessment of postoperative pain may be influenced by profession (38).

An additional feature of our study was the surveying of the staff nurse participants at the end of the study to obtain their feedback regarding the use of the two tools. The nurses responses in our study were comparable to those that have been previous reported for the NVPS (original) (18) and the CPOT (39). In our study, 60% indicated that they would “use the CPOT routinely in their practice” and that “it is helpful for nursing practice”; this contrasts with the 72.7% and 54.5% reported in the Gélinas study (39). Interestingly, fewer of the nurses indicated that they would use the NVPS-R routinely in their practice (30%), that it is helpful for nursing practice (20%) or that it influenced their practice in assessing patients’ pain (20%). These lower ratings are comparable to what we have shown previously with the original NVPS (18). Following implementation of the NVPS in the ICU, only 34% of nurses reported that having a pain rating scale would ease the assessment of their patient’s pain, 28% indicated it would make them more confident in requesting more or less analgesics for their patient, and 25% believed it would improve their practice related to pain management (18). Overall, the nurses’ ratings and feedback comments suggested that while feasibility is comparable for the two tools, the acceptability or relevance of the CPOT is stronger.

CONCLUSIONS

As recommended by the recently released Society of Critical Care Medicine guidelines (9), our study data suggests that the CPOT has stronger validity than the NVPS-R for the assessment of pain among critically ill, nonverbal patients, particularly those with neurological and traumatic injuries. Future studies may also evaluate the validity of the use of these tools using analgesia trials, and larger studies are needed to examine the effect of implementation of these tools on patient care and patient outcomes. When comparing the two tools side-by-side, the nurses in our study appeared to have a slight preference for the use of the CPOT. It is interesting that comments regarding the preference for the use of vital signs persist among the ICU nurses, although the empirical evidence available does not support the use of vital signs as a specific indicator of pain, and should only be used as a cue to begin further assessment of pain (8,9). Future research is still needed to explore the role of vital signs in pain assessment.