A novel pain assessment tool incorporating automated facial analysis: interrater reliability in advanced dementia

Abstract

Objectives

Regardless of its severity, dementia does not negate the experience of pain. Rather, dementia hinders self-reporting mechanisms in affected individuals because they lose the ability to do so. The primary aim of this study was to examine the interrater reliability of the electronic Pain Assessment Tool (ePAT) among raters when assessing pain in residents with moderate-to-severe dementia. Secondly, it sought to examine the relationship between total instrument scores and facial scores, as determined by automated facial expression analysis.

Study design

A 2-week observational study.

Setting

An accredited, high-care, and dementia-specific residential aged care facility in Perth, Western Australia.

Participants

Subjects were 10 residents (age range: 63.1–84.4 years old) predominantly with severe dementia (Dementia Severity Rating Scale score: 46.3±8.4) rated for pain by 11 aged care staff. Raters (female: 82%; mean age: 44.1±12.6 years) consisted of one clinical nurse, four registered nurses, five enrolled nurses, and one care worker.

Measurements

ePAT measured pain using automated detection of facial action codes and recordings of pain behaviors.

Results

A total of 76 assessments (rest =38 [n=19 pairs], movement =38 [n=19 pairs]) were conducted. At rest, raters’ agreement was excellent on overall total scores (coefficient of concordance =0.92 [95% CI: 0.85–0.96]) and broad category scores (κ=1.0). Agreement was moderate (κ=0.59) on categorical scores upon movement, while it was exact in 68.4% of the cases. Agreement in actual pain category scores gave κ_w=0.72 (95% CI: 0.58–0.86) at rest and κ_w=0.69 (95% CI: 0.50–0.87) with movement. All raters scored residents with higher total scores post-mobilization compared to rest. More facial action unit codes were also detected during pain (mean: 2.5 vs 1.9; p<0.0012) and following mobilization (mean: 2.5 vs 1.7; p<0.0001) compared to no pain and rest, respectively.

Conclusions

ePAT, which combines automated facial expression analysis and clinical behavioral indicators in a single observational pain assessment tool, demonstrates good reliability properties, which supports its appropriateness for use in residents with advanced dementia.

Introduction

In residential aged care facilities (RACFs), dementia is common in >50% of residents.^1–3 Up to 97% of individuals with advanced dementia exhibit behavioral (eg, aggression) and psychological (eg, anxiety) symptoms that lead to poor quality of life in this setting.⁴ More than 90% of aged care staff had been exposed to physical or emotional aggression from residents.⁵ This problem is in part due to lack of self-report and inadequate identification of pain resulting in its subsequent poor management in this vulnerable population.⁶ There is strong evidence to suggest that behavioral and psychological symptoms of dementia (BPSD) are often associated with uncontrolled underlying pain from clinical and observational studies.^7–9 Moreover, in a large Swedish cohort (n=120,067) study of older adults (≥75 years) with advanced dementia, 38.6% received at least one medication of questionable benefit including psychotropic drugs in their final year of life.¹⁰ Delayed identification of pain may also influence drug-prescribing patterns. In a recent Italian study of nursing home patients, psychotropic drugs were among the top 10 most commonly prescribed drugs (quetiapine ranked fifth).¹¹ Pain has also been reported to be significantly associated with BPSD, higher number of antipsychotic prescriptions, reduced quality of life, and premature mortality.¹²

Regardless of its severity, dementia does not negate the experience of pain although there is evidence that pain processing may be altered.^13–15 It is inferred that pain experience might be increased in individuals with Alzheimer’s dementia as evident in pain responses recorded from brain activity and facial expressions.^13,16–19 Of particular importance, patients with dementia are more facially expressive than healthy subjects when they display pain.²⁰ In the absence or lack of self-rating report, facial expressions become an essential component of communicating the existence of pain, particularly for those with dementia.²⁰ Facial expressions provide instant and brief signals to alert the onlooker. Facial descriptors are also valid indicators in observational pain scales for nonverbal patients with dementia.²¹ However, reliability of observers in identifying these descriptors is often low because included items are generic, vague, and not able to be consistently recorded.²² Further, these descriptors such as “grimacing” in the Pain Assessment Checklist for Seniors with Limited Ability to Communicate (PACSLAC) and Abbey Pain Scale (APS) are not specific to pain as they could overlap with other emotions such as sadness.^23–25 Thus, it has been suggested that objective and comprehensive criteria, such as the Facial Action Coding System (FACS), should be considered among these tools to improve their reliability.^22,26–28 FACS is an anatomical catalog of facial expressions that annotates each individual facial action unit (AU) with a unique numerical label and specific description.²⁹ Pain-related AUs include eyelid tightening (AU7) and lips parting (AU25). Proficiency in the manual decoding of these AUs requires at least 100 hours of training, while each minute of video requires generally 1 hour of expert’s observations.²⁹ It is, hence, preferred to use automated facial decoding because it reduces the reliance on human rating, which may introduce subjectivity and is likely to be associated with judgment bias.

To address the suboptimal management of pain in people with dementia, novel means of detecting pain in clinical practice are urgently needed. This is because none of the currently available observational pain assessment tools used for people with dementia possess sufficient evidence of validation and reliability to be considered the gold standard.³⁰ Attempts to integrate computer vision (eg, artificial intelligence or AI) and facial recognition technologies into clinical tools have been made possible with the introduction of smart devices that provide agile platforms for software applications or apps. These intricacies have inspired us to develop the electronic Pain Assessment Tool (ePAT).^30,31,61 In this study, we aimed to examine interrater reliability of ePAT as a means of evaluating pain in aged care residents with moderate-to-severe dementia. Further, we examined the relationship between facial scores (which are determined using automated facial analysis) and total pain scores.

Materials and methods

Ethics

This study is part of a larger clinical trial (Australian New Zealand Clinical Trials Registry Number: ACTRN12616001003460), which was approved by the ethics review board of the participating aged care facility and the Human Research Ethics Committee (HR 10/2014) of Curtin University, Bentley, Western Australia. The study was conducted according to the Declaration of Helsinki, Alzheimer’s Australia Guidelines and the Australian National Statement for Ethical Conduct in Human Research.

All participating staff provided written informed consent. For residents, the capacity to consent was determined by the level of cognitive impairment. All residents had moderate-to-severe dementia or cognitive impairment, which makes them incapable of providing consent. Therefore, proxy informed written consent was obtained for each participant (resident) through their authorized and legal representatives. Consent was also given to the publication of images displayed in this manuscript.

Pain measure

The ePAT

The ePAT was designed by Curtin University researchers after reviewing the literature of pain, dementia, geriatric care, and pain facial expressions.^30,31,61 ePAT is a smart device application (App) that uses a combination of a selected set of facial AU codes and common pain behaviors reported in the literature (eg, items included in the American Geriatric Society [AGS] Indicators of Persistent Pain [2002]) to assess pain at the point of care.^29,32 A predefined set of facial AUs were included in the tool because they were associated with pain.^{17,20,33–35}

The tool uses digitization, real-time automated facial recognition and decoding using a deep learning (AI) approach, as a means of identifying and evaluating pain.⁶¹ Digitization and smart device technology serve as a platform to facilitate documentation, while automated FACS decoding is integrated in the tool with the view to improve objectivity through reducing human observation errors.⁶¹ Automated facial analysis identifies subtle facial muscle movements called AUs, which represent the smallest building blocks responsible for exerting microexpressions, each of which lasts for 100–500 milliseconds.³⁶ The automated facial assessment consists of three steps (Figure 1):

1. Face detection and tracking (Figure 2)

2. Localization and extraction of facial features (Figure 3)

3. Detection of facial AUs (Figure 4).

Figure 1.

Steps of automated facial analysis in the Face domain of the ePAT to identify pain-related facial action units in patients with dementia.

Abbreviation: ePAT, electronic Pain Assessment Tool.

Figure 2.

Face detection using the ePAT (step 1).

Abbreviation: ePAT, electronic Pain Assessment Tool.

Figure 3.

Automated facial recognition and extraction of facial action units (step 2) using active appearance model and facial landmarking.

Figure 4.

Detection of facial actions using AU descriptors of FACS (step 3).

Abbreviation: AU, action unit; FACS, Facial Action Coding System.

We have tested the ePAT application on Samsung Note 3 (SM-N9005) operating on Android 4.4 KitKat using the lowest available frame per second mode (ie, 30 fps). However, a frame rate of ≤5 fps is adequate for the application to perform its facial analysis. The duration of automated facial analysis to process the detection of pain-related facial AUs is ~10 seconds.

The output of the processing is a list of numerical values that represent the confidence level for each AU that we detect. The application will then combine an “x” number of reports obtained for a processed grabbed image to create a consolidated report for the 10 second recordings.

Once detected, facial AUs related to pain are then used in conjunction with other observation-based clinical data (eg, vocalization parameters) recorded by the user to obtain a pain intensity score.

The ePAT is composed of six domains (Face, Voice, Movement, Behavior, Activity, and Body), which contains a total of 42 items.^31,61 Table 1 describes the ePAT domains and the corresponding items along with their operational definitions and primary conceptual basis. Each domain contains a number of items. For example, the Face domain (Domain 1) consists of nine descriptors, which correspond to AUs 4, 6, 7, 9, 10, 12, 20, 25, and 43, automatically recognized by the App. Other domains (Domains 2–6) are based around descriptors drawn from the literature (eg, the AGS Indicators, other observational pain scales, recommendations by Herr et al, and Pasero and McCaffrey’s Hierarchy of Pain Assessment Techniques).^32,37,38 The latter is also supported by American Society for Pain Management Nursing recommendations about patients unable to self-report.^37,39

Table 1.

The six domains, corresponding items, and conceptual basis of the ePAT

Domain	Item number	Item description	Operational definition of item	Primary conceptual basis of selected items
Domain 1: The Face	1	Brow lowering	Facial action unit and the corresponding descriptor  AU4: brow lowerer Muscular basis  Depressor glabellae, depressor supercilii, and corrugator Anatomical changes in upper Face (lower central forehead)  • Lowering down of both eyebrows  • Movement of eyebrows toward each other  • Appearance of vertical or oblique wrinkles between eyebrows in the lower central part of the forehead	FACS29
	2	Cheek raising	Facial action unit and the corresponding descriptor  AU6: cheek raiser Muscular basis  Outer portion of orbicularis oculi (pars orbitalis) Anatomical changes in central face (infraorbital region)  • Pulling of skin toward the eye  • Pulling the cheeks upward by lifting of the infraorbital triangle  • Narrowing the eye aperture and wrinkling the skin below the eye  • Appearance of Crow’s feet lines or wrinkles  • Deepening of the lower eyelid furrow
	3	Tightening of eyelids	Facial action unit and the corresponding descriptor  AU7: lid tightener Muscular basis  Inner portion of orbicularis oculi (pars palpebralis) Anatomical changes in upper face (orbital region)  • Tightening of the eyelids  • Narrowing of the eye aperture  • Raising of lower lid
	4	Wrinkling of nose	Facial action unit and the corresponding descriptor  AU9: nose wrinkler Muscular basis  Levator labii superioris alaeque nasi Anatomical changes in central face  • Pulling of skin upward along the side of the nose toward the root of the nose  • Appearance of wrinkles along the side and root of nose  • Wrinkling of infraorbital furrow  • Lowering the medial portion of the eyebrows  • Pulling the center of the upper lip upward
	5	Raising of upper lip	Facial action unit and the corresponding descriptor  AU10: upper lip raiser Muscular basis  Levator labii superioris Anatomical changes in central-lower face (infraorbital, nasolabial, infranasal and upper lip regions)  • Prominent deepening or wrinkling of infraorbital furrow  • Deepening of nasolabial furrow  • Pouching at upper lip and nasal passages  • Widening and raising of the nostril wings
	6	Pulling at corner lip	Facial action unit and the corresponding descriptor  AU12: lip corner puller Muscular basis  Zygomatic major Anatomical changes in lower face (mouth/lips region)  Drooping or oblique movement of lateral corners of the lips
	7	Horizontal mouth stretch	Facial action unit and the corresponding descriptor  AU20: lip stretcher Muscular basis  Risorius Anatomical changes in lower face (mouth/lips region)  Bilateral stretch of lips
	8	Parting lips	Facial action unit and the corresponding descriptor  AU25: lips part Muscular basis  Depressor Labii, or relaxation of Mentalis or Orbicularis oris Anatomical changes in lower face (mouth/lips region)  Relaxed opening of the mouth, that is jaw drop
	9	Closing eyes	Facial action unit and the corresponding descriptor  AU43: eye closure Muscular basis  Relaxation of levator palpebrae superioris Anatomical changes in upper face (orbital region)  Shutting both eyes for at least half a second
Domain 2: The Voice	10	Noisy pain sounds, for example, ouch, ah, mm	Sounds or utters related to pain, for example, ouch, ah, mm	AGS (verbalizations and vocalizations)32
	11	Requesting help repeatedly	Include one or more of the following:  • Expressing numerous verbal requests of help within short periods of time, for example, “help me, help me”  • Constant talking  • Repetitive use of words or phrases (eg, echophrasia)  • Verbal nonsense  • Vocalizations with/without discernible meaning Exclude verbal requests for ADL purposes
	12	Groaning	Using a deep, creaking, or incoherent sound
	13	Moaning	Producing a long, low-pitched, and inarticulate sound
	14	Crying	Weeping, sobbing, or whimpering
	15	Screaming	Using excessively loud voice when communicating (as in shouting or yelling)
	16	Loud talk	Making louder than normal pitched voice
	17	Howling	Producing a long wailing cry sound
	18	Sighing	Breathing in followed by long audible sound upon breathing out
Domain 3: The Movement	19	Altered or random leg/arm movement	Changed or random movement of any of the limbs	AGS (body movements)
	20	Restlessness	Unable to relax, that is fidgeting
	21	Freezing	Sudden stiffening, avoiding movement, holding breath
	22	Guarding/touching body parts	An abnormally stiff, rigid, or interrupted movement while changing position Protecting affected area by holding body part
	23	Moving away	Avoiding being touched or staying away from the interaction
	24	Abnormal (altered) sitting/standing/walking	Distorted, asymmetrical or changed sitting, standing (eg, posture), and/or imbalanced gait (eg, limping)
	25	Pacing/wandering	Roaming restlessly and aimlessly back and forth
Domain 4: The Behavior	26	Introvert (unsocial) or altered behavior	Being unsocial or socially isolated, that is reluctant to be involved in social activities	AGS (changes in interpersonal interactions, mental status changes)
	27	Verbally offensive	Verbally abusive or aggressive, cursing, swearing, or using foul/insulting language
	28	Aggressive	Involved in combative or violent behavior
	29	Fear or extreme dislike of touch, people	Phobias of being touched or interaction with people including family members, other residents, and/or aged care staff
	30	Inappropriate behavior	Aberrant or socially unacceptable behavior, for example, fiddling
	31	Confused	Unclear in thinking or understanding, for example, unable to follow instructions or repetitive questioning
	32	Distressed	Anxious, worried, and agitated
Domain 5: The Activity	33	Resisting care	Unable to cooperate or become compliant, or refuse to receive care, for example, food, medicine	AGS (changes in activity patterns or routine)
	34	Prolonged resting	Long resting periods without apparent reasons
	35	Altered sleep cycle	Changed sleep–wake cycle, for example, long sleeps during the day
	36	Altered routines	Changed the order or timing of activities from the norm
Domain 6: The Body	37	Profuse sweating	Excessive sweating in various parts of the body excluding circumstances due to environmental factors such as no air conditioning or lack of proper ventilation	AGS
	38	Pale/flushed (red) face	Color faded or red-colored face
	39	Feverish/cold	Changes in body temperature either too hot or too cold
	40	Rapid breathing	Fast rate of breathing
	41	Painful injuries	Injuries are known to induce pain, for example, falls, bed sores, active wounds
	42	Painful medical conditions	Conditions known to cause pain including currently presented, for example, dental infections, urinary tract infections, or previously documented chronic conditions in medical history, for example, arthritis

Abbreviations: ADL, activities of daily living; AGS, American Geriatric Society; AU, action unit; ePAT, electronic Pain Assessment Tool; FACS, Facial Action Coding System.

Each domain provides a checklist of pain indicators from which the user makes binary selections (ie, present yes/no) for each indicator on the smart device touch screen based on clinical observations of the patient.^31,61 Domain scores are automatically calculated to provide a final pain score. Based on the published results of the validation study where ePAT scores were compared to APS cutoff scores (no pain: 0–2; mild pain: 3–7, moderate pain: 8–13, severe pain: 14 or more), the following categorical ratings have been derived: no pain: 0–6; mild pain: 7–11; moderate pain: 12–15; severe pain: 16 or more.³¹ Each domain also has a blank field at the bottom of the screen for the user to record any additional and/or relevant observation(s).^31,61

The Face domain (AU score) of ePAT was blindly evaluated against self-reporting (gold standard) measures (visual analog scale [VAS], numerical rating scale [NRS], and verbal rating scale [VRS]) of cognitively intact people with chronic pain (n=43 [21 male, 22 female], mean age=54±14 years) in unpublished study.⁴⁰ When the AU score was classified into two groups (0–2 vs 3 or more), it was highly correlated with the gold standard measures of pain (t-tests or Wilcoxon: p<0.0001 for each measure). These measures were then classified into two groups (low or high pain) as follows: VAS: 0–50 vs 51–100; VRS: 0–3 vs 3.5–5; NRS: 0–4 vs 5–10. Cross tabulations of the categorized AU score against these binary variables showed that a high AU score had over 95% sensitivity to identify high pain scores and high specificities (69%, 90%, and 95% for each measure, respectively). Participants were classified into those recording high pain on any of the three validated measures vs low pain on all measures. The AU score was able to identify high pain with 95.7% sensitivity and 95% specificity.⁴⁰

In a published study by Atee et al, the complete ePAT tool was tested in 40 residents (aged 60–98 years) with moderate-to-severe dementia (Psychogeriatric Assessment Scale–Cognitive Impairment Scale scores: 10–21) from three RACFs in Western Australia. Based on 353 paired pain assessments, the tool demonstrated excellent concurrent validity (r =0.882, 95% CI: 0.857–0.903), good discriminant validity (random regression model is not timing-dependent, p=0.795), good interrater reliability (weighted κ= 0.74, 95% CI: 0.68–0.80), and excellent internal consistency (Cronbach’s alpha (α)=0.925).³¹ This observational study assessed the psychometric properties of ePAT compared to the APS, which is the widely used observational pain scale for people with dementia in Australia.³¹ These findings were also confirmed in another cohort of people (n=34, 68.0–93.2 years old) with similar demographics.⁴¹

Setting

Single-site, accredited, high-care, and dementia-specific RACF. The facility has a capacity of 65 beds and is located in Perth, Western Australia.

Participants

Pain raters (users)

Raters were aged care staff working in the facility using the ePAT as an assessment scale of pain. Staff were recruited if they had been working for 3 months or more in the facility, were familiar with residents, able to converse in English, and keen to participate in the study. Staff were excluded if they had fears associated with using technologies or were likely to be absent for any period during the study.

Residents (subjects)

Residents were included in the study if they had moderate-to-severe dementia as indicated by Dementia Severity Rating Scale (DSRS) scores >18, and had documented behavioral problems and a history of painful conditions. Patients were excluded from the study if they were deemed medically unfit for participation.

Protocol

This substudy was a 2-week observational study, in which a convenience sampling technique was employed. Staff who consented to participate attended an education and training program prior to the study. The program involved a single session, which was conducted by the principal investigator over 4.5 hours at the study site. Attendees received education about pain, pain and dementia, pain assessment, and pain management. The contents of the program were developed after reviewing the International Association for the Study of Pain Curricula⁴² and current literature with modifications made appropriate to the setting and demographics of raters. The session also included a demonstration of the ePAT and practical training on its use.

Staff rater data were collected using a 14-item questionnaire, which included a mix of open- and closed-ended questions. The questionnaire was piloted using five test subjects prior to administration to ensure readability and ease of completion.

Testing of the ePAT was undertaken indoors at the RACF in September 2016. Testing involved the use of the ePAT by pairs of independent staff raters who were blinded to each other’s assessments, scores, and to the use of analgesics. Raters were instructed to conduct their assessments independently using own ePAT device without consulting or conversing with the other rater involved in the study. No discussions were made regarding each assessment, and scores obtained were not shared nor exchanged between paired raters. One of the study authors monitored the data collection process to ensure that this was being followed throughout the study.

Automated facial analyses were conducted consecutively to allow each rater access to a full frontal view of the resident and prevent any possible discrepancies (eg, physical hindrance) that might arise during the process. Paired ratings were scheduled randomly to reduce learning bias and subsequent systematic error. Ratings also occurred within a time frame of 2–3 minutes to ensure that the results obtained were comparable. As far as possible, recording conditions of automated facial analyses (eg, lighting, distance from subject) were essentially the same for all cases.

Residents with dementia were assessed for pain during routine nursing activities or activities of daily living (ADL) that involved mobilization and during periods of rest. Over the study period (ie, 2 weeks), each resident was assessed by two different raters on four separate occasions: on each of the 2 days ~1 week apart, the assessors rated the resident’s pain while at rest and shortly afterward while receiving care activities. Raters were instructed to observe the resident under the assessment for pain-induced behaviors for at least 5 minutes before commencing pain scoring on the ePAT.

Statistical analyses

Descriptive statistics (eg, mean, range, standard deviation) were used to summarize the profiles of the raters, residents, and pain scores including automated facial scores. Agreement on categorical pain data was evaluated using kappa statistics. The kappa coefficient measures interrater reliability or the agreement between two observers and takes into account the agreement expected by chance. It is, therefore, a more robust measure than percentage agreement.⁴³ A value of 0.6 or above indicates moderate agreement or good interrater reliability.⁴³ Cohen’s kappa (κ) statistic was used to assess agreement between raters on the presence or absence of pain, whereas weighted kappa (κ_w) was employed to evaluate agreement when pain was divided into >2 categories. Agreement on continuous pain data (ie, total pain scores) was measured by Lin’s concordance correlation coefficient (CCC).⁴⁴ Values of CCC range from 0 to ±1 where +1 is perfect concordance and −1 is perfect discordance. To assess the strength of agreement, we used Altman’s criteria as a guide to interpret CCC values: <0.20=“poor” and >0.80=“excellent”.⁴⁵ Using a published chart of the score range of the ePAT, total pain scores were allocated into broad pain categories: no pain (0–6), mild pain (7–11), moderate pain (12–15), and severe pain (16–42).³¹ Further, a regression model was used to examine the relationship between automated facial scores and total instrument scores (pain vs no pain) of ePAT under various testing conditions.

Level of significance was expressed by 95% CI range or p-value <0.05. All data were analyzed using the Statistical Package for the Social Sciences (SPSS), Version 24 Software (SPSS Inc., Apache Software Foundation, Chicago, IL, USA).

Results

Demographic data

Demographics of resident sample

Ten residents with an age range of 63.1–84.4 years (mean: 74.4±5.9 years) were recruited into the study. The gender ratio of residents was 50:50 and the vast majority (90%) were Caucasians. Half of the residents had Alzheimer’s dementia and 80% were classified as having severe dementia (mean DSRS score: 46.3±8.4). Table 2 provides demographic characteristics of resident sample.

Table 2.

Demographic characteristics of resident sample at baseline (n=10)

Characteristics	Number (%)	Mean (SD)
Age (years) (range: 63.1–84.4)
Female		78.2 (5.2)
Male		70.6 (5.0)
Overall		74.4 (5.9)
Gender
Female	5 (50)
Male	5 (50)
Ethnicity
Caucasian	9 (90)
Other	1 (10)
Primary language
English	10 (100)
Mobility
Limited	4 (40)
Immobile	4 (40)
Bedridden	2 (20)
Dementia severity (DSRS range: 30–54)		46.3 (8.4)
Diagnosis of dementia
Alzheimer’s disease	5 (50%)
Alcoholic-related dementia	2 (20%)
Frontotemporal dementia	1 (10%)
Parkinson’s dementia	1 (10%)
Vascular dementia	1 (10%)

Abbreviations: DSRS, Dementia Severity Rating Scale; SD, standard deviation.

Movement-based activities ranged from independent (eg, walking) to assisted (eg, transfer) events.

Demographics of rater cohort

A cohort of 11 staff with a mean age of 45.3±13.4 years were recruited into the study, two of whom were male. Working hours in the facility ranged from 20 to 38 hours per week with five staff employed as fulltime (ie, 38 hours/week). The average length of staff employment in the facility was 10.6±9.1 years. The cohort included 10 nurses of various hierarchical roles (one clinical nurse, four registered nurses, and five enrolled nurses), plus a trained carer. Range of nursing or caring experience among staff varied from 1 to 30 years, while aged care experience was 1–33 years. The mean years of experience in cognitive impairment or dementia care were 11.5±7.9 years. All staff reported receiving pain education in the past. Demographics of raters are shown in Table 3.

Table 3.

Demographic characteristics of rater cohort (n=11)

Characteristics	Number (%)	Mean (SD)
Age (years)		45.3 (13.4)
Gender (female)	9 (81.8)
Ethnicity
Caucasian	6 (54.5)
Asian	3 (27.3)
Other	2 (18.2)
Primary language
English	8 (72.7)
Other	3 (27.3)
Employment status (hours)		33.3 (7.2)
Part time (range: 20–26)	6 (54.5)
Full time (38 hours)	5 (45.5)
Years of experience
Nursing/caregiving		15.5 (11.8)
Aged care		15.1 (11.1)
Cognitive impairments/dementia care		11.5 (7.9)
Employment in facility		10.6 (9.1)
Role in facility
Enrolled nurse	5 (45.4)
Registered nurse	4 (36.4)
Clinical nurse	1 (9.1)
Personal care worker	1 (9.1)
Past pain education
Yes	11 (100)
No	0 (0)
Last received pain education
<3 months	1 (9.1)
<12 months	1 (9.1)
>12 months	1 (9.1)
>3 years	1 (9.1)
Not specified	7 (63.6)

Abbreviations: SD, standard deviation.

Pain data

All residents had four pairs of ePAT ratings over the 2-week study period except one resident who had only two pairs during the same period. This resulted in a total of 76 assessments for the sample. Of these, almost two-thirds (65.8%) were scored as “no pain” while less than a third (29%) scored “mild pain” as shown in Table 4. Pain-associated conditions documented for residents were diverse with 80% of the sample having two or more chronic painful conditions. Residents had a mean pain score of 5.6±3.5 (median=5) with a range of 1–18. Table 4 provides a description of pain-related data in residents who underwent pain assessment using the ePAT.

Table 4.

Pain-related data of residents (n=10) who underwent pain assessment using the ePAT

Variables	Number (%)	Mean (SD)
Pain assessments	76 (100)	7.6 (1.3)
Rest	38 (50)	3.8 (0.6)
Movement	38 (50)	3.8 (0.6)
Pain scores (median: 5, range: 1–18)		5.6 (3.5)
Pain categories
No pain	50 (65.8)
Mild pain	22 (29)
Moderate pain	2 (2.6)
Severe pain	2 (2.6)
Number of documented chronic painful diagnoses (median: 3, range 1–5)		3.0 (1.6)
1	2 (20)
2	4 (40)
3	1 (10)
4	1 (10)
5	2 (20)
Prescribed analgesia
Regular		2.0 (1.5)
PRN		0.4 (0.8)
Non-opioid analgesics
Regular
Celecoxib capsules 100 mg	1 (10)
Diclofenac gel 11.6 mg/g	1 (10)
Paracetamol tablets 500 mg	5 (50)
Paracetamol tablets SR 665 mg	1 (10)
Paracetamol oral liquid 240 mg/5 mL	1 (10)
PRN
Paracetamol tablets 500 mg	2 (20)
Opioid analgesics
Regular
Fentanyl patches 12 mcg/hour	3 (30)
Oxycodone tablets 5 mg	2 (20)
PRN
Oxycodone tablets 5 mg	1 (10)

Abbreviations: ePAT, electronic Pain Assessment Tool; PRN, pro re nata [when necessary]; SD, standard deviation.

Interrater reliability data of the ePAT instrument

Kappa statistics

Rater agreement in broad categories of pain (no pain, mild, moderate, or severe pain) using kappa statistics was classified as excellent (κ=1.0) at rest, where both raters agreed on the absence of pain on 17 occasions, and mild pain on two occasions (Table 5). With movement, agreement was moderate (κ=0.59), but assessments were in complete agreement for 13 (68.4%) out of the 19 paired assessments; the remaining six pairs differed only by one category.

Table 5.

Agreement between raters in their assessments of total pain: kappa statistics for total pain scores within pain categories, and using raw pain scores

Activity	Broad pain categories Cohen’s kappa (κ)	95% CI	Raw total pain scores weighted kappa (κw)	95% CI
Rest	1.0	Not applicable	0.72	0.58–0.86
Movement	0.59	0.27–0.91	0.69	0.50–0.87

Note: The figures are based on the 19 pairs of measurements with movement and 19 pairs without (ie, at rest).

Lin’s concordance analysis

Lin’s concordance correlation coefficient (CCC) was used to calculate agreement between total score values produced by the paired raters. The value of CCC was calculated to be 0.92 (95% CI: 0.85–0.96), which is classified as an excellent agreement.⁴⁶

Means and standard deviations of total pain scores and facial scores of ePAT at various occasions

The difference between the pairs of measurements in producing raw total pain scores while performed at rest and with movement appeared to be small, as suggested by the kappa statistics (Table 5). A linear model confirmed that the agreement was very good, and that the agreement did not depend on the conditions (rest vs movement; p=0.91). In this model, the resident identifier was named as a random effect, the dependent variable was the difference in measurements made by the two raters on each occasion, and the independent variable was the condition (rest/movement). Because the p-value associated with “condition” was not significant, this suggested that the agreement between raters was similar for both conditions. In addition, the intercept obtained from the model (overall mean) was close to zero (0.05; p=0.87), suggesting that there was no consistent bias between raters.

The mean of the pain assessments made on each occasion by the two raters was calculated (n=38 occasions), and these were entered into a random-effects model to compare the measurements made at rest with those taken with movement. The model showed that the mean scores seen with movement (7.3±3.7) were significantly higher than those observed at rest (4.0±2.2; p<0.0001; standard error [SE] estimated from the regression model: 0.81). Similarly, the scores on the Face domain were significantly different (p<0.0001; SE: 0.17) between those taken with movement (mean: 2.5±0.6) and those at rest (mean: 1.7±0.7). These data are presented in Table 6.

Table 6.

Means and standard deviations of total pain scores and facial scores of ePAT at various occasions

	N (%)	Total ePAT scores (mean±SD)	Automated facial (AU) scores (mean±SD)
Episode
No pain	50 (65.8)	3.6±1.7a	1.9±0.8b
Pain	26 (34.2)	9.3±2.9a	2.5±0.6b
Occasion
Rest	38 (50%)	4.0±2.2c	1.7±0.7c
Movement	38 (50%)	7.3±3.7c	2.5±0.6c

Notes:

^aThe two-tailed p-value is <0.0001.

^bThe two-tailed p-value equals 0.0012.

^cThe two-tailed p-value is <0.0001.

Abbreviations: AU, action unit; ePAT, electronic Pain Assessment Tool; SD, standard deviation.

Discussion

Our study aimed to investigate the reliability of a new tool, named the ePAT in individuals with moderate-to-severe dementia living in RACFs. The tool takes advantage of advanced computational capabilities together with the cameras in smart devices and automated facial recognition technology to identify the presence and severity of pain.^31,61

Our findings suggested that agreement between raters was greater during rest (Table 5) because fewer behaviors were observed and hence recorded. In contrast, during movement, pain-induced behaviors are likely to increase because of the experienced nociceptive stimuli associated with movement-related activities (eg, turning).^47,48 Some pain behaviors incorporated in ePAT are subtle and difficult to identify by raters, which may contribute to some degree of interrater variability.⁴⁹ This variation in agreement between rest and movement is consistent with other observational pain assessment tools as indicated by these kappa value ranges: The Checklist of Nonverbal Pain Indicators (CNPI): 0.625–0.819, Mahoney Pain Scale (MPS): 0.55–0.77, Mobilization-Observation-Behavior-Intensity-Dementia-2 (MOBID-2): 0.44–0.90 for observed behaviors.⁵⁰

With regard to automated facial scores produced by the ePAT, the mean values were significantly higher (p<0.001) for “pain” events compared to those recorded under “no pain” (Table 6). This indicates that the facial AUs are sensitive to aversive events that trigger painful stimuli. This was also supported by Lints-Martindale et al in a study investigating facial reactions to experimentally induced pain stimuli.³⁵ They found that noxious electrical stimuli produce much greater FACS activity with “pain” compared to “no pain” events.³⁵ Automated software scoring is also a reliable way of recognizing expressions in comparison with human observation. In a study by Peter Lewinski, automated facial coding software (eg, FaceReader) outperformed human observers in recognizing neutral faces by 31%.⁵¹ Bartlett et al also found that automated decoding of facial expressions was far superior (85% accuracy) in identifying genuine from fake pain compared to untrained and trained observers (50% and 55%, respectively).⁵²

A significantly higher mean facial score using automated facial analysis was also observed on movement compared to that during rest (Table 6). Research suggests that joint movements generate shear forces on the axolemma of the “free” nerve endings resulting in nociceptive signals as pain.⁵³ Our results are similar to those of Hadjistavropoulos et al who found that more facial activity was produced in movement activities.¹⁷ They reported that the FACS score was significantly greater during walking compared to reclining or transferring.¹⁷ In addition, the difference in the average number of AUs detected for residents after movement was significantly greater (p<0.0001) than for those at rest.

Current research suggests that integrating automated FACS descriptors with observational tools is psychometrically sound and clinically useful.^27,54 Beach et al also support this endeavor, reporting that pain-relevant FACS scores and modified PAINAD scale scores were highly correlated in older adults with Alzheimer’s dementia.⁵⁴ Pain-related FACS was also found to be clinically relevant for inclusion in observational pain assessment scales designed for people with dementia.⁵⁴ Observational tools with pain-related AUs have also shown higher sensitivity and better psychometric properties than those that contain generic facial expression descriptors.²¹

Lin’s CCC is a relative index of reliability where agreements on total pain numerical scores are compared. Our statistical analysis showed an excellent agreement (CCC=0.92). As far as we know, there are no CCC values of pain assessment scales in dementia reported in the literature. CCC values were previously reported for observational pain scales in other noncommunicative populations such as the Nonverbal Pain Assessment Tool (NPAT): 0.21–0.72 (95% CI).⁵⁵ Our results demonstrated higher values than NPAT.

Our study has tested the interrater reliability of a novel tool that integrates pain-relevant FACS items (ie, facial AU codes) with other communicative (eg, vocalization items), protective (eg, guarding), and subtle (eg, resistance to care) pain behaviors. This approach has emerging support in the literature.^27,54,56 The total number of pain behaviors is also significantly related to self-reported pain intensity in older adults.⁵⁷ It is essential to highlight that using an observational pain assessment tool improves detection of presence and severity of pain in people with cognitive impairment.⁵⁸ Further, ePAT uses automated facial recognition and analysis to detect pain-relevant AU codes.^31,61 Given that patients with dementia have an enhanced facial activity as illustrated in previous studies^16,20,35 and that observational tools improve pain recognition in this population,⁵⁸ we believe that ePAT can facilitate the process of pain detection in these patients.

Strengths and limitations

This study had several merits and limitations. Generalizations to other settings and populations are limited by the sampling method (ie, convenience, purposive sampling) and sample size. Therefore, the risk of committing Type II errors in this study remains a possibility. Despite the small sample size, an equal number of pain assessments were performed on most (ie, 9 out of 10) residents. The resident cohort was homogenous although it lacked ethnic diversity. Gender and cultural disparities were only evident in the rater group. This group had a diverse range of skills representative of the hierarchical workforce in the residential aged care setting. Learning effect associated with repeated use of the tool on the same subject is inevitable in agreement studies. The short time frame of the study may have influenced how raters remembered pain-related behaviors and how they may carry forward this information to the following week because of memory bias. Pain assessments were delivered during clinical rounds while residents were receiving their standard care, in order to minimize interference to work-flow. As such, this perhaps contributed to variations in pain scores, which are associated with consecutive delivery of the assessments, individual observation skills of a rater to record nonfacial pain-related behaviors, and the general subjective nature of pain. In addition, assessments were delivered during ADL, such as walking, to provide a real-world context of actual use of the tool in clinical scenarios.

In the current study, we tested the interrater reliability by comparing two ePAT users. Head-to-head comparative studies of observational pain assessment scales can provide valuable data to guide the process of tool development and refinement.⁵⁶ Further, interrater reliability is one of the key psychometric properties of observational scales because arriving at similar pain scores by different clinicians provides confidence in the tested tool.²⁷ In our study, there was a small number (26 out of 76) of “pain” cases detected, perhaps due to the adequate pain management in the sample. Although we acknowledge this limitation, we believe the testing discussed here is sufficient to address the objectives of the study. In fact, identifying “pain” from “no pain” or neutral cases consistently is considered a useful criterion of reliability in judgment studies.²⁷ Our findings were based on clinical validation (ie, clinical pain from ADL) and, therefore, results obtained from experimental studies (pressure or temperature pain-induced modalities) may vary. Notwithstanding, there is some evidence that experimental pain response is different from clinical pain response and that the predictive value of experimental pain for clinically induced pain is weak and not reliable.⁵⁹ Lichtner et al in their systematic meta-review and Closs et al in their meta-review recommended that validation work should be conducted in clinical settings, so that it informs the applicability of the tool and its potential value in everyday clinical practice.^50,60 This is because pain assessment tools that are experimentally tested in research do not necessarily transfer easily and effectively in clinical settings.⁶⁰

In the study design, we allowed access of raters to all available information (except for analgesics) to minimize the chances of underestimating pain. An equal access to medical profiles by both raters means that raters were well informed about the patients’ diagnoses of possible painful chronic conditions. This strategy may have strengthened raters’ evaluation when conducting clinical pain assessments.

Another strength of the study is that various reliability measures were used including kappa, weighted kappa, and CCC. Reliability statistics that consider chance agreement between raters will account for the variation in frequency of AUs distribution. This is important because it will assist in extrapolating the findings into other populations. However, measurement errors are still possible because of confounding effects linked to uncontrolled conditions inside the aged care facility such as lighting, shadowing, and random movement that might have affected the performance of the tool.

Conclusion

Facial scores were significantly higher during “pain” compared to those scores clinically recorded as “no pain”. Similarly, automated scoring of facial AUs was higher for residents with movement compared to rest. This indicates that the Face domain of the ePAT has a good sensitivity to the presence of pain. Combining automated facial expression analysis and clinical behavioral indicators in a single observational pain assessment scale affords ePAT good reliability properties. This supports its appropriateness for use in nonverbal residents with advanced dementia. Reliable clinical tools particularly for pain assessment are desired to improve therapeutic outcomes. It should be stressed, however, that currently there is no gold standard pain assessment tool available for noncommunicative people with dementia, and any attempt to work toward this goal must be encouraged. Innovative approaches of pain assessment such as those included in the ePAT can assist clinicians to more objectively assess pain in challenging populations, such as those with dementia.