Diurnal Pain Classification in Critically Ill Patients using Machine Learning on Accelerometry and Analgesic Data

Abstract

Quantifying pain in patients admitted to intensive care units (ICUs) is challenging due to the increased prevalence of communication barriers in this patient population. Previous research has posited a positive correlation between pain and physical activity in critically ill patients. In this study, we advance this hypothesis by building machine learning classifiers to examine the ability of accelerometer data collected from daily wearables to predict self-reported pain levels experienced by patients in the ICU. We trained multiple Machine Learning (ML) models, including Logistic Regression, CatBoost, and XG-Boost, on statistical features extracted from the accelerometer data combined with previous pain measurements and patient demographics. Following previous studies that showed a change in pain sensitivity in ICU patients at night, we performed the task of pain classification separately for daytime and nighttime pain reports. In the pain versus no-pain classification setting, logistic regression gave the best classifier in daytime (AUC: 0.72, F1-score: 0.72), and CatBoost gave the best classifier at nighttime (AUC: 0.82, F1-score: 0.82). Performance of logistic regression dropped to 0.61 AUC, 0.62 F1-score (mild vs. moderate pain, nighttime), and CatBoost’s performance was similarly affected with 0.61 AUC, 0.60 F1-score (moderate vs. severe pain, daytime). The inclusion of analgesic information benefited the classification between moderate and severe pain. SHAP analysis was conducted to find the most significant features in each setting. It assigned the highest importance to accelerometer-related features on all evaluated settings but also showed the contribution of the other features such as age and medications in specific contexts. In conclusion, accelerometer data combined with patient demographics and previous pain measurements can be used to screen painful from painless episodes in the ICU and can be combined with analgesic information to provide moderate classification between painful episodes of different severities.

I. Introduction

Pain diagnosis is critical in the ICU as several adverse outcomes are associated with underdiagnosed pain, including increased infection rate, prolonged mechanical ventilation, hemodynamic derangements, delirium, and compromised immunity [1]. Moreover, inadequate management of pain can have serious physiological and psychological effects [2]. Research indicates that appropriate pain management, adequate analgesia, and less sedation can help reduce the number of days spent on the ventilator, increase mobility, and decrease the incidence of delirium and length of stay in the ICU [3], thus improving quality of patient care.

Critically ill patients frequently experience pain, and despite the existence of clinical scoring systems to quantify pain, its assessment in intensive care units (ICUs) appears to be scarce and inconsistent [4]. Further, many critically ill patients are incapable of communicating clearly. Pre-existing factors such as native language differences, history of cognitive deficit, developmental disability, or certain psychiatric disorders might prevent communication with caregivers. Furthermore, interventions such as endotracheal intubation, tracheostomies, and medical sedation, coupled with the increased prevalence of delirium and altered mental status in ICUs, exacerbate patients’ communication difficulties [5]. In non-verbal patients, pain levels are assessed by ICU nurses based on the Behavioral Pain Scale (BPS), Critical Care Pain Observation (CPOT), and Non-Verbal Pain Scales (NVPS) [6], [7]. Due to the human component involved, visual pain assessment is infrequent and often inconsistent with established guidelines [8]. Automated pain detection using deep learning may obviate this problem.

Presently, the most common models to autonomously predict patients’ pain using machine learning in the ICU have used videos of patients’ facial expressions [9], [10]. Despite the high model performance, privacy concerns arising from the storage of sensitive patient information present a roadblock to the widespread clinical acceptance of these models. On the other hand, some studies have investigated the feasibility of pain detection using vital signs [11], [12]. However, research indicates that vital signs are not strong indicators of pain [13]. In contrast, accelerometers worn on patients’ hands, ankles, and wrists can collect a rich repertoire of data concerning patient mobility which has been associated with pain [14], [15].

Wearables, encompassing diminutive and unobtrusive accelerometers often taking the form of wristwatches, hold the distinct advantage of seamlessly integrating into patients’ routines. This integration is achieved without compromising patient safety, comfort, or the smooth operation of ICU care procedures. This unobtrusive nature ensures that their presence does not disrupt the intricate operations concerning patient care within the ICU environment. Additionally, their cost-effectiveness is a noteworthy attribute, offering a readily available avenue for continuous sensory data capture. Its data, in turn, provides valuable insights spanning a spectrum of domains such as mobility patterns, sleep quality, overall comfort, and levels of sedation experienced by the patients [16].

This study comprehensively assessed the efficacy of three distinct machine-learning techniques for the classification of pain scores using primarily accelerometry data. These classifiers leveraged statistical features derived from accelerometer data, complemented by prior pain measurements and patient demographics. We hypothesized that accelerometry data, in combination with previous pain assessment and demographics, will be able to robustly classify painful from painless ICU episodes and moderately classify between episodes of varying pain severity in ICU patients. Therefore, the evaluation encompassed three distinct scenarios: a) Pain vs. No Pain, wherein we scrutinized the methods’ capacity to screen for painful periods; b) Mild vs. Moderate; and c) Moderate vs. Severe, where we scrutinized the methods’ effectiveness in distinguishing between varying pain severity classes.

Our contributions are as follows: a) to the best of our knowledge, this is the first study evaluating the effectiveness of accelerometer data in classifying pain levels; b) we performed pain/no-pain as well as mild/moderate and moderate/severe classifications c) we performed a SHAP (SHapley Additive exPlanations) [17] analysis to interpret the importance of different features on our models.

II. Methods

A. Patient Recruitment

The data included in this analysis were collected via three clinical studies conducted at the University of Florida (UF) Health Shands Hospital in Gainesville, Florida, USA. The UF Institutional Review Board approved these studies, and all patients provided written informed consent prior to study enrollment. For patients unable to provide self-consent, a legally authorized representative (LAR) assented on their behalf. For the purpose of brevity, these studies will hereafter be referred to as I2CU, PAIN, and ADAPT, respectively.

Patients were considered eligible to participate if they were over the age of 18, admitted to a UF Health Shands intensive care unit (ICU), and expected to remain in the ICU for at least 24 hours after the time of consent. Participant exclusion criteria included expected discharge, transfer, or expiration within 24 hours of study initiation, any form of isolation or contact precaution, and lack of LAR for patients unfit for self-consent. Examples of instances where motion data could not be collected included the existence of intravenous lines, wounds, and patient objection to wearable devices. We excluded patients for whom accelerometer data collection or retrieval was impossible.

I2CU:

Data and informed consent were obtained from 70 participants between July 2016 and December 2019. Their biological sex composition was 61.0% male and 39.0% female, with a mean age of 60.8 years. Patients wore ActiGraph GT3X+ devices on their wrists, which collected data at a sampling rate of 100 Hz. The average data collection length was 2.93 days.

PAIN:

Data and informed consent were collected from 29 patients between June 2021 and December 2021. Their biological sex composition was 64.5% male and 35.5% female, with a mean age of 57.9 years. These patients wore Shimmer3 accelerometers on their wrists, which collected data 100/512 Hz sampling rates. The average data collection length was 4.61 days.

ADAPT:

Data and informed consent were collected from 29 patients between January 2022 and June 2022. The biological sex composition of these participants was 68.6% male and 31.4% female, with a mean age of 55.7 years. These patients wore either Shimmer3 or ActiGraph GTX+ devices with sampling rates of 20 Hz, 30 Hz, 100 Hz, or 512 Hz. The mean data collection duration was 4.46 days. Patient recruitment is ongoing.

B. Data Collection

Accelerometer data were collected for up to 7 days or till discharge from the ICU, whichever was sooner. Data collected using ActiGraph devices were downloaded using the ActiLife toolbox [https://actilife.theactigraph.com/actilife/]. Accelerometer data collected via the Shimmer device was uploaded and exported to a secured server using Consenysys software [https://www.consensys.net/]. UF’s Integrated Data Repository service extracted the patient’s relevant clinical data from electronic health records (EHR). These data included demographics (age, sex, height, weight, race, and ethnicity), hospital admission, and self-reported pain scores. In this study, we used previous self-reported pain as additional context data to the accelerometer-based model.

Table I shows the distribution of demographic and clinical variables for the patients included in the analyses. Most study participants were elderly, white people with comorbidities. Pain scores were divided into three groups: (1) No Pain; pain score = 0, (2) Mild Pain; pain scores = 1–4, (3) Moderate Pain; pain scores = 5,6, and (4) Severe Pain; pain scores = 7–10. There were 2855 samples in total comprising both daytime and nighttime episodes. Daytime episodes consisted of 1757 samples, where 927 (52.76%) samples were from no pain periods, 435 (24.75%) from mild pain, 179 (10.18%) from moderate pain, and 216 (12.29%) from severe pain episodes. Nighttime consisted of 1098 samples, where 575 (52.36%) samples were from no pain episodes, 210 (19.12%) from mild pain, 142 (12.93%) from moderate pain, and 171 (15.57%) from severe pain episodes.

TABLE I:

Clinical characteristics of the patient cohort.

Variables	Patients (N=128)
Female sex, N(%)	41 (32.03)
Age in years, mean (SD)	59.31 (16.22)
Height in cm, mean (SD)	172.97 (10.64)
Weight in kgs, mean (SD)	87.09 (25.43)
Length of stay in days, mean (SD)	22.47 (27.28)
Race, White(%)/ Black (%)/ Other (%)	83%/ 11%/ 6%
Cancer, N(%)	17 (13.28)
Cerebro-vascular, N(%)	15 (11.71)
Dementia, N(%)	6 (4.68)
Paraplegia Hemiplegia, N(%)	5 (3.90)
Congestive Heart Failure, N(%)	22 (17.18)
Chronic Obstructive Pulmonary Disease, N(%)	20 (15.62)
Diabetes, N(%)	35 (27.34)
Metastatic Carcinoma, N(%)	7 (5.46)
Liver, N(%)	28 (21.87)
Peptic Ulcer, N(%)	7 (5.46)
Renal, N(%)	30 (23.43)
Rheumatologic, N(%)	2 (1.56)

C. Analysis

To examine the viability of using accelerometer data to classify pain levels, we used the general analysis pipeline shown in Figure 1. This conceptual workflow consists of three steps: pre-processing, feature extraction, and modeling.

Fig. 1:

Pain classification pipeline.

1). Data Pre-processing:

We collected 15-minute windows of accelerometer data, starting 30 minutes before the pain assessment. We excluded the 15-minute window immediately prior to pain assessment because this contained artifacts of the patient’s interaction with the caregiver. Since the accelerometer data had different sampling frequencies depending on the device or time of the collection, we downsampled all data to 10Hz following [18], which showed that human activities are between 0–20 Hz, and 98% Fast Fourier Transform (FFT) power in accelerometer data is contained in the 0–10 Hz range. Furthermore, we rescaled the accelerometer values to a range between 0 and 1 for the ML methods evaluated. Similarly, numeric demographic information (e.g., age) was rescaled to between [0, 1], and categorical demographic information (e.g., sex and race) was one-hot encoded. Previous pain scores were also rescaled to between [0, 1]. Throughout our analysis, 7 am to 7 pm was considered daytime, while 7 pm to 7 am was considered nighttime.

Analgesic information was added to the classifiers to test whether they improved classification above and beyond the accelerometry, previous pain, and demographics information. Analgesics were initially one-hot encoded by the medication name. If a patient received an analgesic, we assessed the proportion of time during a 15-minute interval when the patient was under the influence of the analgesic. This information was calculated based on the time of the last dose and the half-life of the medication in question.

2). Feature Extraction:

To assess the physical activity recorded by the accelerometers, we extracted the activity counts vector across the three cartesian axes (X, Y, and Z) of the accelerometer. The activity counts vector represents a person’s physical activity levels over a certain period of time along a particular axis. The activity counts are calculated once every minute for the 15-minute windows for each axis. Then, we derived six statistical attributes from the mean activity count vector: a) average, b) standard deviation, c) maximum, d) the percentage of time spent immobile (POI), e) skewness, and f) kurtosis. Skewness and kurtosis were used to characterize the tail of the activity count vector’s distribution.

3). Parameter Tuning and Validation:

The patient population was divided into five mutually exclusive parts using cross-validation stratified folds with non-overlapping patient groups. The folds are made by preserving the percentage of samples for each class. It ensures that samples from the same patient remain within the same fold during cross-validation. This was essential to prevent information leakage between different folds during cross-validation. We tuned the hyperparameters of three different ML classifiers, namely Logistic Regression, CatBoost, and XGBoost, within a 5-fold cross-validation setting and reported F1-score, Recall, and Area under the Receiver Operating Curve (AUC). For all metrics, we calculated a 95% confidence interval.

III. Results

A. Evaluation Results

We evaluated the relation between actigraphy and pain level in three scenarios: no pain vs pain, mild vs moderate pain, and moderate vs severe pain. To account for the diurnal change in pain sensitivity in patients, which has been reported in past studies [14], we examined pain classification separately for daytime and nighttime. This distinction was further made necessary because baseline activity levels in ICU patients are different in the daytime versus at nighttime.

Table II provides an overview of the performance of the evaluated machine learning classifiers for the prediction setting of No Pain (0 in the DVPRS scale) versus Pain (1 to 10 in the DVPRS scale). During the daytime, the logistic regression classifier achieved an F1-score of 0.72, indicating a balanced trade-off between precision and recall. Similarly, the classifier displayed a recall and an area under the receiver operating characteristic curve (ROC AUC) value of 0.72. CatBoost and XGBoost classifiers followed, with F1-scores of 0.69 and 0.65, respectively.

TABLE II:

Performance of different ML classifiers for predicting No Pain (0) vs. Pain (1–10). Results are reported as ”Metric ± 95% Confidence Interval”.

Daytime
Method	F1-score	Recall	ROC AUC
Logistic reg	0.72 ± 0.07	0.72 ± 0.05	0.72 ± 0.05
Catboost	0.69 ± 0.10	0.70 ± 0.10	0.69 ± 0.10
XGBoost	0.65 ± 0.09	0.65 ± 0.09	0.65 ± 0.09
Nighttime
Method	F1-score	Recall	ROC AUC
Logistic reg	0.70 ± 0.19	0.71 ± 0.18	0.70 ± 0.18
Catboost	0.82 ± 0.27	0.82 ± 0.27	0.82 ± 0.27
XGBoost	0.82 ± 0.24	0.82 ± 0.21	0.81 ± 0.20

The logistic regression classifier maintained its performance for nighttime predictions, achieving an F1-score of 0.70, recall of 0.71, and ROC AUC value of 0.70. In contrast, the CatBoost and XGBoost classifiers exhibited significant improvements, showcasing an F1-score of 0.82 and a recall of 0.82. The ROC AUC of CatBoost and XGBoost were significantly higher than their daytime values, at 0.82 and 0.81, respectively. These findings underscore the efficacy of machine learning classifiers in screening painful events, with CatBoost and XGBoost notably excelling in nighttime predictions.

In Table III, we evaluated the performance of the classifiers in differentiating between Mild Pain (1–4 in the DVPRS scale) and Moderate Pain (5,6 in the DVPRS scale) occurrences. During the daytime, CatBoost demonstrated the best performance with an F1-score of 0.66, a recall of 0.67, and a ROC AUC of 0.58. The logistic regression classifier displayed the highest performance for nighttime predictions with an F1-score and recall score of 0.62 and a ROC AUC of 0.61.

TABLE III:

Performance of different ML classifiers for predicting Mild Pain (1–4) vs. Moderate Pain (5,6). Results are reported as ”Metric ± 95% Confidence Interval”.

Daytime
Method	F1-score	Recall	ROC AUC
Logistic reg	0.56 ± 0.09	0.53 ± 0.10	0.54 ± 0.08
Catboost	0.66 ± 0.08	0.67 ± 0.11	0.58 ± 0.08
XGBoost	0.63 ± 0.07	0.63 ± 0.10	0.57 ± 0.06
Nighttime
Method	F1-score	Recall	ROC AUC
Logistic reg	0.62 ± 0.22	0.62 ± 0.20	0.61 ± 0.17
Catboost	0.53 ± 0.18	0.53 ± 0.19	0.53 ± 0.18
XGBoost	0.51 ± 0.15	0.50 ± 0.17	0.51 ± 0.16

The results of classification between Moderate Pain (5,6 on the DVPRS scale) and Severe Pain (7–10 on the DVPRS scale) are presented in Table IV. During the daytime, the initial approach using accelerometer data, previous pain measurement, and demographic information resulted in relatively low classification performance across all models. However, a noticeable improvement in classification performance was observed upon incorporating analgesic information alongside the aforementioned features. CatBoost gave the highest performance with ROC-AUC of 0.61 and 0.60 for the F1-score and Recall. A similar trend was evident for nighttime predictions with the initial feature space. However, unlike daytime, adding analgesic information did not improve classification performance in this setting. Across all models, the differences in performance observed between daytime and nighttime scenarios emphasize the significance of considering diurnal cycles in pain classification.

TABLE IV:

Performance of different ML classifiers for predicting Moderate Pain (5,6) vs. Severe Pain (7–10). Results are reported as ”Metric ± 95% Confidence Interval”.

Daytime
Input	Method	F1-score	Recall	ROC AUC
Accelerometer, Previous Pain, Demographics	Logistic reg	0.39 ± 0.05	0.39 ± 0.05	0.40 ± 0.05
	Catboost	0.49 ± 0.09	0.49 ± 0.09	0.48 ± 0.09
	XGBoost	0.48 ± 0.11	0.48 ± 0.11	0.48 ± 0.12
Accelerometer, Previous Pain, Demographics, Analgesic	Logistic reg	0.58 ± 0.11	0.58 ± 0.12	0.61 ± 0.11
	Catboost	0.60 ± 0.07	0.60 ± 0.07	0.61 ± 0.06
	XGBoost	0.56 ± 0.10	0.56 ± 0.11	0.56 ± 0.10
Nighttime
Input	Method	F1-score	Recall	ROC AUC
Accelerometer, Previous Pain, Demographics	Logistic reg	0.43 ± 0.15	0.49 ± 0.16	0.48 ± 0.15
	Catboost	0.44 ± 0.12	0.46 ± 0.13	0.48 ± 0.15
	XGBoost	0.44 ± 0.08	0.46 ± 0.08	0.48 ± 0.11

B. SHAP Analysis

Figure 2 shows SHAP interpretability analysis used for finding relative feature importances in different pain classification/diurnal settings. This representation established a link between patients’ behavior and their pain levels, a critical step in identifying the source of pain and initiating appropriate treatment [19]. Figure 2a shows the SHAP feature importance plot for pain vs no-pain classification during daytime. The most important feature was the maximum value of movements per minute. Higher movement was associated with lower pain in this setting. This is followed by previous pain measurement, where a higher previous pain value is predicted for higher pain measurement in the current window. The third most significant feature was the standard deviation of movement. A higher standard deviation of movement was associated with higher pain values. The combination of maximum movement being negatively predictive of pain and standard deviation of movement being positively predictive of pain indicates that during the daytime, patients suffering from pain made variable, low-intensity movements, while participants not suffering from pain made consistent high-intensity movements.

Fig. 2:

SHAP beeswarm plot illustrating feature importance. MPM is the acronym for Movement per Minute. POI is the acronym for the percentage of time spent on mobile. Best viewed in color.

Figure 2b shows the SHAP feature importances for no-pain versus pain classification at night-time. The best feature in this setting was the skewness of the movement per minute (MPM) distribution, where higher skewness indicated higher pain. Similar to the previous setting, the previous pain measurement was the second most significant feature and was directly associated with the current pain measurement. The third feature was the kurtosis of the MPM distribution. Higher values of this feature predicted lower values of pain. The positive relation between pain and skewness of the MPM distribution indicates that people in pain at night time experienced extended periods of activity at different levels and low restfulness (i.e., zero activity). The negative relation between kurtosis of the MPM distribution and pain indicates that people without pain engage in short, high-intensity activities during night-time while maintaining restfulness during the remaining periods. The difference in feature importance between daytime and nighttime reflects the difference in baseline activity levels during different times of the day. It highlights the importance of stratifying pain classifiers by the diurnal cycle.

Figure 2c shows SHAP feature importances for classifying mild and moderate pain during daytime. In this setting, the most significant feature was skewness, which had higher values predictive of mild pain. The second most significant feature was kurtosis, and higher values of this variable indicated moderate pain. The third most significant variable was age. Being older was directly related to experiencing moderate pain. The relationship between skewness and kurtosis of the MPM distribution and mild/moderate pain levels indicates that patients engage in activities of different intensities at mild pain during the daytime. In contrast, at moderate pain levels, they are generally immobile, punctuated by short episodes of intense movements.

Figure 2d shows the feature importances for mild versus moderate pain classification at night-time. This showed that the maximum movement value was positively related to moderate pain at night. This feature was followed by kurtosis of the MPM distribution, and higher values of this feature were associated with moderate pain, indicating more intense night-time activities in patients experiencing moderate pain. The third most significant variable was standard deviation, with higher values indicating moderate pain. These relationships indicate that at night, patients with moderate pain show higher movement in general (higher maximum movement, higher kurtosis, and higher standard deviation of activity) than those with mild pain.

Figure 2e shows the feature importance for daytime classification of moderate from severe pain. The most important feature in this setting was maximum movement, which is negatively related to severe pain. This was followed by standard deviation, a higher value indicating severe pain. The third most significant factor was acetaminophen, which was the most common analgesic in our dataset. Higher durations under acetaminophen influence indicated moderate pain, as expected. These relationships indicate that people experiencing severe pain in the daytime show episodes of variable low-intensity movements. In general, the results of the SHAP analysis underscore that the relation between pain and mobility is highly non-linear and is greatly affected by the time of the day. It also shows that considering analgesic dosage is important when classifying moderate from severe pain episodes during the daytime.

IV. Conclusions

This study addresses the challenge of quantifying pain in ICUs by advancing the hypothesis that the relationship between pain and physical activity in critically ill patients can be harnessed for pain prediction. We extracted different statistical features such as maximum movement, average movement, standard deviation, skewness, and kurtosis of the MPM distributions to characterize pain levels during the day and night times. Separating the samples based on time of day was essential given the different activity baselines during daytime compared to nighttime and higher pain sensitivity during restful hours. We leveraged machine learning models with different inductive biases, such as Logistic Regression, CatBoost, and XGBoost, to improve the generalizability of our classification findings. The models were able to faithfully distinguish painful from painless episodes but showered moderate performance in classifying different levels of pain. Therefore, we incorporated analgesic information to improve the classification performance between moderate and severe pain episodes. Interpretability analysis with SHAP indicated that pain has a non-linear relationship with activity level, which is governed by the time of the day. The relation between moderate to severely painful episodes was additionally influenced by the presence of analgesics.

Finally, it is pertinent to mention that the task of predicting self-reported pain scores is inherently challenging due to the subjective nature of self-reporting owing to differences in pain tolerance levels between different patients. This is the first study that has examined the feasibility of automating pain measurement in the ICU using accelerometry data. Our results indicate that accelerometry can be used to screen painful from painless episodes in the ICU but exhibits scope for improvement when classifying different pain levels.