Enhancing Neonatal Pain Assessment Transparency via Explanatory Training Examples Identification

Abstract

Deep Learning (DL)-based solutions have shown promising performance in assessing neonatal pain. However, the occlusion of the visual modality (face and body) is common in clinical settings due to several factors, including a prone sleeping position, low light, or swaddling. In such scenarios, other pain signals, such as audio signals, can be used as the major behavioral signs of pain. Although DL-based methods are proposed to assess pain from audio, these methods lack transparency and explainability (black box), which can decrease the user’s trust in the automated decision. In this work, we visualize the neonate’s audio signal as a spectrogram image to classify it as pain or no pain and present an instance-based approach for explaining the decision of the black-box model. Further, this work provides an analysis of the most helpful and harmful training instances using an influence score followed by assessing their impact on pain prediction. Experimental results demonstrate that the proposed approach can detect and remove harmful instances, eventually leading to a compressed dataset. Our results also show that the proposed work can add explainability to the current DL-based pain detection methods, which can enhance users’ trust and provide a viable approach toward pain assessment in clinical settings.

I. Introduction

Deep neural networks (DNNs) have achieved remarkable success automating and standardizing analysis in several fields, including healthcare [1]. However, there are several factors that prevent the broad adoption of DNNs in clinical practice. Examples of these factors include the limited generalizability and the lack of explainability. The majority of DL-based solutions, e.g., convolutional neural network (CNN), function as a black box [2] with no rigorous explanation about the decision-making process. This lack of explainability is a major challenge that decreases the acceptance of such solutions in medical settings as these settings require clarity and transparency in the decision-making process.

In recent years, effort [3] has been made to address this explainability issue and develop a set of strategies, procedures, and algorithms to answer the “how and why” of the decision-making process. Existing explainable artificial intelligence (XAI) algorithms for images can be divided into attribution-based or non-attribution-based. The attribution-based algorithms visualize the salient pixels of an image that are related to the model’s prediction. Some commonly used attribution-based algorithms include Class Activation Mapping(CAM) [4], Gradient-weighted CAM (Grad-CAM) [5], Local Interpretable Model-agnostic Explanations (LIME) [6] and Integrated Gradient (IG) [7]. On the other hand, non-attribution-based algorithms are helpful for model debugging, model behavior analysis, and identifying any biases in the model’s decision-making process. Examples of non-attribution-based algorithms are concept-based [8], prototype-based [9], counterfactual-based [10], and influence function-based [11] algorithms.

Given that neonates can not express their pain verbally, crying sound is one of the major behavioral indicators of neonatal pain [12]. Other behavioral indicators of pain include facial expression and body movement. However, these two indicators contain noise and are harder to capture than crying sounds [13]. This is attributed to several factors, such as the occlusion of a neonate’s face due to a prone sleeping position, swaddling, and low-light conditions; all are common in the neonatal intensive care unit (NICU). In such cases, crying sounds can be utilized as the major modality for the automated assessment of neonatal pain. In fact, our previous works [14] showed that DL-based methods can effectively measure neonates’ pain from spectrogram images.

In this paper, we extend our previous work [14], which focuses on analyzing spectrogram images of audio signals from neonates for pain assessment to integrate explainability. We implement an instance-based XAI technique to calculate the influence of each training instance on the model’s predictions for a particular test instance, allowing us to identify the most helpful and harmful training examples without retraining the model. Next, we assess the model’s accuracy changes to compress the dataset when specific percentages of the harmful and helpful data are dropped. Finally, we validate the explanation results by human evaluators to ensure that our explanation outcomes are trustworthy. Our review of the literature reveals that there is currently no work that integrates XAI methods into neonatal pain assessment methods from audio signals.

From an application standpoint, this is the first work that 1) applies XAI to estimate neonatal postoperative pain from audio signals and 2) utilizes a real-world dataset gathered in a NICU environment for evaluation. The technical contributions of this paper are summarized as follows.

• We implement an influence function-based method, which is considered a non attribution-based method, to identify the most helpful and harmful training instances for neonatal postoperative pain assessment.

• To contrast the influence score and evaluate our results, we employ a CNN-based image embedding to calculate the cosine similarity between the test and training images. Further, we use a human-grounded evaluation to ensure that the results produced by our influence function-based method are correct.

• We assess the accuracy of pain estimation after dropping the harmful and helpful training data identified by the proposed XAI method. Our results indicate that dropping harmful training instances does not decrease the accuracy; hence, these instances can be removed to create a more compact dataset without degrading the model’s performance.

II. Technical Background & Justification

This section presents the technical background and justification for using the influence function [11].

A. Influence Function Background

Problem Summary:

Let $S$ be the training set, which includes $N$ training instances. For a training instance $z_i=\left(x_i,y_i\right),x_i$ represents the input image in the input space $X$, and $y_i$ represents the corresponding label in the output space $Y$. For a test image $x_t$, a predictive model $f$, and its output ${\hat{y}}_t=f\left(x_t\right)$, the instance-based method selects the most responsible training instances $\bar{z}\in S$ of a test instance $z_{\mathit{test}}=\left(x_t,{\hat{y}}_t\right)$ using $\bar{z}={\mathit{argmax}}_{zϵS}IF\left(z_{\mathit{test}},z\right)$ as explained in [15]. IF is the influence score of a training instance $z$ for the test instance $z_{\mathit{test}}$.

The aim of the influence function is to determine the impact of each training instance on the model’s predictions by evaluating how the predictions change when a specific training instance is removed from the set $S$. For a given training instance $z$ with parameters $\theta \in \mathrm{\Theta }$ and a loss function $L(z,\theta )$, the empirical risk is denoted in [16] as $\frac{1}{n}{\sum }_{i=1}^{n}L\left(z_i,\theta \right)$. We can obtain the parameters of the predictive model by minimizing the empirical risk defined by $\hat{\theta }\stackrel{\text{def}}{=}{\mathit{arg\; min}}_{\theta ϵ\mathrm{\Theta }}\frac{1}{n}{\sum }_{i=1}^{n}L\left(z_i,\theta \right)$ as described in [16].

When removing a training instance $z$ from the set $S$, the changes in the parameters of the model are represented by $(\widehat{{\theta }_{-z}}-\hat{\theta })$, where $\widehat{{\theta }_{-z}}\stackrel{\text{def}}{=}{\mathit{arg\; min}}_{\theta ϵ\mathrm{\Theta }}{\sum }_{z_i\ne z}L\left(z_i,\theta \right)$.

As it is computationally expensive to calculate the parameter change for each removed $z$ instance to identify the responsible training instances, an efficient approximation was proposed in [16] to compute the change by upweighting $z$ by a small amount $ϵ$ and calculating the new parameters ${\hat{\theta }}_{ϵ,z}$ using (1):

$${\widehat{\theta }}_{ϵ,z}\stackrel{\text{def}}{=}{\mathit{arg\; min}}_{\theta ϵ\mathrm{\Theta }}(1-ϵ)\frac{1}{n}{\sum }_{i=1}^{n}L\left(z_i,\theta \right)+ϵL(z,\theta )$$ (1)

The influence of upweighting a training instance $z$ on the parameter $\theta$ can be calculated using (2):

$${{ℐ}_{up,param}(z)\stackrel{\text{def}}{=}\frac{{\widehat{\theta }}_{ϵ,z}}{dϵ}|}_{ϵ=0}=-H_{\widehat{\theta }}^{-1}{\nabla }_{\theta }L(z,\widehat{\theta })$$ (2)

where $H_{\hat{\theta }}\stackrel{\text{def}}{=}\frac{1}{n}{\sum }_{i=1}^n{\nabla }_{\theta }^{2}L(z_i,\hat{\theta })$ is the Hessian and a positive-definite matrix [11].

Equation (1) shows that upweighting an instance $z$ by a small amount $ϵ=\frac{1}{N}$ is equivalent to removing that instance from the dataset in terms of the effect on the model’s parameters. Therefore, the parameter change can be approximated by $(\widehat{{\theta }_{-z}}-\hat{\theta })\approx -\frac{1}{n}{\mathcal{I}}_{up,param}(z)$ without the need of retraining the model and considered as the influence on the whole model caused by removing that instance $z$. Further, to identify the key training instances for a prediction, the chain rule [11] is applied to calculate how $z$ upweighting affects $\theta$ functions. Specifically, in [11], the influence of $z$ upweighting on the loss at a test data $z_{\mathit{test}}$ has the closed-form expression as in (3):

$$\begin{gathered} {{ℐ}_{up,loss}\left(z,z_{\mathit{test}}\right)\stackrel{\text{def}}{=}\frac{dL\left(z_{\mathit{test}},{\widehat{\theta }}_{ϵ,z}\right)}{dϵ}|}_{ϵ=0} \\ =-{\nabla }_{\theta }L{(z_{\mathit{test}},\widehat{\theta })}^T{H}_{\widehat{\theta }}^{-1}{\nabla }_{\theta }L(z,\widehat{\theta }) \end{gathered}$$ (3)

B. Justification for using Influence Function-based Method

The earlier work of influence functions [16] deals with a linear model for a small dataset and analyzes the effect of perturbing or removing important training instances on the model’s parameters [16]. Recently, the concept of influence functions has been extended to large-scale datasets and used to connect important training instances to the predictions of a black-box model [11]. This recent extension inspires us to use an influence-based function for determining the training instances that are helpful or harmful to the model’s prediction.

In addition, influence-based functions, which are considered non-attribution based, have several advantages over attribution-based methods. First, attribution-based methods (e.g., Gradient Explanation [17]) are sensitive to weight randomization and suffer from repeatability or reproducibility issues [18]. Second, attribution-based methods could not pass robustness evaluations in Alzheimer’s disease classification [19]; this might suggest its instability in determining the crucial parts of the input and hence, its ineffectiveness in practical situations. Third, the saliency maps generated by attribution-based methods can be easily affected by adversarial attacks [20] as small changes in the input image can lead to large variations in the saliency map. Additionally, the bias term has a non-negligible impact on the creation of attribution maps, which is closely related to the output prediction [21].

Other non-attribution methods, such as concept-based [8], counterfactual-based [10], and prototype-based [9] have several limitations. For example, the concept-based XAI methods can produce misleading explanations and require additional annotation as a human needs to manually select concept examples. To address the issue of manual concept selection, ConceptSHAP [22] adapted a game-theoretic approach that requires in-depth mathematical knowledge to comprehend. Counterfactual-based XAI methods are also not reliable as the targeted image perturbation process used to generate explanations may not accurately reflect real-world scenarios. Finally, prototype-based XAI methods are challenging to train and vulnerable to noise [23].

As compared to the above-mentioned methods, influence functions are robust, can be integrated into any model, can be used to analyze the effect of individual training instances, and identify which training instances are degrading the model’s performance. Further, as influence functions are one type of instance-based explanation, they provide information on which training instances support the model’s predictions [15]. For example, the model’s prediction of a given image as an “infant image” is generated based on the similarity of that image to previously seen images in the training set. It has been reported [24] that users put relatively more trust in instance-based XAI methods than in visualizing the feature contribution to the decision.

III. Method

The flow of our method is illustrated in Figure 1. The first step involves converting the original audio signal into spectrogram images followed by sending them to the CNN for pain classification. Then, we calculated the cosine similarity and influence scores between the test and train images. Finally, the outcome of our explanation method is assessed by a human evaluator.

Fig. 1.

Overview of the influence function-based method integrated into the neonatal postoperative pain classification model.

A. Data Preparation and Augmentation

We extracted the audio signals from videos of infants experiencing postoperative pain. We then converted the extracted signals into a well-known frequency representation image known as the spectrogram image [25]. Previous works [14] reported that spectrogram images lead to better performance as they can suppress the noise in the audio signal. Figure 2 illustrates a test audio signal of a neonate in a pain state while Figure 3 shows the spectrogram representation of the same signal. The brighter pixels in the spectrogram image indicate higher energy, and the darker pixels indicate lower energy.

Fig. 2.

An audio signal during neonate’s pain state; time (seconds) in the x-axis, amplitude in the y-axis.

Fig. 3.

Spectrogram image of the corresponding audio (Fig. 2) signal; time (seconds) in the x-axis, frequency (kHz) in the y-axis.

As our dataset has a limited number of crying sound segments (218 segments), we enlarged the dataset as follows. We augmented each raw audio signal by altering the fundamental frequency f at three levels ($f/3,f/2,2f/3$), adding six various levels of noise (0.001, 0.003, 0.005, 0.01, 0.03, 0.05), and combining both noise and frequency; e.g., $f/3$ with noise 0.003 or $f/3$ with noise 0.005. The augmentation generates a total of 27 segments per audio signal; i.e., 3 for frequency variation, 6 for noise addition, and 18 (3 × 6) for the combination of frequency and noise. The augmented signals are then converted to spectrogram images. These images are used to fine-tune a pre-trained (ImageNet weights) VGG-16 architecture as described in [14].

B. Responsible Training Instances Identification

We used the influence function (Section II.A) to explain a given pain segment by identifying the most important training instances that impact the model’s performance. Algorithm 1 details the steps of using the influence function to identify the responsible training instances. To calculate the influence score ${ℐ}_{\mathit{up,loss}}\left(z,z_{\mathit{test}}\right)$ that each training instance has on the model’s prediction for the test instance $z_{\mathit{test}}$, we used Equation (3). This enables us to determine which training instance has the most positive or negative influence on that test instance; the most helpful training instance has the highest influence score, and the most harmful training instance has the lowest score.

To distinguish between the training instances that improve the model’s performance and those that degrade the performance, we further dropped various percentages of the harmful and helpful training instances. Finally, we applied a human-grounded evaluation [26] to verify whether the influence function produced the expected results; i.e., a trained graduate student evaluated the results, which were further verified by a senior researcher in our lab.

C. Cosine Similarity Calculation

In [15], the authors adopted the cosine similarity score to contrast the instance-based explanation. Here, we borrowed the same idea to observe the relationship between cosine similarity and influence function score. One of the first steps in computing cosine similarity is to convert the images into a dense vector representation using an image embedding technique. Image embedding maps an image to a high-dimensional feature space, where each vector dimension represents an image feature such as color, texture, and shape. The mapping to the high-dimensional feature space results in a dense feature vector, where each vector dimension has a non-zero value. There are several methods (e.g., CNN-based, autoencoder-based, hand-crafted feature-based, and transformer-based) for obtaining a dense vector representation.

In this work, we used a CNN-based method as the embeddings obtained through CNNs are rich in spatial and semantic information. Specifically, we selected the last layer of a VGG-16 model to reduce the image dimension while keeping the most important information. The final layer of VGG-16 provides a feature vector $\varphi \left(z\right)$ of 512 values. Once we had the feature vectors for the training ($\varphi \left(z_i\right)$) and test images $\left(\varphi \left(z_{\mathit{test}}\right)\right)$, we computed the cosine similarity of images as follows: ${\mathit{cos}}_{\mathit{sim}}=\mathit{cos}\left(\varphi \left(z_{\mathit{test}}\right),\varphi \left(z_i\right)\right)$. The similarity measurement is 1 or close to 1 when the test image is identical or very similar to the training image. Finally, we contrasted the obtained similarity score with the influence score to evaluate the outcome of our approach. If the test image is similar to a specific training image, the influence score of that training image should be high; i.e., the model’s performance will degrade significantly when that training image is removed.

IV. Experimental Results and Discussion

This section presents the influence-based explanation method’s results on a neonatal pain dataset. It also provides a comparison of the CNN performance before and after removing harmful training instances. To evaluate the influence-based method, we used the cosine similarity and human-grounded evaluation.

A. Dataset

We used a publicly available neonatal postoperative pain dataset known as USF-MNPAD-I [14]. This dataset is collected in a clinical setting and consists of neonates’ face, body, crying sound, and vital signs. The neonate’s age ranges from 27 to 41 gestational weeks [GW], and the neonates belong to a diverse racial population (36.36% Caucasian, 36.36 % White, 20 % Black, and 7.27 % Asian). The dataset includes 218 postoperative pain segments. The pain label or ground truth was documented by two experienced nurses who used the N-PASS scale [27] to classify the neonate’s sounds as either indicating pain or no pain. The nurses’ agreement was evaluated using two different metrics, the Pearson correlation coefficient (0.89) and the Kappa coefficient (0.85). More descriptions about the dataset can be found in [14].

B. Implementation Details

We selected leave-one-subject-out (LOSO) cross-validation to evaluate our model due to the variations between patients, the relatively small dataset, and the realistic nature of LOSO in clinical applications [28]. We resized the spectrogram images to 224 × 224 and fed them to the VGG-16 base model. We used the Adam optimizer with a learning rate of 0.0001 and a binary cross-entropy loss. A batch size of 16 was used, and the training process was carried out for a maximum of 100 epochs. To assess the performance of our trained model, we used the accuracy, precision, recall, and F-1 score weighted by both classes.

C. Influence and Similarity Scores Analysis

After obtaining the pain classification, we applied the proposed influence-based XAI method to understand the reasoning behind the model’s prediction. Table I shows the top-5 most helpful and harmful data while evaluating a neonatal pain segment. The training instance with the highest level of influence has a cosine similarity of 0.991445 for the test image. We observe congruence between the cosine similarity score and the influence score. In other words, the training instance having a cosine similarity value greater than 0.9 significantly impacts the model’s decision. On the contrary, the influence score and cosine similarity are low for harmful data. The visual representations of the two most helpful training instances are displayed in Figure 4 and Figure 5, with a cosine similarity score of 0.991445 and 0.990966, respectively. Figure 6 and Figure 7 show the two most harmful training images for the test image (Figure 3) with low cosine similarity scores of 0.110163 and 0.0.112365, respectively.

TABLE I.

Top-5 training instances with Influence Score and Cosine Similarity score

Training instance	Influence Score (1 × e–06)	Training Data IDs	Cosine Similarity Score
Top-5 helpful training instance	9.993024	3836	0.991445
	9.991485	2884	0.980269
	9.974717	5964	0.990966
	9.970116	3752	0.989771
	9.965725	5145	0.966328
Top-5 harmful training instance	0.101168	2029	0.113826
	0.100961	2463	0.112889
	0.100931	2085	0.113616
	0.100461	1245	0.112365
	0.100215	1371	0.110163

Note: Training instances (bold influence score) are visualized in Fig. (4–7)

Fig. 4.

The top most influential training instance for the test instance with cosine similarity 0.991445 between them.

Fig. 5.

The test instance’s second most influential training instance, with a cosine similarity of 0.990966 between them.

Fig. 6.

The top most harmful training instance for the test instance with cosine similarity 0.110163 between them.

Fig. 7.

The test instance’s second most harmful training instance, with a cosine similarity of 0.112365 between them.

We also re-trained our model with a modified dataset, in which we dropped some harmful and helpful images and reported the performance. Table II presents the average results of five subjects due to dropping various data combinations. From the table, we can observe that dropping approximately the top 1% of harmful data does not negatively impact the model’s accuracy. As a result, we can drop these harmful data and compress the dataset without affecting the model’s accuracy. However, the weighted precision, recall, and F1-score values show slight fluctuations when dropping approximately the top 2% harmful data. On the contrary, removing influential data reduces accuracy and suggests that such data benefits the model’s performance.

TABLE II.

Average result of five subjects

Data Amount	Accuracy	Precision	Recall	F1-score
Total training data	0.7963	0.7964	0.7963	0.7960
Dropping ≈ 1% harmful data	0.7963	0.7964	0.7963	0.7960
Dropping ≈ 2% harmful data	0.7745	0.7775	0.7745	0.7725
Dropping ≈ 1% helpful data	0.7471	0.7438	0.7471	0.7397
Dropping ≈ 2% helpful data	0.7026	0.6995	0.7026	0.6944

D. Human-grounded Evaluation

In this section, we verify the result of our XAI-based approach for identifying the most harmful and helpful pain segment through human evaluators, also known as the human-grounded evaluation [26]. In our case, a trained graduate student evaluated the explanation, and a senior researcher further verified it. A list of the top 20 most helpful and harmful training IDs for a test (pain) segment was created using the explanation methods. Then, each original audio segment of these training IDs was traced and played. As mentioned earlier, two expert nurses established the ground truth using the N-PASS scale to distinguish between painful and non-painful audio segments. The evaluators utilized the ground truth and played the audio file to evaluate the results with two questions. Question 1: ‘Did the helpful training IDs contain a neonate’s crying sound for testing a pain segment? Response (:yes/no)’ and Question 2: ‘Rate the similarity between the test image and the responsible train image identified by the XAI method in between [1= low, 2= moderate, 3 = high]’. We found that the evaluators answered ‘yes’ for all cases of Question 1, indicating that all the helpful training IDs contained neonatal crying sounds. In contrast, harmful IDs either lacked crying sounds or had irrelevant background noise. Moreover, we got an average of 2.96 similarity score for Question 2, i.e., the method’s identified rate of the explanatory training examples was accurate.

V. Conclusion and Future Works

This work presents an influence function-based XAI method applied to the audio signals obtained from neonates while experiencing postoperative pain in a clinical setting. Our experimental results demonstrated our method’s efficacy in identifying the most responsible training instances for pain prediction. We also distinguished the helpful and harmful training instances and examined their effects on the performance. We measured the effectiveness of the XAI results by calculating the cosine similarity scores and discovered that images with high similarity scores also had high influence scores. Further, we used a human-grounded evaluation to assess the results and identify any discrepancies in explanations.

In the future, we plan to perform an expert human-grounded evaluation [26] by involving an expert pediatrician in the evaluation process. Additionally, we plan to integrate the proposed method into other deep learning models (e.g., RNN, transformer networks) as well as evaluate the performance of influence-based explanation with a multimodal network.