Relevance Vector Machine Learning for Neonate Pain Intensity Assessment Using Digital Imaging

Abstract

Pain assessment in patients who are unable to verbally communicate is a challenging problem. The fundamental limitations in pain assessment in neonates stem from subjective assessment criteria, rather than quantifiable and measurable data. This often results in poor quality and inconsistent treatment of patient pain management. Recent advancements in pattern recognition techniques using relevance vector machine (RVM) learning techniques can assist medical staff in assessing pain by constantly monitoring the patient and providing the clinician with quantifiable data for pain management. The RVM classification technique is a Bayesian extension of the support vector machine (SVM) algorithm, which achieves comparable performance to SVM while providing posterior probabilities for class memberships and a sparser model. If classes represent “pure” facial expressions (i.e., extreme expressions that an observer can identify with a high degree of confidence), then the posterior probability of the membership of some intermediate facial expression to a class can provide an estimate of the intensity of such an expression. In this paper, we use the RVM classification technique to distinguish pain from nonpain in neonates as well as assess their pain intensity levels. We also correlate our results with the pain intensity assessed by expert and nonexpert human examiners.

I. Introduction

Pain assessment in patients who are unable to verbally communicate is a challenging problem in patient critical care. This problem is most prominently encountered in sedated patients in the intensive care unit (ICU) recovering from trauma and major surgery, as well as infant patients and patients with brain injuries [1]–[3]. Current practice in patient critical care requires the nursing staff in assessing the pain experienced by the patient, and taking appropriate action to ameliorate the patient's anxiety and discomfort.

Individuals in pain manifest their condition through “pain behavior” [4], [5], which includes facial expressions. The significance of a facial expression as an indicator of pain is discussed and advances in pain assessment using facial expressions are reviewed in [5]. Clinicians regard the patient's facial expression as a valid indicator for pain and pain intensity [6]. Hence, correct interpretation of the facial expressions of the patient and its correlation with pain is a fundamental step in designing an automated pain assessment management system. Of course, other pain behaviors, including head movement and the movement of other body parts, along with physiological indicators of pain, such as heart rate, blood pressure, and respiratory rate responses should also be included in such a system.

Depending on the patient group (e.g., neonates, children, adults, etc.) pain assessment criteria have been developed, and indicators of pain in each group might be different. For example, while the behavioral pain scale for adults focuses on facial expressions, upper limbs, and compliance with ventilation [7], the face, legs, activity, cry, and consolability (FLACC) [8] behavioral pain scale focuses on slightly different set of indicators for postoperative young children. Similarly, the premature infant pain profile (PIPP) [9] considers a special set of pain indicators, including physiological and behavioral indicators for pain assessment in premature infants.

Infants are unable to directly report their level of pain, and hence, medical staff are responsible for pain assessment for neonates. Pain and distress behaviors in neonates, include facial expression, cry, and body movement, and a series of methods have been suggested to objectively assess pain in neonates based on the aforementioned behaviors [5], [8], [9]. In this paper, we focus on the problem of pain assessment in infants using facial expressions.

Although there is a vast potential for using computer vision for agitation and pain assessment, there are very few articles in the computer vision literature addressing this issue. Bonroy et al. [10] have used computer vision for pain assessment in demented elderly patients. An agitation assessment scheme is proposed for patients in the ICU in [11]. The approach of [11] is based on the hypothesis that facial grimacing induced by pain results in additional “wrinkles” (equivalent to edges in the processed image) on the face of the patient, and this is the only factor they use in assessing pain. Although this approach is computationally inexpensive, and especially, appealing for a real-time decision support system, it can be limiting, since it does not account for other facial actions (e.g., smiling, crying, etc.), which may not necessarily correspond to pain. Brahnam et al. [12]–[15] use various face classification techniques, including support vector machines (SVM) and neural networks (NN) to classify facial expressions in neonates into “pain” and “nonpain” classes. Such classification techniques were shown to have reasonable accuracy.

In this paper, we extend the classification technique addressed in [12]–[15] to distinguish pain from nonpain in neonates as well as assess their pain intensity using a relevance vector machine (RVM) classification technique [16]. The RVM classification technique is a Bayesian extension of SVM, which achieves comparable performance to SVM while providing posterior probabilities for class memberships and a sparser model. In a Bayesian interpretation of probability, as opposed to the classical interpretation, the probability of an event is an indication of the uncertainty associated with the event rather than its frequency [17]. If data classes represent “pure” facial expressions, that is, extreme expressions, which an observer can identify with a high degree of confidence, the posterior probability of the membership of some intermediate facial expression to a class can provide an estimate of the intensity of such an expression. This, along with other pain behaviors, can be translated into one of the scoring systems currently being used for assessing pain (e.g., FLACC or PIPP).

The contents of the paper are as follows. In Sections II and III, we review the SVM and RVM classification techniques for pain recognition using facial expressions. Then, in Section IV, we present the results of these classification techniques applied to the infant classification of pain expression (COPE) database [12]. The pain intensity assessment given by the computer classifier shows a strong correlation with the pain intensity assessed by expert and nonexpert human examiners. Finally, we draw conclusions and point to some future research directions in Section V, including opportunities for sedation and agitation assessment using digital imaging in the ICU.

II. Support Vector Machines

As we see in Section IV, the problem of pain and pain intensity assessment using facial images involves a standard problem in machine learning called data classification [17]. Given a series of input variables x₁, x₂, …, x_N in ℝ^D and their corresponding class labels ₁, ₂, …, _p, where p ≤ N, the data classification problem involves assigning the correct class label to a new input variable x. Kernel-based methods are typically used for data classification and regression [17]. A key limitation of many kernel-based learning algorithms is the computational intensity involved in the training, prediction, and decision-making stages of the algorithm. This is due to the fact that the kernel function, which adds a dimension to the data in order to obtain an optimal classification, has to be computed for all pairs of data points. In sparse kernel machine algorithms, however, only a subset of the training data is used, providing a sparse solution. Sparse kernel machines are faster in the training, and the prediction and decision-making stages. In this paper, we consider two sparse kernel-based classification algorithms, namely, SVMs and RVMs.

SVMs [18] involve sparse kernel algorithms used in classification and regression problems, and have their origin in statistical learning theory. Here, we consider the classification problem involving two data classes, namely, ₁ and ₂. The framework can be generalized to a multiclass label problem using a similar approach as outlined later [17]. Let the training set be given by {x₁, x₂, …, x_N}, with target values given by z₁, z₂, …, z_N, respectively, where x_n ∈ ℝ^D and z_n ∈ {−1, 1}, n = 1, …, N, and with x_n ∈ ₁ if z_n = −1, and x_n ∈ ₂ if z_n = 1. To classify a new data point x ∈ ℝ^D, define the classifier function

$$y(x)\triangleq w^{\text{T}}\varphi (x)+b$$ (1)

where ϕ : ℝ^D → ℝ^M is a continuous fixed feature-space transformation, w ∈ ℝ^M is a weight vector, and b ∈ ℝ is a bias parameter. The sign of the classifier function y(x) determines the class of x. More specifically, for a new input variable x, the target value is given by z = sgn(y(x)), where sgn y ≜ y/|y|, y ≠ 0 and sgn(0) ≜ 0.

Next, assume that the training set is linearly separable in the feature space ℝ^M, that is, there exist a weight vector w ∈ ℝ^M and a bias parameter b ∈ ℝ, such that y(x_n) > 0 for x_n ∈ ℝ^D and z_n = 1, and y(x_n) < 0 for x_n ∈ ℝ^D and z_n = −1, or equivalently, z_ny(x_n) > 0 for all x_n ∈ ℝ^D and z_n ∈ {−1, 1}. Later, we will relax the linear separability assumption and consider the more general case of overlapping classes.

Note that the classifier function y(·) separates the feature space ℝ^M into two disjoint regions characterized by y(x) > 0 and y(x) < 0 for x ∈ ℝ^D. The affine hyperplane separating the two disjoint regions, namely y(x) = 0, is called the decision boundary and is denoted by . Note that ϕ(·) can be a nonlinear transformation, which would correspond to a nonlinear decision boundary in the original input space ℝ^D. The minimum distance between the training set and the decision boundary is called the margin. The distance of a point x_n ∈ ℝ^D to the decision boundary is given by

$$\text{dist}(\varphi (x_n),\mathcal{D})=\frac{|y(x_n)|}{\Vert w\Vert }=\frac{z_{n}y(x_n)}{\Vert w\Vert }$$ (2)

where ‖ · ‖ denotes the Euclidean norm on ℝ^M and dist(x, ) ≜ inf_s∈D ‖x − s‖. Hence, the margin is given by

$$\underset{n\in \left\{1,\dots ,N\right\}}{min}\text{dist}(\varphi (x_n),\mathcal{D})=\underset{n\in \left\{1,\dots ,N\right\}}{min}\frac{z_{n}y(x_n)}{\Vert w\Vert }.$$ (3)

As in all classification methods, the goal of the SVM algorithm is to classify a new input variable x ∈ ℝ^D based on the information provided by the training set and the target values. The SVM framework addresses this problem by choosing the decision boundary in such a way so that the margin is maximized. The following problem presents the SVM algorithm as an optimization problem.

Maximum Margin Classification Problem

Consider the training set given by {x₁, x₂, …, x_N} ⊂ ℝ^D and let the classifier function y : ℝ^D → ℝ be given by (1). Find the weight vector w ∈ ℝ^M and the bias parameter b ∈ ℝ such that (3) is maximized.

Theorem 2.1

w* ∈ ℝ^M and b* ∈ ℝ solve the maximum margin classification problem if and only if w* and b* are the solutions to the optimization problem

$$\underset{w\in {ℝ}^M,b\in ℝ}{max}\left\{\frac{1}{\Vert w\Vert }\underset{n\in \left\{1,\dots ,N\right\}}{min}\left[z_n(w^{\text{T}}\varphi (x_n)+b)\right]\right\}.$$ (4)

Proof

The proof is a direct consequence of the definition of a margin given by (3).

The solution to the nonconvex optimization problem (4) is not unique. To see this, note that scaling the weight vector w ∈ ℝ^M and the bias parameter b ∈ ℝ by a positive scalar does not change the value of the function to be maximized. The following theorem presents an alternative characterization to the maximum margin classification problem.

Theorem 2.2

w* ∈ ℝ^M and b* ∈ ℝ solve the maximum margin classification problem if and only if w* and b* are the solutions to the optimization problem

$$\underset{w\in {ℝ}^M,b\in ℝ}{min}\frac{1}{2}{\Vert w\Vert }^2$$ (5)

subject to

$$z_n(w^{\text{T}}\varphi (x_n)+b)\ge 1$$ (6)

where x_n ∈ ℝ^D, z_n ∈ {−1, 1}, and n = 1, …, N.

Proof

Since rescaling the weight vector w ∈ ℝ^M and the bias parameter b ∈ ℝ in (4) by a positive scalar does not change the value of the function to be maximized, the optimization problem (4) has a continuum of solutions corresponding to the same optimal value. Hence, introducing the new constraint

$$z_n\ast (w^{\text{T}}\varphi (x_n\ast )+b)=1$$ (7)

where n* = arg min_{n∈{1, …, N}} dist(ϕ(x_n), ) does not change the optimal value of the optimization problem (4). Thus, the inequality constraint (6) holds for all x_n ∈ ℝ^D, z_n ∈ {−1, 1}, and n = 1, …, N. The proof now follows by noting that the optimization problem (4) subject to (7) is equivalent to the optimization problem (5) subject to (6).

The constrained optimization problem given by (5) and (6) is convex and can be solved using Lagrange multiplier methods. Specifically, introducing the Lagrange multipliers λ_n ∈ ℝ, n = 1, …, N, and forming the Lagrangian

$$ℒ(w,b,\lambda )=\frac{1}{2}{\Vert w\Vert }^2-\sum _{n=1}^N{\lambda }_n\left[z_n(w^{\text{T}}\varphi (x_n)+b)-1\right]$$ (8)

where λ = [λ₁, λ₂, …, λ_n]^T, it follows from the first-order necessary conditions for optimality that

$$w=\sum _{n=1}^N{\lambda }_n{z}_n\varphi (x_n)$$ (9)

$$0=\sum _{n=1}^N{\lambda }_n{z}_n.$$ (10)

Note that (9) and (10) can be used to eliminate w and b from the Lagrangian (8) leading to a dual representation of the optimization problem (4). Namely,

$$\underset{\lambda \in {ℝ}^N}{max}\stackrel{\sim }{ℒ}(\lambda )$$ (11)

subject to

$${\lambda }_n\ge 0,n=1,\dots ,N$$ (12)

$$\sum _{n=1}^N{\lambda }_n{z}_n=0$$ (13)

where

$$\stackrel{\sim }{ℒ}(\lambda )\triangleq \sum _{n=1}^N{\lambda }_n-\frac{1}{2}\sum _{n=1}^N\sum _{m=1}^N{\lambda }_n{\lambda }_m{z}_n{z}_{m}k\left(x_n,x_m\right)$$ (14)

and

$$k(x,x\prime )={\varphi }^{\text{T}}(x)\varphi (x\prime )$$ (15)

is the kernel function. Here, we introduced an alternative formulation of the optimization problem (4) in terms of the kernel function k : ℝ^D × ℝ^D → ℝ, which allows us to avoid working explicitly in the feature space. Note that the classifier function (1) can be rewritten using the kernel function as follows:

$$y(x)=\sum _{n=1}^N{\lambda }_n{z}_{n}k(x,x_n)+b.$$ (16)

The Kuhn–Tucker (KT) necessary conditions for optimality for the constrained optimization problem (11)–(13) are given by

$${\lambda }_n\ge 0$$ (17)

$$z_{n}y(x_n)-1\ge 0$$ (18)

$${\lambda }_n(z_{n}y(x_n)-1)=0$$ (19)

where n = 1, …, N. Now, it follows from (19) that either λ_n = 0 or z_ny(x_n) = 1. The input variables x_n ∈ ℝ^D, n = 1, …, N, for which the corresponding Lagrange multiplier λ_n ∈ ℝ vanishes, do not contribute to the classifier function (16), and hence, can be omitted. The remaining input variables are called support vectors, and by definition, lie on the maximum margin affine hyperplanes w*^T ϕ(x_n) + b* = ±1, n = 1, …, N. Hence, only the support vectors play a role in the classification of the new input variables and the rest of the training set can be discarded.

Next, we consider the case of overlapping classes. For this case, the SVM algorithm considered earlier identifies the decision boundary so that the training set is separated into two data classes with no input variables being misclassified. This results in poor class assignments for new input variables. The SVM algorithm, however, can be modified by allowing input variables in the training set to lie on the “wrong side” of the margin boundary and penalizing such constraint violations. Specifically, for every input variable x_n ∈ ℝ^D, n = 1, …, N, define the slack variable ξ_n ≥ 0 such that ξ_n = 0 if (6) is satisfied, that is, for n ∈ {1, …, N}, x_n is on or inside the correct margin boundary, and ξ_n = |z_n − y(x_n)| otherwise.

The modified SVM algorithm is given by the following optimization problem:

$$\underset{w\in {ℝ}^M,b\in ℝ,\xi \in {ℝ}^N}{min}C\sum _{n=1}^N{\xi }_n+\frac{1}{2}{\Vert w\Vert }^2$$ (20)

subject to

$$z_{n}y(x_n)\ge 1-{\xi }_n,n=1,\dots ,N$$ (21)

$${\xi }_n\ge 0,n=1,\dots ,N$$ (22)

where ξ = [ξ₁, ξ₂, …, ξ_N]^T and > 0 is a complexity parameter controlling the tradeoff between the margin and the slack variable penalty. It can be shown that if ξ_n = 0, then (21) reduces to (6) and the corresponding input variable x_n ∈ ℝ^D will be correctly classified. Moreover, if 0 < ξ_n ≤ 1, then the input variable x_n ∈ ℝ^D is correctly classified while lying inside the margin boundary, whereas if ξ_n > 1, then the input variable is misclassified.

Lagrange multiplier methods can be used to solve the optimization problem (20)–(22) by introducing the Lagrange multipliers λ_n ∈ ℝ and μ_n ∈ ℝ, n = 1, …, N, corresponding to the constraints (21) and (22), respectively. In this case, the Lagrangian is given by

$$ℒ(w,b,\lambda ,\xi )=\frac{1}{2}{\Vert w\Vert }^2+C\sum _{n=1}^N{\xi }_n-\sum _{n=1}^N{\lambda }_n(z_{n}y(x_n)-1+{\xi }_n)-\sum _{n=1}^N{\mu }_n{\xi }_n$$ (23)

where λ = [λ₁, λ₂, …, λ_N]^T. Now, it follows from the first-order necessary conditions for optimality that

$$w=\sum _{n=1}^N{\lambda }_n{z}_n\varphi (x_n)$$ (24)

$$0=\sum _{n=1}^N{\lambda }_n{z}_n$$ (25)

$${\lambda }_n=C-{\mu }_n,n=1,\dots ,N$$ (26)

and the KT necessary conditions give

$${\lambda }_n\ge 0$$ (27)

$$z_{n}y(x_n)-1+{\xi }_n\ge 0$$ (28)

$$z_n(z_{n}y(x_n)-1+{\xi }_n)=0$$ (29)

$${\mu }_n\ge 0$$ (30)

$${\xi }_n\ge 0$$ (31)

$${\mu }_n{\xi }_n=0$$ (32)

where n = 1, …, N. The dual representation of the optimization problem (20) subject to (21) and (22) is given by

$$\underset{\lambda \in {ℝ}^N}{min}\stackrel{\sim }{ℒ}(\lambda )$$ (33)

subject to

$$0\le {\lambda }_n\le C,n=1,\dots ,N$$ (34)

$$\sum _{n=1}^N{\lambda }_n{z}_n=0$$ (35)

where ˜ (λ) is given by (14), the kernel function is given by (15), and where we have used (24)–(32).

III. Sparse Bayesian Learning

The SVM framework is a powerful classifier, but has a number of limitations. A key deficiency of the approach is the fact that the output of the SVM is the binary classification decision and not the class membership posterior probability. As will be discussed in Section IV, methods which possess an inherent Bayesian structure are more powerful and can provide more information. Such methods not only classify a new input variable, but can also provide a degree of uncertainty (in terms of posterior probabilities) for such a classification. The RVM [16], which is a special case of the sparse Bayesian learning algorithm, can be regarded as the Bayesian extension of the SVM approach.

In this section, we consider a classification problem involving two data classes, namely ₁ and ₂, using the sparse Bayesian learning approach. The framework can be generalized to a multiclass classification problem using a similar approach as outlined later [16]. Consider the Laplace approximation method [17] involving the random variable υ ∈ ℝ^M with associated probability density function given by p : ℝ^M → ℝ. Assume that p(υ) = f(υ)/V, where f : ℝ^M → ℝ is a function defined on υ ∈ ℝ^M and V = ∫_ℝ^M f(υ)dυ is the normalization coefficient. The probability density function p(υ) is approximated by a multivariate Gaussian (normal) distribution (υ; υ₀, Σ) with mean υ₀ ∈ ℝ^M and covariance matrix Σ ∈ ℝ^M×M, where υ₀ = arg max_υ∈ℝ^M p(υ) and Σ= −∂²/∂υ² ln f(υ)∣_υ=0. The normalization coefficient V can be approximated by [17]

$$V\simeq f({\upsilon }_0)\frac{{(2\pi )}^{M/2}}{{(det\sum )}^{1/2}}$$ (36)

where det(·) denotes the determinant operator.

Next, let the training set be given by {x₁, x₂, …, x_N} ⊂ ℝ^D, with target values given by z₁, z₂, …, z_N, respectively, where x_n ∈ ℝ^D and z_n ∈ {0, 1}, n = 1, …, N, and with x_n ∈ ₁ if z_n = 1, and x_n ∈ ₂ if z_n = 0. For a new input variable x ∈ ℝ^D, we predict the associated class membership posterior probability distribution p( _k∣x, X, Z), k = 1, 2, where p( _k∣x, X, Z) is the class membership conditional probability of the data class _k given x ∈ ℝ^D, X = [x₁, x₂, …, x_N], and Z = [z₁, z₂, …, z_N]^T. Note that, in contrast to the SVM approach, the sparse Bayesian learning method separates the prediction stage (i.e., finding the posterior class membership probabilities for the new input variable x) from the decision-making stage (i.e., assigning the new input variable x to the appropriate class). This separation is particularly useful when dealing with asymmetric classification costs, where misclassification of input variables belonging to a certain class is more costly [16]. For example, for the problem involving the classification of facial images of patients to pain and nonpain classes discussed in Section IV, the cost of misclassification of a patient in pain to the nonpain class (false negative) is higher than that of a patient with no pain to the pain class (false positive). One of the key advantages of the sparse Bayesian learning approach is its ability to deal with such asymmetric costs.

Define the classifier function

$$y(x)\triangleq w^{\text{T}}\varphi (x)$$ (37)

where ϕ: ℝ^D → ℝ^M is a continuous feature-space transformation and w = [w₁, w₂, …, w_M]^T ∈ ℝ^M is a weight vector. Note that the RVM algorithm is a special case of the sparse Bayesian learning method. Specifically, in the RVM, w^Tϕ(x) in (37) has the special form (similar to the SVM algorithm) given by ${\sum }_{n=1}^N{w}_{n}k(x,x_n)+b$, where k(·, ·) is the kernel function. In the sequel, we consider the general formulation (37).

Following standard statistical practice, we assume that the posterior probability of the target value of an input variable corresponding to the class ₁ is given by p(z_n = 1∣x_n, w) = σ(y(x_n)), n = 1, …, N, where σ(·) is the logistic sigmoidal function defined by σ(s) ≜ 1/(1 + e^−s) [16], [17]. Note that, since there are only two classes, p(z_n = 0∣x_n, w) = 1 − σ(y(x_n)). Assuming that the input variables x_n, n = 1, …, N, are independent, the likelihood function is given by

$$\begin{array}{ll}p(Z\mid X,w)\hfill & =\prod _{n=1}^{N}p(z\mid x_n,w)\hfill \\ \hfill & =\prod _{n=1}^N\sigma {(y(x_n))}^{z_n}{(1-\sigma (y(x_n)))}^{1-z_n}.\hfill \end{array}$$ (38)

Each weight parameter w_n, n = 1, …, M, in (37) is assumed to have a zero-mean Gaussian distribution, and hence, the weight prior distribution is given by

$$p(w\mid \alpha )=\prod _{n=1}^M\mathcal{N}(w_n;0,{\alpha }_n^{-1})$$ (39)

where α_n, n = 1, …, M, is the precision (inverse of the variance of the Gaussian distribution) corresponding to w_n and α = [α₁, α₂, …, α_M]^T ∈ ℝ^M. The parameters α_n, n = 1, …, M, in the prior distribution (39) are called the hyperparameters. Note that, in contrast to other Bayesian classifiers, each weight parameter w_n, n = 1, …, M, has a separate hyperparameter α_n.

Given a new input variable x ∈ ℝ^D, the corresponding target value z ∈ {0, 1} can be predicted using the predictive distribution p(z∣x, X, Z). The predictive distribution is given by

$$p(z\mid x,X,Z)={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }p(z\mid x,X,Z,w,\alpha )\times p(w\mid x,X,Z,\alpha )p(\alpha \mid x,X,Z)\text{d}w\text{d}\alpha$$ (40)

where the distribution is marginalized with respect to the weight vector w ∈ ℝ^M and the hyperparameters α ∈ ℝ^M. Since σ(·) is nonlinear, no closed-form solution exists for (40) [16]. Here, we use the type-2 maximum likelihood [19]—also known as the evidence approximation [20]—to approximate (40) by replacing α ∈ ℝ^M with a constant value α* ∈ ℝ^M corresponding to the mode (i.e., the maximizer) of the marginal likelihood function p(Z∣X, α). In particular, an approximation to the predictive distribution p(z∣x, X, Z) is given by

$$p(z\mid x,X,Z)\simeq p(z\mid x,X,Z,\alpha \ast )={\int }_{-\infty }^{\infty }p(z\mid x,X,Z,w,\alpha \ast )p(w\mid x,X,Z,\alpha \ast )\text{d}w.$$ (41)

The value of α* is found via an iterative process. After initializing α, the posterior distribution p(w∣x, X, Z, α) is approximated by a Gaussian distribution using the Laplace approximation method. The mean of the Gaussian distribution corresponds to the mode (maximizer) of p(w∣x, X, Z, α), which we denote by w*. The maximizer is found using the iterative reweighted least squares (IRLS) method [17], which uses sequential quadratic approximations to find the maximizer. Taking the log of the identity [17]

$$\begin{array}{ll}p(w\mid x,X,Z,\alpha )\hfill & =\frac{p(Z\mid x,X,w,\alpha )p(w\mid x,X,\alpha )}{p(Z\mid x,X,\alpha )}\hfill \\ \hfill & =\frac{p(Z\mid X,w)p(w\mid \alpha )}{p(Z\mid X,\alpha )}\hfill \end{array}$$

the maximization problem is equivalent to

$$\underset{w\in {ℝ}^M}{max}\left\{ln(p(Z\mid X,w)p(w\mid \alpha ))-lnp(Z\mid X,\alpha )\right\}$$ (42)

or equivalently

$$\underset{w\in {ℝ}^M}{max}\left\{\sum _{n=1}^N(z_{n}ln{y}_n+(1-z_n)ln(1-y_n))-\frac{1}{2}{w}^{\text{T}}Aw+c\right\}$$ (43)

where y_n = σ(y(x_n)) ∈ ℝ, A = diag[α] ∈ ℝ^M×M, c ∈ ℝ is a variable independent of z (and hence, plays no role in the optimization), and where we have used (38). Note that the covariance matrix of the Gaussian approximation to the posterior distribution p(w∣x, X, Z, α) is equal to the negative Hessian of ln p(w∣x, X, Z, α) evaluated at the maximizer w*. The mean and covariance of the Gaussian approximation are given by

$$w\ast =A^{-1}{\mathrm{\Phi }}^{\text{T}}(Z-Y)$$ (44)

$$\sum ={({\mathrm{\Phi }}^{\text{T}}B\mathrm{\Phi }+A)}^{-1}$$ (45)

where Φ = [Φ_(i,j)] ∈ ℝ^{N × M} with Φ_(i,j) = ϕ_j(x_i) for i = 1, …, N and j = 1, …, M, B = diag[b₁, b₂, …, b_N] ∈ ℝ^{N × N} with b_n = y_n(1 − y_n) ∈ ℝ, n = 1, …, N, and Y = [y₁, y₂, …, y_n]^T ∈ ℝ^N.

Next, using (36), we approximate the marginal likelihood function as follows:

$$\begin{array}{cc}p(Z\mid X,\alpha )\hfill & ={\int }_{-\infty }^{\infty }p(Z\mid X,w,\alpha )p(w\mid X,\alpha )\text{d}w\hfill \\ \hfill & ={\int }_{-\infty }^{\infty }p(Z\mid X,w)p(w\mid \alpha )\text{d}w\hfill \\ \hfill & \simeq p(Z\mid X,w\ast )p(w\ast \mid \alpha ){(2\pi )}^{N/2}{(det\sum )}^{1/2}.\hfill \end{array}$$ (46)

Following the discussion on the type-2 maximum likelihood method, the value of α* is found by maximizing the approximate marginal likelihood function given by (46). Hence, differentiating (46) with respect to α_n, n = 1, …, M, and setting the result to zero yields

$$-\frac{1}{2}{(w_n^{\ast })}^2+\frac{1}{2{\alpha }_n}-\frac{1}{2}{\sum }_{(n,n)}=0,n=1,\dots ,M.$$ (47)

Solving (47) for α_n, n = 1, …, M, gives the updated estimate for α_n as follows:

$${\alpha }_n^{\text{new}}=\frac{{\gamma }_n}{{(w_n^{\ast })}^2},n=1,\dots ,M$$ (48)

where

$${\gamma }_n=1-{\alpha }_n{\sum }_{(n,n)}$$

n = 1, …, M. Now, using the updated estimate ${\alpha }^{\text{new}}={[{\alpha }_1^{\text{new}},{\alpha }_2^{\text{new}},\dots ,{\alpha }_M^{\text{new}}]}^{\text{T}}\in {ℝ}^M$ for α, the above steps are repeated until a given convergence criterion is met. The algorithm is summarized in Table I.

TABLE I. Sparse Bayesian Learning Algorithm.

Step 1.	Initial parameter computation.
a.	Set Φ(i,j) = ϕj(xi), i = 1, …, N, j = l, …, M.
b.	Set Φ = [Φ(i,j)] ∈ ℝN × M.
c.	Set bn = yn(1 − yn), n = 1, …, N.
d.	Set B = diag[b1, b2, …, bN].
Step 2.	Initialize the hyperparameters α = [α1, α2, …, αM]T.
Step 3.	Find the Gaussian approximation (w; w*, Σ) to the posterior distribution p(w∣x, X, Z, α).
a.	Set A = diag[α].
b.	Set w* = A−1 ΦT(Z − Y).
c.	Set Σ = (ΦT BΦ + A)−1.
Step 4.	Compute the approximate marginal likelihood function using $p(Z\mid X,\alpha )\simeq p(Z\mid X,w\ast )p(w\ast \mid \alpha ){(2\pi )}^{N/2}{(det\sum )}^{\frac{1}{2}}$.
Step 5.	Set γn = 1 − αn Σnn, n = 1, …, M.
Step 6.	Update α using ${\alpha }_n\leftarrow \frac{y_n}{{(w_n^{\ast })}^2},n=1,\dots ,M$.
Step 7.	If ‖Δα‖ > Tol1 or ‖Δw*‖ > Tol2, where Δα and Δw* are the changes in the values of α and w* in the current iteration, respectively, and Tol1 and Tol2 are some prespecified tolerances, then go to Step 3.
Step 8.	Set α* = α.

As a result of the maximization of the marginal likelihood function, a number of hyperparameters α_n approach infinity, and hence, the corresponding weight parameter w_n will be centered at zero with zero variance. Therefore, the corresponding component of the feature-space transformation ϕ_n(·) plays no role in the prediction, resulting in a sparse predictive model. In the case of the RVM, the input variables x_n belonging to the training set {x₁, x₂, …, x_N}, which have a nonzero weight w_n, are called relevance vectors. Only the relevance vectors play a role in the classification of new input variables and the rest of the training set can be discarded.

Finally, we note that the posterior probability for the membership of a new input variable x to the class ₁ represented by p( ₁ ∣x, X, Z) can be approximated by the logistic sigmoidal function σ(y(x)) using the calculated value of the weight vector w*. This approximatation becomes exact as the number of input variables in the training set approaches infinity [16], [21].

IV. Pain and Pain Intensity Assessment in Neonates

In this section, we use the classification techniques described in Sections II and III, in order to assess pain and pain intensity in infants using their facial expressions. For our dataset, we use the infant COPE database [12]. As was shown in [12], the SVM can classify facial images into two groups of “pain” and “nonpain” with an accuracy between 82% and 88%. Here, we extend the results of [12] to additionally assess pain intensity using the class membership posterior probability. Note that although we consider infants, studies have shown that the pain-induced facial expressions in newborns are similar to those observed in older children and adults [22]. However, neonatal facial expressions are characterized by some unique features that are not found in adults, such as “primal face of pain” [23]. In addition, adults can control nonverbal expressions of pain [24].

Before applying the classification techniques to the facial images, we give a brief description of the infant COPE database used in our experimental results.

A. Infant COPE Database

The infant COPE database is composed of 204 RGB color photographs of 26 Caucasian neonates (13 boys and 13 girls) with a resolution of 120 × 100 per photograph and an infant age range of 18 h to 3 days. The photographs were taken after a series of stress-inducing stimuli were administered by a nurse. The stimuli consist of the following [12]:

1. transport from one crib to another;

2. air stimulus, where the infant's nose was exposed to a puff of air;

3. friction, where the external lateral surface of the heel was rubbed with a cotton wool soaked in alcohol;

4. pain, where the external surface of the heel was punctured for blood collection.

The facial expressions induced by the first three stimuli are classified as nonpain. Four photographs of a typical subject are given in Fig. 1. One of the challenges in the recognition of pain, even for clinicians, is the ability to distinguish an infant's cry induced by pain and some other nonpainful stimulus.

Fig. 1.

Four different expressions of a subject. The two left images correspond to nonpain, whereas the two right images correspond to pain.

B. Pain Recognition Using Sparse Kernel Machine Algorithms

The classification techniques discussed in Section II were used to identify the facial expressions corresponding to pain. A total of 21 subjects from the infant COPE database were selected such that for each subject at least one photograph corresponded to pain and one to nonpain. The total number of photographs available for each subject ranged from 5 to 12, with a total of 181 photographs considered. We applied the leave-one-out method for validation [17]. In particular, the classifier is trained on all photographs of the subject except for one test photograph, which is used to validate the algorithm. The test photograph corresponds to either a pain or nonpain condition.

In the preprocessing stage, the faces were standardized for their eye position using a similarity transformation. Then, a 70 × 93 window was used to crop the facial region of the image and only the 8-bit grayscale values were used. For each image, a 6510-dimensional vector was formed by column stacking the matrix of intensity values.

We used the MATLAB version R2008a and the OSU SVM MATLAB Toolbox [25] to run the SVM classification algorithm. The classification accuracy for the SVM algorithm with a linear kernel was 90%, where as suggested in [12], we chose the complexity parameter C = 1. The number of support vectors averaged five. Applying the RVM algorithm with a linear kernel to the same dataset resulted in an almost identical classification accuracy, namely, 91%, whereas the number of relevance vectors was reduced to two. However, in 5 out of the 21 subjects considered, the RVM algorithm did not converge. This is due to the fact that in contrast to the SVM algorithm, the RVM algorithm involves a nonconvex optimization problem [17].

C. Pain Intensity Assessment

In addition to classification, the RVM algorithm provides the posterior probability of the membership of a test image to a class. As discussed in Section I, using a Bayesian interpretation of probability, the probability of an event can be interpreted as the degree of the uncertainty associated with such an event. This uncertainty can be used to estimate pain intensity. In particular, if a classifier is trained with a series of facial images corresponding to pain and nonpain, then there is some uncertainty for associating the facial image of a person experiencing moderate pain to the pain class. The efficacy of such an interpretation of the posterior probability was validated by comparing the algorithm's pain assessment with that assessed by several experts (intensivists) and nonexperts.

In order to compare the pain intensity assessment given by the RVM algorithm with human assessment, we compared the subjective measurement of the pain intensity assessed by expert and nonexpert examiners with the uncertainty in the pain class membership (posterior probability) given by the RVM algorithm. Actual pain modes for each infant were used to train the RVM classifier. We chose all 16 infants (out of the total 21) from the COPE database for which the RVM algorithm converged, and for each subject two photographs of the face corresponding to the nonpain and pain conditions were selected. In the selection process, two training photographs were selected, where the infant's facial expression truly reflected the pain intensity condition—calm for nonpain and distressed for pain—and a score of 0 and 100, respectively, was assigned to these photographs to give the human examiner a fair prior knowledge for the assessment of the pain intensity.

Ten data examiners were asked to provide a score ranging from 0 to 100 for each test photograph (i.e., nontraining photograph) of the same subject, using a multiple of ten for the scores. Five examiners with no medical expertise and five examiners with medical expertise in critical care and pain management were selected for this assessment. The medical experts were members of the clinical staff at the ICU of the Northeast Georgia Medical Center, Gainesville, GA, consisting of one medical doctor, one nurse practitioner, and three nurses. The medical doctor has ten years experience as an anesthesiologist and intensivist in pediatric hospitals. The nurse practitioner and nurses have 6 months to 1 year pediatric floor experience in large community hospitals and are also mothers. They were asked to assess the pain for a series of random photographs of the same subject, with the criterion that a score above 50 corresponds to pain, and with an increasing score corresponding to a higher pain intensity. Analogously, a score below 50 corresponds to nonpain, and with a decreasing score corresponding to a lower level of discomfort. The posterior probability given by the RVM algorithm with a linear kernel for each corresponding test photograph was rounded off to the nearest multiple of ten.

The pain scores for five infant subjects are given in Figs. 2–6, where the average score of the expert and nonexpert human examiners are compared to the score given by the RVM algorithm. In order to measure the agreement between the human examiners and the RVM algorithm, we need to quantify the agreement between two raters classifying an observation into different classes. The kappa coefficient [26] is used to measure the agreement between two raters classifying the same observation into two classes. A kappa coefficient of 0 represents chance agreement and a coefficient of 1 represents a perfect agreement between the two raters. The weighted kappa coefficient is an extension of the kappa coefficient to the case, where there are more than two classes and the classes are ordered [27]. In this case, a smaller difference between the chosen classes by the two raters indicates less disagreement between them. The pain intensity assessment can be regarded as a classification process in which a facial expression of a subject is classified into ten ordered classes, where class 1 corresponds to a pain intensity score of 0–9, class 2 corresponds to a pain intensity score of 10–19, etc. A qualitative evaluation of the observed kappa values is given in Table II [28].

Fig. 2.

Pain score for subject 1.

Fig. 6.

Pain score for subject 5.

TABLE II. Qualitative Evaluation of Observed κ-Values [29].

κ	Strength of agreement
0.00	Poor
0.00–0.20	Slight
0.21–0.40	Fair
0.41–0.60	Moderate
0.61–0.80	Substantial
0.81–1.00	Almost perfect

We used the weighted kappa coefficient to measure the agreement in the pain intensity assessment between the human examiners and the RVM algorithm. This coefficient is 0.47 for human expert examiners (with a 95% confidence interval of 0.37 to 0.57) and 0.46 for nonexpert examiners (with a 95% confidence interval of 0.36 to 0.55) as compared with the RVM for the 16 subjects considered in the study. This shows a moderate agreement between the human expert examiners and human nonexpert examiners as compared with the RVM algorithm based on the qualitative evaluation of the observed kappa values given by Table II. It is interesting to note that the weighted kappa coefficient measuring the agreement between human experts and human nonexperts is 0.78 with a 95% confidence interval of 0.73 to 0.82, which indicates a substantial agreement based on Table II. It is important to note, however, that proxy ratings of pain is a highly subjective process [30].

The results show an almost identical classification accuracy for a binary classification (with a score above 50 corresponding to pain). In particular, the nonexpert human examiner, the expert human examiner, and the RVM classification accuracy is given by 79%, 87%, and 91%, respectively. Moreover, the results show that the expert human examiners tend to be more accurate in the binary classification compared to the human nonexperts.

It is worth noting that Fig. 4 shows a poor correlation between the scores given by the RVM algorithm and the data examiners in the first three photographs. The data examiners assessed a high level of pain for subject 3, whereas the subject was not in pain. This highlights the challenge in distinguishing between pain from discomfort, even for human experts. In this case, the RVM algorithm correctly assessed that the infant has some level of discomfort, but is not in pain.

Fig. 4.

Pain score for subject 3.

Finally, in a repeatability study, the same human expert and nonexpert examiners were asked to assess the intensity of pain for the five subjects considered in Figs. 2–6, after a period ranging from 2 weeks to 4 months. Again, we used the weighted kappa coefficient to measure the agreement between two observations by the same rater. The weighted kappa coefficient in this case can be regarded as a measure of the ability of the human examiner to reproduce his or her own pain scores. The weighted kappa coefficient is 0.79 (with a 95% confidence interval of 0.74 to 0.84) for the human expert examiners and 0.73 (with a 95% confidence interval of 0.68 to 0.78) for the human nonexpert examiners. Based on this analysis, the human expert examiners tend to be slightly more reliable in assessing the pain intensity for the same subjects under the same pain conditions.

V. Conclusion and Opportunities for Future Research

In this paper, the problems of pain and pain intensity assessment using facial expressions in neonates were addressed. Sparse kernel machine algorithms were used to classify the images into pain and nonpain classes. The class membership posterior probability given by the RVM algorithm was interpreted as an estimate of the pain intensity, and this hypothesis was validated by comparing the results with expert and nonexpert human assessments of pain. The results provided by the RVM algorithm can potentially be useful in decision support systems for ICU analgesia, where a reliable objective pain assessment measure is required.

Machine learning techniques, and in particular the RVM algorithm, can potentially be useful in assessing sedation and agitation in the ICU. The fundamental limitations in sedation and agitation assessment in the ICU stem from subjective assessment criteria, rather than quantifiable, measurable data for ICU sedation. This often results in poor quality and inconsistent treatment of patient agitation. Advances in computer vision techniques can potentially assist the medical staff in assessing sedation and agitation by constantly monitoring the patient and providing the clinician with quantifiable data for ICU sedation. An automatic sedation and pain assessment system can be used within a decision support system, which can also provide automated sedation and analgesia in the ICU [31]. In order to achieve closed-loop sedation control in the ICU, a quantifiable feedback signal is required that reflects some measure of the patient's agitation. A nonsubjective agitation assessment algorithm can be a key component in developing closed-loop control algorithms for ICU sedation.

The current clinical standard in the ICU for assessing the level of sedation in adults is an ordinal scoring system, such as the motor activity and assessment scale (MAAS) [32] or the Richmond agitation–sedation scale (RASS) [33], which includes the assessment of the level of agitation of the patient as well as the level of consciousness. For example, the MAAS system evaluates the level of sedation and agitation on a score of 0–6 as follows: 0–unresponsive; 1–responsive only to noxious stimuli; 2–responsive to touch or name; 3–calm and cooperative; 4–restless and cooperative; 5–agitated; and 6–dangerously agitated.

Assessment of the level of sedation and agitation of a patient is, therefore, subjective and limited in accuracy and resolution, and hence, prone to error in assessing the level of sedation, which in turn may lead to oversedation. In particular, oversedation increases risk to the patient since liberation from mechanical ventilation, one of the most common life-saving procedures performed in the ICU, may not be possible due to a diminished level of consciousness and respiratory depression from sedative drugs resulting in prolonged length of stay in the ICU. Prolonged ventilation is expensive and is associated with known risks, such as inadvertent extubation, laryngotracheal trauma, and ventilator-associated pneumonia. Alternatively, undersedation leads to agitation and can result in dangerous situations for both the patient and the intensivist. Specifically, agitated patients can do physical harm to themselves by dislodging their endotracheal tube, which can potentially endanger their life.

While speculative, computer vision techniques offer the possibility to quantify agitation in sedated ICU patients. In particular, such techniques can be used to develop objective agitation measurements from patient motion. In the case of paraplegic patients, whole body movement is not available, and hence, monitoring the whole body motion is not a viable solution. In this case, measuring head motion and facial grimacing for quantifying patient agitation and sedation in critical care can be a useful alternative. Of course, patient occlusions due to medical equipment will need to be accounted for within the machine learning algorithms.

In future research, we will investigate the use of digital imaging and digital video of a patient's entire body movement, as well as facial expressions to assess agitation and sedation in the ICU. In addition, correlations between our objective measurements for agitation and pain intensity using digital imaging and a clinical standard assessment (e.g., MAAS or RASS score) will also be investigated. Furthermore, we will develop an expert control system predicated on digital imaging to emulate a clinician's deductive drug dosing process, that is, the process whereby a clinician successfully infers drug dosing conclusions based on the clinical standard of an assessed MAAS or RASS score. This expert control system can be used within a decision support system to provide closed-loop control for ICU sedation and analgesia, as well as critical care monitoring and lifesaving interventions.

Fig. 3.

Pain score for subject 2.

Fig. 5.

Pain score for subject 4.

Biographies

Behnood Gholami (S'04) received the B.Sc. and M.A.Sc. degrees in mechanical engineering from the University of Tehran, Tehran, Iran, and Concordia University, Montreal, Canada, in 2003 and 2005, respectively. In 2009 he received the M.S. degrees in mathematics and aerospace engineering from the Georgia Institute of Technology, Atlanta, GA. He is currently working toward the Ph.D. degree in aerospace engineering at the Georgia Institute of Technology.

In 2007, he was a summer Intern with the Control and Identification Group, The MathWorks, Inc., Natick, MA. His current research interests include computer vision, machine learning, vision-based control, medical image processing, optimal control, robust control, model predictive control, biomedical and biological systems, active control for clinical pharmacology, and image-guided therapy.

Mr. Gholami was a recipient of the Concordia University Graduate Fellowship and a Power Corporation of Canada Graduate Fellowship.

Wassim M. Haddad (S'87–M'87–SM'01–F'09) received the B.S., M.S., and Ph.D. degrees in mechanical engineering from Florida Institute of Technology, Melbourne, FL, in 1983, 1984, and 1987, respectively, with specialization in dynamical systems and control.

From 1987 to 1994, he was a Consultant with the Structural Controls Group, Government Aerospace Systems Division, Harris Corporation, Melbourne, FL. In 1988, he joined the Faculty of the Mechanical and Aerospace Engineering Department, Florida Institute of Technology, where he founded and developed the systems and control option within the graduate program. Since 1994, he has been a member of the Faculty with the School of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA, where he is currently a Professor. He has authored or coauthored more than 500 archival journal and conference publications with research contributions in linear and nonlinear dynamical systems and control. He is a coauthor of the books Hierarchical Nonlinear Switching Control Design with Applications to Propulsion Systems (Berlin, Germany: Springer-Verlag, 2000), Thermodynamics: A Dynamical Systems Approach (Princeton, NJ: Princeton University Press, 2005), Impulsive and Hybrid Dynamical Systems: Stability, Dissipativity, and Control (Princeton, NJ: Princeton University Press, 2006), Nonlinear Dynamical Systems and Control: A Lyapunov-Based Approach (Princeton, NJ: Princeton University Press, 2008), and Nonnegative and Compartmental Dynamical Systems (Princeton, NJ: Princeton University Press, 2010). His current research interests include nonlinear robust and adaptive control, nonlinear dynamical system theory, large-scale systems, hierarchical nonlinear switching control, analysis and control of nonlinear impulsive and hybrid systems, adaptive and neuroadaptive control, system thermodynamics, thermodynamic modeling of mechanical and aerospace systems, network systems, nonlinear analysis and control for biological and physiological systems, and active control for clinical pharmacology.

Dr. Haddad is a National Science Foundation Presidential Faculty Fellow and a member of the Academy of Nonlinear Sciences.

Allen R. Tannenbaum (M'93–F'09) received the B.A. and Ph.D. degrees in mathematics from Columbia University, New York, NY, and Harvard University, Cambridge, MA, in 1973 and 1976, respectively.

He is currently a Faculty Member with the Schools of Electrical and Computer, and Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA. His research interests include control, image processing, and computer vision.