Low-cost thermal imaging with machine learning for non-invasive diagnosis and therapeutic monitoring of pneumonia

Abstract

Rapid screening and early treatment of lung infection are essential for effective control of many epidemics such as Coronavirus Disease 2019 (COVID-19). Recent studies have demonstrated the potential correlation between lung infection and the change of back skin temperature distribution. Based on these findings, we propose to use low-cost, portable and rapid thermal imaging in combination with image-processing algorithms and machine learning analysis for non-invasive and safe detection of pneumonia. The proposed method was tested in 69 subjects (30 normal adults, 11 cases of fever without pneumonia, 19 cases of general pneumonia and 9 cases of COVID-19) where both RGB and thermal images were acquired from the back of each subject. The acquired images were processed automatically in order to extract multiple location and shape features that distinguish normal subjects from pneumonia patients at a high accuracy of 93$\%$. Furthermore, daily assessment of two pneumonia patients by the proposed method accurately predicted the clinical outcomes, coincident with those of laboratory tests. Our pilot study demonstrated the technical feasibility of portable and intelligent thermal imaging for screening and therapeutic assessment of pneumonia. The method can be potentially implemented in under-resourced regions for more effective control of respiratory epidemics.

1. Introduction

Pneumonia remains one of the most leading causes of morbidity and mortality over the world [1], [2], [3], particularly in under-resourced environments [4]. Since December 2019, the COVID-19 pandemic caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), has imposed a significant burden on the healthcare systems worldwide [5], [6], [7]. Early and rapid detection of respiratory disorders is essential to prevent their further transmission. Commonly used non-invasive imaging techniques for the diagnosis of pneumonia include chest X-ray (CXR) [8] and computed tomography (CT) [9]. While CXR and CT with advanced image processing and artificial intelligent (AI) algorithms are able to detect pneumonia with accuracy and spatial resolution [10], [11], [12], [13], they pose the potential radiation hazard, especially for babies and pregnant women [14]. Besides, the high cost and the poor portability of these equipment items have prevented their broad applications in under-resourced regions. Therefore, it is important to find a portable, low-cost, safe, non-invasive, and fast tool for pneumonia screening.

Infrared thermal imaging (TI) is a fast, non-invasive, cost-effective, and safe imaging technique that has been widely used in many clinical applications, such as early detection of breast cancer [15] and skin cancer [16]; assessment of burn depth [17]; and fever screening at public places during pandemics [18] such as severe acute respiratory syndrome (SARS) [19] and influenza A pandemic (H1N1) [20]. TI can create a visible image from the invisible infrared radiation emitted from the surface of the human body [21], which reveals the distribution of the skin temperature. It is known that skin temperature is a significant indicator of illness. Whether generalized or localized, the change in skin temperature is recognized as the presence of diseases [22]. This is because injury [23], infection [24], inflammation [25], or malignancy [26] is associated with localized vasodilation leading to tissue hyper-perfusion to the affected area and increased metabolic processes, all which lead to a higher emitted temperature. In a recent study [27], Wang et al. investigated the diagnostic accuracy of thermal imaging for focal pneumonia with chest X-ray as the gold standard, they found that the skin temperature of the lung mapping area on the back of patients with pneumonia increased significantly, and this temperature difference can be used to assist the diagnosis of pneumonia. Similar results can be found in the other study [28]. Rafael Y. Brzezinski et al. developed a non-invasive and portable thermal imaging tool to detect COVID-19 and lung injury based on the analysis of thermal images of back. Since TI is very sensitive for detecting small changes in the temperature distribution of the skin, which helps to access valuable information regarding the underlying physiological process causing diseases, it is a potentially new screening method to aid detection and monitoring of pneumonia including COVID-19.

Recently, machine learning (ML) methods have achieved state-of-the-art performance in a wide range of innovative applications of TI, such as livestock science [29], electric substation safety [30], advanced driver-assistance systems [31] and especially disease detection [32], [33]. During COVID-19 pandemic, TI based on ML has been employed to monitor the health conditions of people wearing masks by monitoring the respiratory systems. A Bidirectional Gate Recurrent Unit with an ATtention mechanism (BiGRU-AT) was applied to provide detection of abnormal breathing patterns with an accuracy of 83.69$\%$, sensitivity of 90.23$\%$ and specificity of 76.31$\%$ [34]. Besides respiratory pathological patterns, a temperature asymmetry of $⩾$ ${\text{0.55}}^{°}\text{C}$ between the lacrimal caruncle and the forehead through a random forest analysis was found to distinguish COVID-19 patients from healthy subjects with an accuracy of 82$\%$ [35]. Therefore, ML could constitute a suitable tool, to exploit both spatial and temporal TI features, potentially providing optimal classification performances.

In this manuscript, we developed advanced processing algorithms for thermal images of the back, captured by a portable thermal imager connected directly to a smartphone, to serve as a non-invasive tool for screening and treatment monitoring of pneumonia including COVID-19. Our proposed automated thermal imaging method is portable, low-cost, safe, and fast. It can be potentially used as an auxiliary diagnostic tool in the under-resourced regions to curb the spread of the COVID-19 epidemic.

2. Materials and Methods

In this section, the overview of the study, shown in Fig. 1 , will be discussed in detail. Specifically, the procedures for the experiment, including experiment subjects and data acquisition, are first documented. Then the data preprocessing as well as the feature selection criteria are presented. Finally, the machine learning methods implemented in the data analysis and classification are introduced.

Fig. 1.

Overview of the study.

2.1. Subjects

The objective of this study is to test the clinical feasibility of thermal imaging and machine learning in pneumonia detection. We enrolled normal subjects, fever patients and suspicious pneumonia patients from the First Affiliated Hospital of University of Science and Technology of China. Subjects were excluded if they had congestive heart failure, skin disease or rheumatic disease of the back, or lung malignancy. All patients (fever patients and suspicious pneumonia patients) took thermal imaging at their first visit before receiving any specific medical treatment. The final used pneumonia thermal images were selected from the enrolled suspicious patients based on the CT scans and laboratory characteristics results with diagnosis of pneumonia. Eventually, 69 adult subjects, which consist of 12 females aged 19–64 years and 57 males aged 18–80 years, were recruited in the study, and divided into two groups: pneumonia negative group and pneumonia positive group. The pneumonia positive group comprised of 19 cases of general pneumonia aged 59 $\pm$ 24 years and 9 cases of COVID-19 aged 59 $\pm$ 14 years. The pneumonia negative group included 30 normal adults aged 44 $\pm$ 17 years and 11 cases of fever without pneumonia aged 55 $\pm$ 16 years. Except age, the height and weight of all subjects were recorded to calculate their body mass index (BMI). As shown in Table 1 , each subject has similar BMI. The use of human subject in this study was approved by the ethics committee of the First Affiliated Hospital of University of Science and Technology of China with the reference number 2020-KY645, and all subjects provided their written informed consent.

Table 1.

Basic information of all subjects.

	Pneumonia Negative		Pneumonia Positive
	Normal	Fever	General Pneumonia	COVID-19
Number of Subjects	30	11	19	9
Age	44 $\pm$ 17	55 $\pm$ 16	59 $\pm$ 24	59 $\pm$ 14
BMI	24 $\pm$ 3	23 $\pm$ 4	23 $\pm$ 6	23 $\pm$ 2

2.2. Clinical data acquisition

Given that infrared radiation emitted from skin surface depends on several environmental factors like surrounding temperature, humidity, and the physiological condition of subjects[36], it is therefore essential to perform the medical thermal imaging experiments in a controlled environment and at a similar time. Our experiments were performed in the morning and in a climate control room with temperature and humidity respectively ranging from ${23}^{°}\text{C}$ to ${24}^{°}\text{C}$ and 45–55$\%$ and without direct sunlight or air flow. Before starting the experiment, subjects were required to wait for 5–10 min so as to get used to the temperature of the room. Then, they were asked to remove their clothing to expose the back and sit still on a chair. In our study, we used a FILR ONE pro thermal imager directly connecting to an Android smartphone to acquire the back images of each subject. The FILR ONE pro thermal imager is low-cost and portable and can detect signals over the spectral range of 8–14 μm. In addition, its detecting temperature ranges from −${20}^{°}\text{C}$ to ${400}^{°}\text{C}$ with a temperature resolution of $0.1^{°}\text{C}$. Moreover, this thermal imager contains two cameras, which can capture a RGB image with image resolution of 1440 × 1080 pixels and a thermal image with image resolution of 160 × 120 pixels of the object at the same time. Hence, two back images including a RGB image, and a thermal image were captured for each subject. The subject is 1 m distance away from the detector, as presented in Fig. 2 .

Fig. 2.

(a) schematic diagram of thermal imaging system for pneumonia screening. (b) procedures of feature selections.

To characterize the pneumonia and develop viable classification models, it is necessary to perform feature selection to reduce dimensionality of raw data. According to Fig. 3 , normal subjects, general pneumonia patients and fever patients show characteristic patterns in the region around temperature maxima. Specifically, a high temperature region congregates around spines for normal subjects, while this region lies around the lung area for pneumonia patients. And for the fever patients, the high temperature regions have no clear correlation with the lung or the spine. For the sake of quantification, the high temperature region should be defined. In this work, we chose two temperature bands with bandwidth of 1 Celsius as the region of interest (ROI). Those two temperature bands are referred to as high temperature region and sub-high temperature region defined in the Step 3 below. It is worth mentioning that the bandwidth is selected as 1 Celsius simply for convenience. The formal feature selection procedures, as illustrated in Fig. 2(b), can be found below:

• •Step 1: Finding the maximal temperature of each thermal image, denoted by $T_{\mathit{max}}$.
• •Step 2: Marking the lung region in the corresponding RGB image by experienced practitioners.
• •Step 3: Locating the region in the thermal image where the temperature is ranging from $T_{\mathit{max}}$-1 to $T_{\mathit{max}}$ (such region is named high temperature region for convenience) as well as the region from $T_{\mathit{max}}$-2 to $T_{\mathit{max}}$-1 (such region is referred to as sub-high temperature region).
• •Step 4: Extracting the temperature in the overlapping area of step 2 and step 3.
• •Step 5: Calculating the indices respectively for high temperature region and sub-high temperature region, defined in Table 3.
• •Step 6: Performing Principal Component Analysis (PCA) to the indices.

Fig. 3.

Typical examples of thermal imaging results of one normal person, one pneumonia patient and one fever patient without pneumonia. (a) thermal imaging result of one normal person; (b) thermal imaging result of one pneumonia patient shows temperature increases (red or white areas) in both right and left lung; (c) thermal imaging result of one fever patient without pneumonia.

2.3. Machine Learning Models

The extracted indexes are then subject to data analysis for characterizing pneumonia. To do so, different machine learning models are employed on thermal images of patients in healthy state, general pneumonia, and fever for pneumonia differentiation and for ease of monitoring of recovery phases. The classification models involved in this study are support vector machine (SVM), K-nearest neighbor (KNN), decision tree, Gaussian Naïve Bayes classifier, linear or quadratic discriminant analysis (LDA and QDA). In this study, depending on the types of labels, binary classification (pneumonia/normal or pneumonia positive/negative) and multiclass classification problem (pneumonia/fever/normal) will be achieved. All the programs are written in Python. Leave-one-out cross validation is performed to the dataset for training and testing. The brief introduction of applying each model to the collected data is presented as following.

2.3.1. Support Vector Machine

Support vector machine (SVM) is a supervised learning model which has been widely adopted in classification and regression problem. The basic idea of SVM is to find a hyperplane to maximize the distance to the dataset. In binary classification, the dataset $D=\{({\overrightarrow{x}}_1,y_1),({\overrightarrow{x}}_2,y_2),\dots ,({\overrightarrow{x}}_n,y_n)\},y_i\in \{-1,+1\}$ comprises of the indexes extracted from thermal images and the corresponding labels. Then the hyperplane satisfies:

$$\left(\overrightarrow{w},\overrightarrow{x}\right)+b=0$$

where w denotes normal vector of the hyperplane, and b is the offset. $\left(·,·\right)$ represents inner product. If $\left(\overrightarrow{w},\overrightarrow{x}\right)+b>0$, then the output $\widehat{y_i}=+1$, otherwise, the output $\widehat{y_i}=-1$.

The goal of training SVM is to optimize the margin, which is defined as the summation of distance from support vectors in different classes to the hyperplane, i.e.,

$$\left\{\begin{array}{c}{\displaystyle \underset{\overrightarrow{w},b}{\max }}\frac{2}{||\overrightarrow{w}||}\hfill \\ y_i\left(\left(\overrightarrow{w},\overrightarrow{x}\right)+b\right)⩾1\hfill \end{array}\right)$$

Above method can be generalized to nonlinear problem, where the dataset is not linearly divided. In such case, the kernel functions are introduced to replace inner product. The commonly chosen kernel functions include:

• •
  polynomial kernel:

  $$K(x,y)={((x,y)+1)}^d\text{.}$$
• •
  RBF kernel:

  $$K(x,y)=\exp \left(-\frac{{‖x-y‖}^2}{2{\sigma }_0^2}\right)\text{.}$$

When dealing with multiclass problems, the one-vs-one or one-vs-rest strategies are utilized. In one-vs-one, multiple SVM classifiers are trained to differentiate all possible pairs of classes whereas in one-vs-rest, the classifiers are to distinguish each class with respect to the entity of all remaining classes.

In SVM, per-class probabilities for each sample can be estimated by the logistic regression. The basic form of logistic regression for binary classification is:

$$\mathit{logist}(x)=\frac{1}{1+\exp (-x)}$$

By fitting an additional cross-validation on the training set, the output of logist, which lies between 0 and 1, is the probability. In our study, such probability is used for assessing the likelihood of the patient’s suffering pneumonia and also used for recovery monitoring.

2.3.2. K-Nearest Neighbors

K-Nearest neighbors algorithm is a non-parametric learning model, where the prediction is only determined by the labels of nearest data points. Since the model relies on the distance between the target and the dataset, normalization of the features, such that each feature is in the same scale, will lead to a boost in performance. In this algorithm, an integer N is first selected (usually 5). Then prediction of a certain data point is based on the highest votes among K closest data points in the training set.

Consider binary classification (pneumonia/normal), assuming the training dataset is $D=\{({\overrightarrow{x}}_1,y_1),({\overrightarrow{x}}_2,y_2),\dots ,({\overrightarrow{x}}_n,y_n)\},y_i\in \{-1,+1\}$. By sorting the elements with respect to a certain norm $||·||$, we get $\{({\overrightarrow{x}}_{(1)},y_{(1)}),({\overrightarrow{x}}_{(2)},y_{(2)}),\dots ,({\overrightarrow{x}}_{(n)},y_{(n)})\}$ such that $||{\overrightarrow{x}}_{(1)}-\overrightarrow{x}||⩽\dots ⩽||{\overrightarrow{x}}_{(n)}-\overrightarrow{x}||$. The target label $\widehat{y}$ of input x is determined by the majority of first K labels.

Euclidean distance is usually selected as the criterion in the algorithm. When the variables are discrete, e.g. text recognition, Pearson coefficient or other distances are usually adopted [37]. In addition, some modern KNN algorithms, e.g. Large Margin Nearest Neighbor (LMNN) or Neighborhood components analysis (NCA) can automatically optimize and redesign the choice of measure.

2.3.3. Decision tree

Decision tree is widely used in data mining, decision analysis et al. In this study, decision tree can distinguish pneumonia data well since it has high interpretability while not sacrificing accuracy. By establishing a tree-structure, the decision-making process is visualized, and the label is determined by querying the value of each index until the leaf node. During training process, the essential knowledge of indexes is purified by branching and calculating the entropy. Such model is hierarchical, and the classification rules are clear by identifying the path from the root to leaf.

The procedures for performing decision tree on pneumonia data can be summarized as:

• •Identifying the types of each index.
• •Deciding the index for the current branch.
• •Finding the optimal threshold for querying the certain feature by evaluating information gain.
• •Repeating the above steps until reaching the leaf node.

The information entropy is commonly adopted to determine the feature and threshold in branching. Assuming N class, the definition is

$$H_N(D)=-{\displaystyle \sum _{i=1}^2}{p}_i{\log }_{10}{p}_i$$

where $p_i$ denotes the ratio of data occupying $i^{\mathit{th}}$ class in the current dataset D. Smaller value of $H_N(D)$ represents higher purification of current branching. Assume a feature is discrete, and collection of values are in $\{x^1,x^2,\dots ,x^m\}$. Let $D_i$ represents the quantity of sample whose value of such feature is $x_i$. Then by entropy, the information gain is

$$\mathit{Gain}(D)=H_N(D)-{\displaystyle \sum _{i=1}^m}\frac{D^i}{D}{H}_N(D^i)$$

By comparing the information gain of each index, the optimal division is determined, and finally, the label, e.g., pneumonia or normal, can be obtained.

2.3.4. Discriminant Analysis

Proposed by Ronald A. Fisher in 1936, linear discriminant analysis (LDA) is an important statistical method, which is commonly adopted in classification or dimensionality reduction. The goal of LDA is to transform the pneumonia data onto the low dimensional space, such that the transformed data set is linearly separatable. LDA has the advantages of avoiding over-fitting and low computational cost. QDA (quadratic discriminant analysis) and Gaussian Naïve Bayes classifier are the variants of LDA. The difference between three models is the simplification of the covariance matrix. The brief introduction of discriminant analysis is shown below:

Considering a multi-class problem, for each sample x,

$$P(y=k|\overrightarrow{x})=\frac{P(y=k)P(\overrightarrow{x}|y=k)}{P(\overrightarrow{x})}$$

where k is the label. Assuming $P(\overrightarrow{x}|y=k)$ satisfies Gaussian distribution, it holds that

$$P(\overrightarrow{x}|y=k)=\frac{1}{\sqrt{{(2\pi )}^d|{\mathrm{\Sigma }}_k{|}^2}}\exp \left(-\frac{1}{2}{(\overrightarrow{x}-{\overrightarrow{\mu }}_k)}^t{\mathrm{\Sigma }}_k^{-1}(\overrightarrow{x}-{\overrightarrow{\mu }}_k)\right)$$

where d is the dimension of $x,{\mathrm{\Sigma }}_k,{\overrightarrow{\mu }}_k$ represent covariance matrix and mean vector of class k respectively. Then,

$$\log P(y=k|\overrightarrow{x})=-\frac{1}{2}\log |{\mathrm{\Sigma }}_k|-\frac{1}{2}{(\overrightarrow{x}-{\overrightarrow{\mu }}_k)}^t{\mathrm{\Sigma }}_k^{-1}(\overrightarrow{x}-{\overrightarrow{\mu }}_k)+\log P(y=k)+C$$

The constant C is independent to the class k. In QDA, $\log P(y=k|\overrightarrow{x})$ is obtained by estimating ${\mathrm{\Sigma }}_k$ without simplification. In particular, Gaussian Naïve Bayes classifier is a special case where the covariance matrix ${\mathrm{\Sigma }}_k$ is assumed to be diagonal. Different from QDA and Gaussian Naïve Bayes, it is assumed in LDA that all classes k share the same covariance matrix, i.e.,

$$\log P(y=k|\overrightarrow{x})=-\frac{1}{2}\log |\mathrm{\Sigma }|-\frac{1}{2}{(\overrightarrow{x}-{\overrightarrow{\mu }}_k)}^t{\mathrm{\Sigma }}^{-1}(\overrightarrow{x}-{\overrightarrow{\mu }}_k)+\log P(y=k)+C$$

Therefore,

$$\log P(y=k|\overrightarrow{x})=-a_k^t\overrightarrow{x}+b_k+C$$

where $a_k={\mathrm{\Sigma }}^{-1}{\overrightarrow{\mu }}_k,b_k=-\frac{1}{2}{\overrightarrow{\mu }}_k^t{\mathrm{\Sigma }}^{-1}{\overrightarrow{\mu }}_k+\log P(y=k)$.

The choice of models heavily depends on the basic structure of dataset, i.e., the correlation between different indexes of thermal images. In this study, LDA, QDA and Gaussian Naïve Bayes model are all implemented for comparison.

3. Results

In this section, a culmination of all the experimental tests were presented. Specifically, the classification results using different machine learning models to distinguish pneumonia patients from other subjects and the monitoring results during treatment of one recovered pneumonia patient and one unrecovered pneumonia patient.

3.1. Detection of pneumonia patients

Here, we confirmed and validated the ability of our proposed method to distinguish pneumonia patients from other subjects in three different settings using different machine learning models, as shown in Table 2 . The first setting is a two-classification including normal subjects (n = 30) and pneumonia positive patients (n = 28). All the six machine learning models performed well. For examples, SVM, KNN and Decision tree even achieved a high accuracy more than 90$\%$. However, except pneumonia, fever may also alter the thermal patterns of back from normal by increasing the absolute temperature of the back. Thereby, introducing potential confounding factors to the analysis, the second setting is a two-classification including pneumonia negative patients (n = 41) and pneumonia positive patients (n = 28). Both SVM and Gaussian NB got an accuracy more than 80$\%$. However, the rest models only got an accuracy between 70$\%$ and 80$\%$. The last setting is a three-classification including normal subjects (n = 30), fever patients without pneumonia (n = 11), and pneumonia negative patients (n = 28). Compared to the results from the first and second setting, all six models didn’t show good performance in the third setting. Only SVM and Gaussian NB had an accuracy more than 70$\%$. The other methods had an accuracy between 60$\%$ to 70$\%$. This is probably because there is no clear boundary for back thermal images between normal person and fever patient. In fact, the temperature change of fever patient is not always reflected on the back thermal images. It may be difficult to differentiate the patients with low-grade fever and normal people. In addition, it is worthwhile to mention that the performance for four-class classification (normal/fever/general pneumonia/Covid-19) is poor. This is because the back thermal images for general pneumonia and Covid-19 have little difference. Overall, SVM had the best performance in the three settings among all six models. Therefore, we used SVM model to monitor the recovery of two pneumonia patients during treatment.

Table 2.

Classification results based on different machine learning models. Compared with other methods, SVM shows the best performance. Note: The pneumonia negative group includes 30 normal adults and 11 cases of fever without pneumonia. The pneumonia positive group comprised of 19 cases of general pneumonia and 9 cases of COVID-19.

	Classification Accuracy
	Norm vs. Pneu Pos	Pneu Pos vs. Pneu Neg	Norm vs. Fever vs. Pneu Pos
SVM	93%	86%	81%
KNN	91%	78%	68%
Decision tree	93%	78%	70%
Gaussian NB	90%	83%	75%
LDA	86%	71%	67%
QDA	85%	75%	65%

3.2. Monitoring of pneumonia patients

After validating the feasibility of our method to distinguish pneumonia patients from other subjects, we also verified the ability of this method to monitor the recovery of pneumonia patients during treatment. Fig. 4 showed the thermal imaging results during a 3-day therapy for a patient recovered from general pneumonia. And Fig. 5 presented the thermal imaging results during a 4-day therapy for a patient failed to recover from general pneumonia. According to Fig. 4, the areas of temperature maxima shrink, shift from the lung areas to the spine areas from day 1 to day 3, and from a pneumonia temperature pattern to a normal temperature pattern, indicating the recovery of pneumonia. However, in Fig. 5, the areas of temperature maxima didn’t change significantly from day 1 to day 4, indicating the failure of recovery from pneumonia. Our results of the two pneumonia patients were coincident with their clinical therapeutic assessments based on daily laboratory test results.

Fig. 4.

Typical examples of one recovered pneumonia patient after 3-days therapy. The thermal imaging images were taken from day 1 to day 3.

Fig. 5.

Typical examples of one unrecovered pneumonia patient after 4-days therapy. The thermal imaging images were taken from day 1 to day 4.

For further quantification, we used three SVM models based on three different classification settings to assess the probability of pneumonia of two panumonia patients during treatment. The first model was a SVM model based on a two-classification including normal subjects (n = 30) and pneumonia positive patients (n = 28). Fig. 6 (a) showed the plot of pneumonia probability of all subjects. The blue marker represented normal subjects, whose pneumonia probability were almost all around 0. The red marker represented pneumonia positive patients, whose pneumonia probability were almost all above 0.9. Fig. 6(b) was the plot of pneumonia probability of one recovered pneumonia patient after 3-days therapy. The probability of pneumonia decreased from day 1 to day 3, which was from 0.96 to 0.31 indicating the recovery of the patient. Fig. 6(c) was the plot of pneumonia probability of one unrecovered pneumonia patient after 4-days therapy. The probability of pneumonia almost matained above 0.9 from day 1 to day 4, indicating the unrecovery of the patient. The second model was a SVM model based on a two-classification including pneumonia negative patients (n = 41) and pneumonia positive patients (n = 28). Fig. 7 (a) showed the plot of pneumonia probability of all subjects. The blue marker represented pneumonia negative subjects, whose pneumonia probability were almost all under 0.2. The red marker represented pneumonia positive patients, whose pneumonia probability were almost all above 0.8. Compared to Fig. 6(a), the number of false points increased in Fig. 7(a). Fig. 7(b-c) were the plot of pneumonia probability of one recovered pneumonia patient after 3-days therapy and one unrecovered pneumonia patient after 4-days therapy, respectively. Similar trend in Fig. 6(b-c) can be found in Fig. 7(b-c). However, the value of the probability of pneumonia decreased in Fig. 7(b-c). The third model was a SVM model based on a three-classification including normal subjects (n = 30), fever patients without pneumonia (n = 11), and pneumonia negative patients (n = 28). Fig. 8 (a) showed the plot of pneumonia probability of all subjects. The blue marker represented normal subjects, whose pneumonia probability were almost all under 0.2. The cyan maker represented fever patients without pneumonia. Though most of the fever patients had a low pneumonia probability under 0.4, two fever patients were above 0.5 and one fever patient was around 0.5, which were the false points. The red marker represented pneumonia positive patients, whose pneumonia probability were almost all above 0.8. Fig. 8(b-c) were the plot of pneumonia probability of one recovered pneumonia patient after 3-days therapy and one unrecovered pneumonia patient after 4-days therapy, respectively. Similar trend in Fig. 8(b-c) can be found in Fig. 6(b-c) and Fig. 7(b-c). However, the value of the probability of pneumonia decreased in Fig. 8(b-c), compared to Fig. 6(b-c), and Fig. 7(b-c).

Fig. 6.

Plots of pneumonia probability by SVM model based on normal and pneumonia. (a) Plot of pneumonia probability of all subjects. (b) Plot of pneumonia probability of one recovered pneumonia patient after a 3-day therapy. (c) Plot of pneumonia probability of one unrecovered pneumonia patient after a 4-day therapy.

Fig. 7.

Plots of pneumonia probability by SVM model based on pneumonia positive and pneumonia negative. (a) Plot of pneumonia probability of all subjects. (b) Plot of pneumonia probability of one recovered pneumonia patient after 3-days therapy. (c) Plot of pneumonia probability of one unrecovered pneumonia patient after 4-days therapy.

Fig. 8.

Plots of pneumonia probability by SVM model based on normal, fever and pneumonia positive. (a) Plot of pneumonia probability of all subjects. (b) Plot of pneumonia probability of one recovered pneumonia patient after 3-days therapy. (c) Plot of pneumonia probability of one unrecovered pneumonia patient after 4-days therapy.

4. Discussions

Current efforts to use thermal imaging in the fight against the COVID-19 pandemic have focused primarily on measuring absolute body temperature and screening for individuals with fever in crowded settings. However, these attempts have demonstrated limited effectiveness. Our innovative approach to extract advanced location and shape features from the thermal image other than absolute body temperature value, could help improve these screening tools and aid in the diagnosis of acute inflammatory changes in deep tissues, such as lung infection. We designed this study to determine if the thermal pattern of the back could be used as a diagnostic marker of pneumonia including COVID–19.

The thermal pattern is outlined by the high temperature region whose temperature ranges from $T_{\mathit{max}}$-1 to $T_{\mathit{max}}$. As shown in Fig. 3, significant differences were found between the normal, fever and pneumonia groups for the thermal patterns on the back, indicating its powerful discriminatory capacity to detect pneumonia. Though 1 degree Celsius difference in temperature is remarkable for human bodies, the temperature distribution outside of 1 degree Celsius may also have contribution to differentiating pneumonia. Hence, except the high temperature regions, we also defined a sub-high temperature region ($T_{\mathit{max}}$-2 to $T_{\mathit{max}}$-1) to expand the interested temperature region so that a significantly broader range of temperature distribution is considered.

Before performing the machine learning models, several indices (defined in Table 3) are proposed in this work to characterize the thermal patterns. For the sake of understanding the contribution of indices in diagnostics, feature ranking algorithm should be leveraged. Regarding this, we adopted permutation feature importance algorithm which randomly shuffles one feature value each time and then evaluates the model performance. Fig. 9 shows some top-ranked features, among which the standard deviation of lung temperature is the most important one. This agrees with our observation that the lung temperature variation in the patients with pneumonia is much larger than normal people or fever patients. Besides $S_r$, the area ratio index, distance index and torque index are similarly important.

Fig. 9.

Ranking of top feature importance. The feature importance is evaluated based on SVM models.

Although the thermal images used for classification were taken before the patients’ undergoing any medical treatment, the thermal patterns of the skin may be influenced by age, gender and BMI differences. Therefore, these factors should be taken into account. In our study, all the enrolled subjects have similar BMI. However, pneumonia positive group is older than the normal group and the number of females is much smaller than the number of males. Therefore, future studies should take these into account and control as much as possible the effect of age and gender factors and the effect of medication on thermal patterns of the back.

Though the results of detection and monitoring of pneumonia patients were promising, there are several limitations. First, the sample size is not very large, and the cases of fever patients without pneumonia and the cases of COVID–19 patients are limited. As a preliminary study, the relatively high accuracy classification results and two typical monitoring results can reveal the feasibility of our method to distinguish pneumonia patients from other subjects and monitor the recovery of pneumonia treatment. However, a lager sample size with other will be needed for further validation of the clinical performance in the future. Secondly, due to the limitation of the cases of COVID–19, we didn’t analyze the difference between general pneumonia and COVID–19. So, after increasing the number COVID–19 patients, further analysis will be conducted. Thirdly, the lung and spine areas were marked manually by an experienced respiratory doctor before feature extraction, which may introduce variations. Therefore, in the future, we may use deep learning methods to mark the lung and spine areas automatically.

5. Conclusions

In this manuscript, we present a low-cost, portable, safe, and non-invasive thermal imaging method to aid diagnosis and monitoring of pneumonia based on a dual-mode thermal imager, which can record both RGB and thermal images of the back of each subject. In our method, we first used novel image processing algorithms to extract multiple location and shape features of the thermal images automatically and then applied machine learning analysis to achieve good performance on detection of pneumonia. For examples, a high accuracy of 93$\%$ in a two-classification setting including normal subjects (n = 30) and pneumonia positive patients (n = 28), a high accuracy of 86$\%$ a in a two-classification setting including pneumonia negative patients (n = 41) and pneumonia positive patients (n = 28), and a relatively high accuracy of 81$\%$ in a three-classification setting including normal subjects (n = 30), fever patients without pneumonia (n = 11), and pneumonia negative patients (n = 28). In addition, the monitoring results of two pneumonia patients by our method during treatment were consistent with their clinical results based on laboratory characteristics. Hence, our proposed method may facilitate the screening of people to detect and monitor pneumonia in real time, especially in low-income regions with limited imaging resources, which can help limit the spread of COVID-19. However, despite the very promissory results of our pilot study, more extensive trials are required to confirm our findings and validate them. Future studies should focus on determining whether the thermal patterns we identified can be used to discriminate between general pneumonia patients and COVID-19 patients.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Description of Indices Used in the Study

The following table summarizes the indices extracted from thermal imaging for the machine learning.

Table 3.

Definition and calculation of indices.

Index	Symbol	Description
Area ratio index of left lung	$R_l$	$\frac{A_{H,l}}{A_l}$
Area ratio index of right lung	$R_r$	$\frac{A_{H,r}}{A_r}$
Temperature index of left lung	$T_l$	average $T_i$ of the left lung
Temperature index of right lung	$T_r$	average $T_i$ of the right lung
Standard deviation of left lung	$S_l$	std of $T_i$ of the left lung
Standard deviation of right lung	$S_r$	std of $T_i$ of the right lung
Distance index of left lung	$P_l$	$\frac{{\sum }_{i=1}^{A_{H,l}}{(d_{l,\mathit{spine}})}_i}{A_{H,l}\times \sqrt{A}}$
Distance index of right lung	$P_r$	$\frac{{\sum }_{i=1}^{A_{H,r}}{(d_{r,\mathit{spine}})}_i}{A_{H,r}\times \sqrt{A}}$
Torque index of left lung	$M_l$	$\frac{{\sum }_{i=1}^{A_{H,l}}{(d_{l,\mathit{spine}})}_i·T_i}{A_{H,l}\times \sqrt{A}}$
Torque index of right lung	$M_r$	$\frac{{\sum }_{i=1}^{A_{H,r}}{(d_{r,\mathit{spine}})}_i·T_i}{A_{H,r}\times \sqrt{A}}$
H–L y index of left lung	${\mathit{Wy}}_l$	$\frac{{(C_{H,l})}_x-{(C_l)}_x}{\sqrt{A}}$
H–L y index of right lung	${\mathit{Wy}}_r$	$\frac{{(C_{H,r})}_x-{(C_r)}_x}{\sqrt{A}}$
H–L x index of left lung	${\mathit{Wx}}_l$	$\frac{{(C_{H,l})}_y-{(C_l)}_y}{\sqrt{A}}$
H–L x index of right lung	${\mathit{Wx}}_r$	$\frac{{(C_{H,r})}_y-{(C_r)}_y}{\sqrt{A}}$
Maximal spine temperature	${\mathit{Sp}}_{\mathit{max}}$	max temperature of spine
Average spine temperature	${\mathit{Sp}}_{\mathit{avg}}$	average temperature of the spine

The explanations of some indices are:

• •$R_l/R_r$: the percentage of area of high temperature regions in left/right lung region.
• •$T_l/T_r$: the average temperature of left/right lung.
• •$S_l/S_r$: standard deviation of temperature distribution in left/right lung.
• •$P_l/P_r$: the normalized distance between the centroid of high temperature region in left/right lung to the spine. The factor $\sqrt{A}$ is for normalization to eliminate the influence of lung size or the angle and distance when capturing the thermal images.
• •$M_l/M_r$: the normalized weighted temperature of high temperature region in left/right lung. This index is conceptually an extension of the torque in physics.
• •${\mathit{Wx}}_l/{\mathit{Wx}}_r/{\mathit{Wy}}_l/{\mathit{Wy}}_r$: the normalized distance between the centroid of high temperature region and lung region. $x/y$ means the x or y-axis coordinate. The index name H–L is short for high temperature region to lung region.
• •$A_{H,l}/A_{H,r}$: total number of pixels in high temperature region of left/right lung
• •A: total number of pixels of both lungs
• •$A_l/A_r$: total number of pixels of left/right lung
• •${(d_{l,\mathit{spine}})}_i/{(d_{r,\mathit{spine}})}_i$: distance from the pixels in left/right lung to the spine
• •$C_{H,l}/C_{H,r}$: coordinate of the centroid of left/right lung’s high temperature region