Autonomous Multi-modality Burn Wound Characterization using Artificial Intelligence

ABSTRACT

Introduction

Between 5% and 20% of all combat-related casualties are attributed to burn wounds. A decrease in the mortality rate of burns by about 36% can be achieved with early treatment, but this is contingent upon accurate characterization of the burn. Precise burn injury classification is recognized as a crucial aspect of the medical artificial intelligence (AI) field. An autonomous AI system designed to analyze multiple characteristics of burns using modalities including ultrasound and RGB images is described.

Materials and Methods

A two-part dataset is created for the training and validation of the AI: in vivo B-mode ultrasound scans collected from porcine subjects (10,085 frames), and RGB images manually collected from web sources (338 images). The framework in use leverages an explanation system to corroborate and integrate burn expert’s knowledge, suggesting new features and ensuring the validity of the model. Through the utilization of this framework, it is discovered that B-mode ultrasound classifiers can be enhanced by supplying textural features. More specifically, it is confirmed that statistical texture features extracted from ultrasound frames can increase the accuracy of the burn depth classifier.

Results

The system, with all included features selected using explainable AI, is capable of classifying burn depth with accuracy and F1 average above 80%. Additionally, the segmentation module has been found capable of segmenting with a mean global accuracy greater than 84%, and a mean intersection-over-union score over 0.74.

Conclusions

This work demonstrates the feasibility of accurate and automated burn characterization for AI and indicates that these systems can be improved with additional features when a human expert is combined with explainable AI. This is demonstrated on real data (human for segmentation and porcine for depth classification) and establishes the groundwork for further deep-learning thrusts in the area of burn analysis.

INTRODUCTION

Burn injuries represent a prevalent threat in warfare, accounting for up to 20% of combat-related injuries.¹ These complex and devastating injuries have long been a major cause of morbidity and mortality for servicemen and civilians.^2,3

In the exigent and resource-limited environment of the battlefield, rapid and accurate assessment of burn injuries is crucial for effective treatment.⁴ The severity and extent of burns play a critical role in determining the priority of treatment, transportation to appropriate facilities, and the overall therapeutic approach for wounded individuals.^5,6 Skilled wound care is essential for mitigating the consequences of burn injuries, including the risk of infection and mortality, as well as optimizing patient outcomes.⁷

Burn depth and the percentage of total body surface area (%TBSA) affected by the wound are the two key factors in evaluating burn injuries.⁸ However, assessing burn depth and surface area remains a significant clinical challenge—current clinical examination often yield unsatisfactory accuracy, even among experienced practitioners.⁹

In the conducted study, an artificial intelligence (AI) system was developed using the latest advancements in deep learning. This system was designed to predict the severity and surface area of burns; thereby enhancing the accuracy and efficiency of burn assessments. B-mode ultrasound data were employed to create a convolutional neural network (CNN)-based algorithm for burn severity classification, which was further enhanced in terms of interpretability using Explainable AI I (XAI) techniques. A separate module for estimating the burn’s surface area was also developed. The introduction of this AI-assisted approach will offer health care professionals access to more reliable and detailed injury information, enabling well-informed diagnoses and the provision of superior care in military environments.

BACKGROUND

Burn Wound Diagnosis

The characterization of a burn wound is largely determined by the depth and TBSA of the wound. Burn depth, in particular, is considered a major clinical challenge. For the purposes of this work, burns can be separated into three levels of severity: (1) Superficial burns, which only damage the epidermis. This classification includes sunburns. They are red, dry, warm, painful, and without blisters. They also blanch when pressure is applied. Superficial burns do not scar and heal in 5 to 10 days. (2) Partial-thickness burns, which damage the epidermis and the dermis, but do not move into the subcutaneous tissues. They can be swollen and painful, and appear red, moist, and blistered. Partial-thickness burns heal within 14 to 21 days with minimal to no scarring. (3) Full-thickness burns, which burn all the way into the subcutaneous tissues. There is usually minimal pain, and dry, leathery, white, or brown/black in appearance. Some involve damage to deeper structures like muscle and bone—these can be classified into another class but burns that severe are trivial to characterize. They heal with severe scarring and contractures, and the patients might experience itch, pain, loss of function, restricted movement, and physical deformity. In order to prevent these outcomes, early excision and grafting are required.^10,11

Clinical methods to characterize burns are manual in nature—conducted by the surgeon with very little assistance from advanced tools. In the case of the burn size estimation, the Wallace Rule of Nines and the Lund and Browder chart tend to significantly overestimate the TBSA burned, with a mean overestimation accuracy of 170%.^12,13 This can lead to excessive fluid resuscitation, in pulmonary complications, compartment syndrome, and an increased need for escharotomy.¹² For burn depth, it is determined by clinical assessment based on appearance, blanching to pressure, sensation to pin prick, and bleeding on needle prick. This visual and tactile inspection approach introduces inter-subject variability, especially when partial-thickness burns are involved.¹³ Experienced burn surgeons achieve an accuracy of 67–76%, value that decreases to 50% for inexperienced practitioners.^14,15 Wider estimates put clinical accuracy between 60% and 80%.¹⁶ This underestimation or overestimation can lead to delayed healing times, significant hypertrophic scarring, even unnecessary surgery.¹⁷

Deep Learning in Burn Wound Assessment

Many deep learning architectures have been proposed recently for medical imaging and diagnostics purposes, as well as a variety of imaging modalities used in medical imaging, such as ultrasound, CT, MRI, and X-rays.^18,19 The tasks targeted by these models can include classification, segmentation, image upscaling, and other forms of image-to-image and image-to-data learning. Deep learning with ultrasound has been applied effectively to detect liver diseases²⁰ as well as some cancers.²¹ In the area of burn assessment, work has been accomplished in machine learning overall²² and deep learning more specifically.^23,31 Many approaches rely on RGB images of the skin,^25,26 and while these can achieve decent accuracies, they are not invariant to skin pigmentation, ambient light, or camera positioning.

Explainable AI (XAI)

An XAI is an intelligent system which can make decisions that can be explained and interpreted by a human.²⁷ XAI produce explanations of the data which can tell the user why the AI made the decision that it did (local explanations), or how the AI makes decisions in general (global explanations). XAI can produce rules, data visualizations, or images. No work known to the authors has addressed the use of XAI in the burn assessment domain.

METHODS AND MATERIALS

Overview

The depth classifier and explainer modules collaboratively predicted the depth class of burn wounds, utilizing B-mode ultrasound as the primary modality. Meanwhile, the segmentation module differentiated the burn wound and body from overhead RGB images. The depth classifier and segmentation network were trained on distinct datasets, whereas the explainer module necessitated no training.

Datasets Acquisition

An unreleased B-mode dataset was acquired for each of the three burn classes, as well as unburnt healthy skin. Female Yorkshire pig models were selected as an in vivo wound-healing animal model. Pigs were selected because of the high similarity between pig and human skin. Dermal–epidermal ratio, the dermal collagen, the distribution of blood vessels, and an abundant subdermal adipose tissue are all similar between the two.^28,29

The burning process and ultrasound data collection were reported previously to maintain reproduciblity.^30,31 The GE Logiq E9 device was used to capture ultrasound video clips, which were then split into frames. The scanning of the wound was recorded as videos starting from the midline towards the outside for every timepoints. Data extraction from the collected video clips resulted in thousands of frames.

Female pigs were anesthetized with Telazol, followed by isoflurane inhalation. Eight (2 × 2 inch) burn sites were marked in the previously shaved dorsal region of anesthetized female pigs. Each pig has six wounds (2 superficial burn, 2 partial thickness, 2 full thickness, and 2 marked unburnt sites) and the burns were created using 150 °C heated device with standardized at 1, 10, and 60 s. The wounds were distributed across the pig dorsum starting with superficial thickness, then partial thickness, then full thickness, and finally the unburnt sites. To eliminate bias stemming from the anatomical variations in skin thickness, the placement of each type of burn was randomized on each individual pig. The study primarily used two pigs, each with eight wound-sites. Four of the wound sites from each pig were used for medical validation, while the other four were scanned—one for each type of wound and one healthy skin site. In addition, four more pigs were utilized exclusively for training data, each receiving only full thickness burns for each wound. Cross-validation, a technique commonly used to evaluate the performance of machine learning models, was applied in the study. Initially, one pig was designated for training and model validation, while the other was used for evaluation. Upon completing the first round, the roles of the two pigs were reversed for a second round. A more reliable and comprehensive assessment of the model’s performance was obtained by averaging the results from both rounds.

After filtering out incomprehensible frames, this process produced a dataset of 6,421 full-thickness frames, 740 partial-thickness frames, 669 superficial frames, and 2,235 healthy skin frames. Class weighting based on the training set was used during training to overcome class imbalance, but no special measures were taken that could contaminate the testing set. These images were cropped from the ultrasound UI and resized to 224 × 224 pixels. Scans were taken on days 0, 3, 7, 14, 28, 35, and 42. Only day 0 was utilized in training and testing the model for this work to eliminate the factor of long-term healing. Histological analysis was performed to ensure that the ground truth class labels for each burn site were correct.

The Google search engine was used to create a dataset of RGB images for segmentation. The keywords used included burn, burn injury, burn wound, scald, partial-thickness burn, full-thickness burn, second degree burn, and third-degree burn. Only images visually containing partial-thickness and full-thickness burn injuries were included as these are considered in TBSA estimation. As a result, a dataset of 338 images each was obtained. These were resized to 256 × 256 pixels based on the resolution of the collected images and because it has shown satisfactory results for semantic segmentation problems. Additionally, the dataset was randomly divided into 246 images for training, 47 images for validation, and 45 images for testing.

As the dataset is collected from web sources, demographics are determined solely through individual examination of the images. Most of the images feature individuals with fair/light skin tones (298), followed by medium/tan (30), and brown/black (9). The burns appear on various body parts, with the arm (95) being the most common location, followed by the leg (65), hand (51), and others. The image quality is generally high, with 235 images being very clear and 102 moderately clear. In terms of lighting, the majority of the images are brightly lit (284), with a smaller number having medium lighting (54) and only one image having dim lighting. Furthermore, these images and their segmentation mask targets could not be validated biologically. Instead, the annotations produced for these burns were visually confirmed by an expert burn researcher.

Depth Classification with Ultrasound

The classification of images into burn depth categories was carried out using a deep CNN. Rather than training from scratch, the model parameters from the widely used classification network ResNet18, pretrained on the ImageNet dataset, were employed. Following common practice, the last layer of the network (the classification head) was removed, with the remaining network serving as a feature extractor that generated a latent representation of the image. This latent representation was then fed into a compact, two-layer fully connected classification network.

As detailed in the subsequent section, both prior literature^32–34 and the explainer system suggested that texture-based data would be crucial for this classification, and directly supplying such data would be beneficial. Consequently, five statistical texture features (each with six configurations) were extracted from the B-mode frames. These Haralick texture features³⁴ were derived from the image’s Gray Level Co-occurrence Matrix. The features were concatenated with the latent features produced from ResNet18 and fed into the fully connected classification network, which in turn output a probabilistic (Softmax) classification of the burn depth class. The model is visualized in Fig. 1.

FIGURE 1.

Our classification model. This makes use of a pre-trained ResNet34 convolutional neural network component modified with dropout to reduce overfitting to our small dataset. GLCM texture features are extracted and included as features in the final stage of the classifier. The entire classifier is trained as a unified model using gradient descent. The output is a four-part vector representing the probability of each label being the true burn depth class.

The model was trained on the B-mode ultrasound dataset for 15 epochs, utilizing a batch size of 8 and a 10-5 learning rate without decay. During this training process, the cross-entropy loss function was used for classification in combination with the Adam optimizer.³⁵ The loss function can be defined as:

(eq1)

where N is the number of classes (4), y_i is the target (ground truth label), and yˆ_i is the output of the model.

XAI-based Classifier Improvement

LIME (Local Interpretable Model-agnostic Explanations)³⁶ is primarily used in this work to provide explanations of how the input features of a data record are utilized by the classifier—indicating to what degree each part of an image is used, and whether it supports or contradicts the classification.

With the explainable algorithm and medical professionals in the team, an XAI-based human-in-the-loop system was established during the development. The process started when training and evaluating an initial classifier model on the burn ultrasound dataset. Then, a mix of hand-selected and random classifications were analyzed by the explainer system—including classifications that turned out to be incorrect. The LIME-based explainer outputted a series of explanations in the form of saliency maps. These visualizations communicated to human experts what parts of the input are being utilized for any given prediction. This was most obviously useful to determine if the classifier is using incorrect parts of the image, leading to an untrustworthy classification. However, the XAI system was also useful to be a feature filter to get cross verification between machine and burn experts. The experts combined the generated explanations with their prior medical knowledge—such as that severe burns change the echogenicity of healthy skin—altering the physical texture of the anatomy. Further, the experts knew that physical textural features in ultrasound translate to statistical textural changes in the image.³¹ This implies that, in addition to key areas like skin layer transitions, a classifier utilizing this texture should have a nebulous saliency map, extracting features from across the image instead of focusing in on acute areas. The expert team observed dense saliency maps, including misclassified examples. This implies the CNN was not effective at learning texture from the dataset, likely because of the small number of samples. After the interaction between the explainer and the expert team, the necessary modifications in the feature set of the CNN were made. As discussed in the previous section, this was done by adding GLCM texture features—selected in the hope that they can assuage the “feature bottleneck” of texture and improve accuracy without requiring more data samples. Finally, the model was re-trained and re-tested to prove that the updated classifier leads to a good prediction. This human-in-the-loop system allows the design of a more robust medical diagnosis system.

LIME is designed around working with tabular data, so an algorithm is required to split the ultrasound frames into sensible chunks. Quickshift,³⁷ Felzenszwalb segmentation,³⁸ and a custom method of splitting by depth-wise bars of ten pixels were tried as splitting methods. This last method was designed to explicitly capture the variable of depth in the ultrasound scan. A total of 10,000 permutations were sampled and the LIME algorithm was run. LIME enabled the extraction of a variety of rich information from its outputs, including a heatmap, a visual overlay of the top K features, and a quantitative list of the LIME scores for each feature. In addition to LIME, the CNN model was also adapted to generate a pixel-wise saliency map using backpropagation on the input image³⁹ as an extra explanation modality.

Wound Segmentation with RGB Images

U-net,⁴⁰ a specialized auto-encoder ideal for medical image segmentation, was adapted by adding a batch normalization and dropout layer in each network block for this limited dataset. The model is shown in Fig. 2. Before training, input values were normalized within the [0,1] interval. The cross-entropy loss and the Adam optimizer were employed. Key hyperparameters included a 0.001 initial learning rate, 0.95 exponential learning rate decay, 0.2 dropout rate, and a batch size of 4.

FIGURE 2.

(Left) The modified U-net used in segmentation. As an image-to-image model, the U-net takes an RGB image as input and outputs a mask image, denoting what parts of the input correspond to burnt skin and what parts are background. (Right) The architecture of the U-net.

Ethics Review Statement

All animal experiments were performed in compliance with the protocols approved by the Indiana University School of Medicine Institutional Animal Care and Use Committee (SoM-IACUC) under protocol 21,147.

RESULTS

Depth Classification and XAI

Results are given in the lower left and lower right panels of Table I. The proposed system identified full-thickness burns 576 times, partial-thickness burns 518 times, superficial burns 494 times, and unburnt skin 519 times. Full-thickness burns were misclassified as partial 36 times and as unburnt skin 9 times. Partial-thickness burns were misclassified as full-thickness 103 times, as superficial 10 times, and as unburnt skin 109 times. Superficial burns were misclassified as full-thickness 143 times, as partial-thickness 22 times, and as unburnt skin 10 times. Unburnt skin was misclassified as full-thickness 47 times, as partial-thickness 19 times, and as superficial burns zero times. Of all predictions across both pigs, 12.77% were overestimates, predicting a more severe burn than the ground truth label, and 6.66% were underestimates, predicting a less severe burn than the ground truth label. The module used in this system was compared with several competing modules, including VGG16—another CNN architecture—and SVM RBF given only texture feature data. A grid search of commonly selected values was used to tune the cost and gamma values of the SVM, and the learning weight and weight decay of each neural network. Ablation was also conducted on the five statistical texture features extracted from the ultrasound frames. Contrast was the most effective of these features.

TABLE I.

Quantitative Results and Information about Our Experiments. (Upper Left) Results on overhead RGB burn image segmentation. Our customized U-net outperforms a stock U-net in global accuracy and intersection-over-union (IoU). (Upper Right) A description of our datasets used in both tasks. (Cv) indicates that cross-validation was used as described in the experimental section. (Lower) Results from our classifier, outperforming neural and non-neural baselines.

Segmentation Results					Datasets
Metric	Stock U-Net (mean, SD)		Custom U-Net (mean, SD)			Ultrasound frames		RGB images
Global Accuracy	0.795	0.112	0.843	0.106	Train/valid	10,085 (cv)		293
IoU	0.672	0.143	0.742	0.142	Test	10,085 (cv)		45
Classification Results Algorithm Comparison
Method	Features (img, tex)		Accuracy	Precision	Recall	F1 Score	Overest %	Underest %
RBF SVM	❌	✔	0.4101	0.4136	0.4101	0.3997	24.04	34.95
VGG16	✔	✔	0.7025	0.791	0.7244	0.7003	24.55	5.2
ResNet34	✔	❌	0.7473	0.807	0.7503	0.7421	9.24	16.03
Ours	✔	✔	0.8058	0.8254	0.8072	0.8	12.77	6.66

Underlined elements indicate the best values for the given metric and category.

Wound Segmentation

Results are given in the upper left panel of Table I. The system in this study also performed well on segmentation. Two common segmentation metrics are global accuracy—the accuracy of pixel-level classifications for each segmentation mask—and intersection-over-union, which is a measure of the extent of overlap between the predicted and target masks. In both metrics, the developed customized U-net does better than a stock U-net after six trials. Qualitative results were also collected and are displayed in Fig. 3. These samples were selected to show the robustness of the model to variable burn depths, lighting conditions, body parts, distances, and angles.

FIGURE 3.

(Top) Segmentation results from two burns. On the left is the original image, in the center is the ground-truth burn area, and on the right is the predicted masks from our U-net. Prediction is very close to target. (Bottom) The input (left) and output (right) of our classifier. The input includes B-mode ultrasound and TDI ultrasound. The output calculates a ∼100% chance of the input being full thickness. The green and red masked image is the explanation of the B-mode frame. Green regions support the full-thickness prediction and red regions slightly contradict it.

CONCLUSION

This work has introduced a multi-characteristic, multi-modality system for predicting burn depth using CNNs and XAI, as well as predicting segmentation maps of burn wounds which can be used to estimate burn surface area. The proposed framework is also a first step toward including burn surgeon experts “in the loop” for burn assessment when using AI based decision-making systems. In addition, the system would allow nurses, medics, and generalists to assess burn injuries independently and in the field. Such a system could be deployed in a tablet or a smartphone, which have embedded cameras and are compatible with USB handheld ultrasound scanners. In the case of a disaster or in austere environments, a lightly trained practitioner could conduct preliminary burn assessment independently. As a result, this system would enhance the accuracy of burn injury assessments for frontline medical personnel, facilitating the optimal allocation of limited resources to the wounded in need. Furthermore, it would assist frontline health care providers to evacuate patients to the most appropriate medical facilities for the best therapy, ultimately reducing mortality rates.

A limitation of the wound segmentation dataset is that the influence of race or skin pigmentation on segmentation accuracy could not be assessed, as the images were collected from various web sources without specific racial information. However, this is not the case for the depth classification dataset of this research, as ultrasound is not affected by skin pigmentation. While we recognize that the use of a porcine model and controlled thermal burns in the burn depth dataset may limit generalization, it is important to note that our dataset was carefully designed to ensure ideal data for our current study. As more diverse human ultrasound data becomes available, we will leverage transfer learning to enhance the system’s ability to generalize. In future work, this process will be adapted to different ultrasound modalities, with increased automation and explainability within the framework. It is necessary to expand the dataset to a more representative size, augment it with synthetic data produced by generative models, and increase the clarity of the explainer system to better assist users in burn characterization. Furthermore, human data collection is under progression to evaluate the system on human subjects.

SUPPLEMENT SPONSORSHIP

This article appears as part of the supplement “Proceedings of the 2022 Military Health System Research Symposium,” sponsored by the Assistant Secretary of Defense for Health Affairs.

FUNDING

This work was supported by the Office of the Assistant Secretary of Defense for Health Affairs under Award No. 6W81XWH-21-2-0030 and by the National Science Foundation under Grant NSF #2140612.

CONFLICT OF INTEREST STATEMENT

None declared.

CLINICAL TRIAL REGISTRATION

Not applicable.

INSTITUTIONAL REVIEW BOARD (HUMAN SUBJECTS)

Not applicable.

INSTITUTIONAL ANIMAL CARE AND USE COMMITTEE (IACUC)

All animal experiments were performed in compliance with the protocols approved by the Indiana University School of Medicine Institutional Animal Care and Use Committee (SoM-IACUC) under protocol 21,147.

INDIVIDUAL AUTHOR CONTRIBUTION STATEMENT

M.J.J. drafted the original manuscript. All authors reviewed, edited, and approved the final manuscript.

DATA AVAILABILITY

The data that support the findings of this study are available from the corresponding author, upon reasonable request.

INSTITUTIONAL CLEARANCE

Institutional clearance approved (or does not apply).