Machine Vision and Image Analysis in Anesthesia - Narrative Review and Future Prospects

Abstract

Machine vision describes the use of artificial intelligence to interpret, analyze, and derive predictions from image or video data. Machine vision–based techniques are already in clinical use in radiology, ophthalmology, and dermatology, where some applications currently equal or exceed the performance of specialty physicians in areas of image interpretation. While machine vision in anesthesia has many potential applications, its development remains in its infancy in our specialty. Early research for machine vision in anesthesia has focused on automated recognition of anatomical structures during ultrasound-guided regional anesthesia or line insertion; recognition of the glottic opening and vocal cords during video laryngoscopy; prediction of the difficult airway using facial images; and clinical alerts for endobronchial intubation detected on chest radiograph. Current machine vision applications measuring the distance between endotracheal tube tip and carina have demonstrated noninferior performance compared to board-certified physicians. The performance and potential uses of machine vision for anesthesia will only grow with the advancement of underlying machine vision algorithm technical performance developed outside of medicine, such as convolutional neural networks and transfer learning.

This article summarizes recently published works of interest, provides a brief overview of techniques used to create machine vision applications, explains frequently used terms, and discusses challenges the specialty will encounter as we embrace the advantages that this technology may bring to future clinical practice and patient care. As machine vision emerges onto the clinical stage, it is critically important that anesthesiologists are prepared to confidently assess which of these devices are safe, appropriate, and bring added value to patient care.

INTRODUCTION

As artificial intelligence (AI) and machine learning (ML) progressively augment our personal lives, it may not be long before anesthesia providers routinely use AI- and ML-powered decision support techniques during delivery of patient care. These technologies are already extensively used in clinical radiology,¹ ophthalmology,² and dermatology,³ where they provide high-performance improvements in diagnostic accuracy, especially in image interpretation. In some cases, these image interpretation applications equal or exceed the performance of specialty physicians.

AI algorithms, or models, perform data interpretation and produce predictions with minimal additional human input. They use large datasets, such as those extracted from electronic health records (EHRs), to produce mathematical models that can prospectively interpret complex, non-linear relationships between factors that may influence patient care. Such algorithms are designed to predict a particular outcome, for instance advising the perioperative team of the volume and type of blood product to pre-operatively order for an individual patient.⁴ The benefit of AI and ML to expert clinicians is that of decision support—the models provide automated, precise, and patient-specific evaluations of complex information that can be used to complement clinical judgement.

Machine vision (MV) can use AI to analyze and derive predictions from image or video data. The process may begin by detecting or acquiring an image, which is then processed, classified, analyzed, and interpreted by computer algorithms (Figure 1). Several successful uses of MV in clinical medicine outside of anesthesia currently outperform clinical experts in image interpretation.^5,6 The use of MV to augment the use of ultrasound (US) imaging is an active area of research, especially in fetal medicine.⁷ MV has potential to aid anesthesiologists in multiple areas of perioperative care (Figure 2). Imaging data can also be combined with non-image data, such as patient demographics, past medical history, surgical factors, to create hybrid ML models. Detailed summaries of AI⁸ and MV⁹ in medicine have previously been published.

Figure 1.

Schematic diagram of a real-time machine vision system. Abbreviation: ML, machine learning; MRI, magnetic resonance imaging; CT, computed tomography.

Figure 2.

Stages of perioperative journey with the potential to be enhanced by machine vision. Abbreviations: US, ultrasound; IV, intravenous; ECG, electrocardiograph; CXR, chest radiograph; ETT, endotracheal tube.

Compared to radiology, machine vision for anesthesia remains in its infancy, but current applications under investigation include automated recognition of anatomical structures during ultrasound-guided regional anesthesia or line insertion; recognition of the glottic opening and vocal cords during video laryngoscopy; prediction of difficult airway using facial images; and clinical alerts for endobronchial intubation detected on chest radiograph.

In this article we review the process and concepts underlying machine vision in peri-anesthesia practice, explain frequently used terms, and provide a brief overview of techniques used to create machine vision applications. We also summarize published examples of MV for clinical practice and discuss the challenges that lie ahead for future integration of MV into anesthesia clinical workflow and its regulation.

MACHINE VISION TECHNOLOGY

The core terminology used to describe AI, ML, and MV differs from that used in traditional biostatistics. Table 1 summarizes useful AI, ML, and MV terminology.

Table 1. -.

Useful Terms in AI, ML, and Machine Vision

Term	Description
Domains
Artificial Intelligence (AI)	Any intelligence demonstrated by machines that are typically attributable to humans.
Deep Learning (DL)	A subset of ML leveraging a multi-layer neural network architecture.
Machine Learning (ML)	A subset of AI in which a machine improves with additional iterations without the need for explicit programming.
Techniques
Classification	A SL technique in which the target feature is a categorical variable.
Clustering	An UL technique in which data is iteratively grouped until data within each group are more similar than those outside the group.
Regression	An SL technique in which the target feature is a continuous variable.
Supervised Learning (SL)	An ML task in which data has a target feature(s) and the model seeks to learn the data distribution from this/these features.
Transfer Learning	A technique in which a model trained on one dataset is retrained on another dataset.
Unsupervised Learning (UL)	An ML task in which data in not labeled and the model seeks to learn the distribution from the data characteristics.
Architecture
Artificial Neural Networks (ANN)	A model composed of artificial neurons (nodes) consisting of an input layer, one or more hidden layers and an output layer.
Autoencoder	A model composed of an encoder that converts input features to latent space representation and a decoder that converts latent space representation to input features.
Convolutional Neural Networks	An ANN in which input features are combined together before being processed by the hidden layers.
Data
Categorical Variable	A variable that measures membership in a different group. Generally, has a fixed number of values.
Class Imbalance	The ratio of Majority to Minority class(es).
Continuous Variable	A variable taking on an infinite range of values.
Input Features	Set of predictive features used as the model input.
Majority Class	The most frequent class in classification.
Minority Class	The least frequent class(es) in classification.
Target Feature	The feature of interest to be predicted.
Test Set	Data used to measure the generalizability of the model. Generally, consists of 20%-50% of the original data.
Training Set	Data used to train a model. Generally, consists of 50%-80% of the original data.
Validation Set	A subset of the training set used to evaluate the model after each iteration. Normally, consists of 10%-20% of the Training Set.
Machine Vision
Bounding Box	A box of fixed sized used to describe the spatial location of an object within an image/video. One or more bounding boxes are used to scan an image in object detection.
Data/Image Augmentation	The process of modifying an image (rotating it, blurring it, changing the scale, etc.) to improve model generalizability and prevent overfitting.
Object Detection	A technique for locating objects present within an image or video.
Saliency Map	Identifies parts of the images based on contribution to the activation in a layer. Helps to explain model decisions by showing what parts of the image contributed the most to the output.
Synthetic Data	Artificial data generated using real data to improve/enhance data available for training.

Neural networks (NNs) serve as the foundation for deep learning architectures, such as those used in MV.¹⁰ They are composed of artificial neurons (nodes) consisting of an input layer, one or more hidden layers, and an output layer—an arrangement very loosely modeled on a primitive nervous system. Each node takes an input from the previous layer, analyzes it, adds further processing, and sends it on to the next step. The advantage of this approach is that the complex non-linear relationships frequently encountered in medical data are more effectively modeled. Convolutional neural networks (CNNs), one variant of NNs, use a filter¹¹ to combine multiple adjacent input values to better capture features unique to geometric objects, such as edges, and act as a form of regularization to prevent overfitting. CNNs have been widely used for image and video processing and represent a standard approach in the machine vision problem space, including in the medical domain.⁹ Developing CNNs requires a large amount of domain specific data (i.e., images), which can be a limiting factor in medicine. An accessible video for understanding more about these techniques is published by MathWorks.¹²

Transfer learning is a promising approach to overcoming limitations of dataset size. Models previously trained on large datasets in one domain can be retrained on smaller datasets in a different domain. With the advent of ImageNet in 2009, a database of over 15 million hierarchically organized images with annotations, training data for CNNs became readily available.¹³ Since 2010, this dataset has been used as the foundation for the annual ImageNet challenge, in which participants competed to build image classification models leveraging the accompanying large, opensource dataset.¹⁴ This has resulted in the release of multiple high-performance models that are available for transfer learning in the image processing domain. The first of these, AlexNet, represented a monumental step forward in applying CNNs to image processing tasks by producing an error rate of 15.3% compared to 26.2% for the next closest model at the time.¹⁵ GoogLeNet produced another leap forward, reducing this error to 6.7%.¹⁶ GoogLeNet was based on Inception architecture, which was notable for changes to the CNN that improved performance while keeping the processing resources constant. The notable trend is the increasing depth of the networks and reduction in error rates accomplished without substantial increase in computational resources. A limitation of this method is that the ImageNet database contains very few medical images, such as ultrasound views and radiographs. However, a MV model that has been trained on millions of everyday objects can then be further trained to efficiently recognize, for example, the endotracheal tube tip and carina on chest radiographs.

After a MV model is fully developed, its performance is assessed using a testing set of data held out from the full dataset for this purpose. Many metrics are available for this purpose, some specific to the type of model or the problem being addressed. A standardized set of reporting metrics is not yet available in AI for medicine, and this means that authors choose the metrics to report on their models. Commonly used metrics are defined in Table 2.

Table 2.

Metrics Commonly Used to Assess Performance in Machine Vision Technology

Metric	Description
Accuracy	Correctly predicted instances over total instances.
True Negative (TN)	Total number of negative classes correctly predicted.
True Positive (TP)	Total number of positive classes correctly predicted.
False Negative (FN)	Total number of positive classes incorrectly classified as negative.
False Positive (FP)	Total number of negative classes incorrectly identified as positive.
Sensitivity (true positive rate, TPR)	Fraction of true positive values that are predicted as positive. TP / (TP + FN)
Specificity (true negative rate, TNR)	Fraction of true negative values that are predicted as negative. TN / (TN + FP)
Positive Predictive Value (PPV, Precision)	Fraction of predicted positive values that are true positive. TP / (TP + FP)
Negative Predictive Value (NPV)	Fraction of predicted negative values that are true negative. TN / (TN + FN)
F1-Score	The harmonic mean of precision and sensitivity. 2 x (PPV x TPR) / (PPV + TPR)
Receiver-Operating Characteristic (ROC) Curve	Plot of sensitivity (TPR) vs 1 – specificity (False positive rate).
Area Under ROC Curve (AUROC)	Integrate of the ROC Curve, bounded within [0, 1], measuring the overall performance of the model. An AUROC = 0.5 indicates performance consistent with random guessing.
Confusion Matrix	Matrix showing TP, FP, TN, and FN.
Mean-Squared Error	The average error measured over the entire sample, $\frac{1}{\mathrm{n}}\sum _{\mathrm{i}=1}^{\mathrm{n}}{({\mathrm{y}}_{\mathrm{i}}-\widehat{{\mathrm{y}}_{\mathrm{i}}})}^2$, where n = number of samples, yi is the true value, and $\widehat{{\mathrm{y}}_{\mathrm{i}}}$ is the predicted value.
Logloss	−(ylog(p) + (1+y)log(1-p)), where p = Pr(y=1). A measure of the probability that an input belongs to the true label.
Intersection Over Union	In machine classification, a measure of how closely the model-identified object box matches the labeled box. Calculated as area of overlap divided by area of union.
Mean Average Precision	Average precision for each class averaged over the number of classed.
Saliency Map	Identifies parts of the images based on contribution to the activation in a layer. Helps to explain model decisions by showing what parts of the image contributed the most to the output.

METHODS

A literature search was performed to find articles of interest on 12/12/2022. PubMed was searched for articles published in the last ten years using the strategy provided in Supplemental Digital Content 1. A total of 399 unique articles were screened, 66 underwent full-text review and 30 were selected by the authors to meet the objective of reviewing recently published and notable works relevant to the fields of anesthesiology and machine vision.

APPLICATIONS OF MACHINE VISION IN ANESTHESIA

Airway Management

Machine vision has been used to assist in identification of patients with increased risk of difficult airway.^17–19 Otherwise apparently normal patients can present with difficultly in mask ventilation and/or endotracheal intubation after induction of anesthesia, with an overall incidence of around 5.8%.¹ A meta-analysis of commonly used bedside screening tests²⁰ found that each method individually had poor to moderate sensitivity and moderate to fair specificity. A combination of Mallampati classification and thyromental distance performed best, with an area under receiver operating characteristic curve (AUROC) of 0.84 and a positive likelihood ratio of 9.9.²⁰ This performance can be used as a reasonable comparator for the assessment of MV models developed to predict risk of difficult airway.

Several different definitions of difficult airway have been used in the literature.²¹ This poses a problem for MV, where the accurate definition of an output ground truth is essential to the development of a clinically useful model. The most common definition of a difficult intubation (DI) is Cormack-Lehane (CL) grade III or IV.²⁰ In addition, DI is a rare event, and failed intubation even more rare.²² This therefore leads to a significant class imbalance for machine vision models. Class imbalance poses significant challenges for machine learning since the majority class is much easier to learn than the minority class. Yet, in clinical diagnosis and practice, it is often the minority class that represents the class of interest. A large class imbalance can thus result in a model that may appear to perform well in terms of raw accuracy, but that is actually unable to predict the minority class (e.g. difficult intubation). One possible solution is to enrich the dataset with examples of DI with an incidence greater than one would encounter in clinical practice in order to establish a more balanced set of classes. No machine vision studies that predict difficult airway beyond difficult intubation were found at the time our searches were performed.

Connor et al.(2011)¹⁷ used front and side face images of Caucasian male patients with a MV facial recognition system to predict intubation as “easy” or “difficult.” Easy was defined as a single attempt with a Macintosh size 3 blade and CL grade I view. Difficult was defined as one of the following: >1 attempt by an operator with at least one year of anesthesia experience; CL grade 3 or 4; need for second operator; non-elective use of alternative airway device. Their technique created a mathematical model of the face in a semi-automated process that took 15 minutes per patient and required the manual placement of fiducial markers on the image to guide reconstruction. They used 61 variables—59 facial proportions plus thyromental distance and Mallampati class—as inputs to their MV models. The models performed well, with AUROC of 0.89, sensitivity 0.9, specificity 0.85, and positive predictive value (PPV) 0.86. Interestingly, Mallampati score did not affect the final model, nor add further predictive information. These excellent results may have been achieved because the group reduced the variability of their dataset by limiting their subject groups to either “easy” or “difficult,” excluding airways of intermediate difficulty. A further reduction in variability was achieved by including only Caucasian males. The model may therefore not perform as well in routine clinical practice with a heterogenous group of patients. The limitation of datasets to certain gender or racial groups raises ethical concerns for the development and future implementation of AI in medicine.²³ To assess real-world performance of the model, a validation set should include both males and females of multiple racial groups and all levels of intubation difficulty. It is currently unknown if anesthesiologists would accept the use of clinical decision support tools, especially those based on AI/ML, during their routine bedside airway assessments.

Cuendet et al.¹⁸ developed an automatic facial analysis approach to detect morphological traits related to DI and improve its prediction. The group defined DI as CL grade 3-4 or Intubation Difficulty Scale (IDS) >5. Easy intubation was when IDS=0 (equal to CL grade 1, plus first-time success). Intermediate intubations were all the remaining patients, IDS >1 to ≤5 and CL <3. The system took multiple facial motions, including neutral face, mouth open, sticking out tongue, neck rotation, and extension. The dataset contained 2725 scanned patients, but documented intubation ground truth information in the form of IDS and/or CL grade was only available for 970. This could introduce selection bias, as DI may have more detailed documentation than straightforward intubation. Class imbalance was overcome by adjusting the weighting of type of error. The team first developed a classification model to predict easy vs. difficult intubation in a similar fashion to the study by Connor et al.,¹⁷ achieving AUROC 0.81. They then developed a second model for real-world prediction of DI in all patients. AUROC fell to 0.779, still reasonable performance but an illustration of the challenge to MV with increased patient variability.

This study assessed additional elements of intubation, including mouth opening and pharyngeal tissue, and the fully automated process—at one second to take each image and 30-40ms processing time—was much faster than that of Connor et al.¹⁷ The specialized camera equipment would however limit use in routine clinical practice. The dataset contained patients from a broader range of ethnic and racial groups than in the study by Connor et al.,¹⁷ but a detailed breakdown was not provided. Interestingly, the team also recorded voice and facial depth maps, intending to add novel modalities to future model development.

Tavolara et al.¹⁹ used a transfer learning approach, first training multiple CNNs on a large, publicly available dataset with 494,414 facial images from 10,575 celebrity subjects. They then used a smaller set of patient images to classify patients as difficult or easy to intubate using a similar definition of DI as the other studies in this section. No demographic information was provided. The AUROC of 0.71 out-performed Mallampati and thyromental distance bedside tests but did not outperform a combination of the two. The authors suggest that model performance could be improved by inclusion of side facial images.

An ideal machine vision model for DI needs to gather inputs either from routinely collected data or from methods that are as quick and easy as current bedside predictive tests. It must then produce an accurate prediction with good discriminatory ability—a high PPV—and provide transparency to engender trust from end-users.

A challenge to the development of a predictive algorithm using ML is that the current Mallampati and thyromental distance bedside tests can be done by expert practitioners in a matter of seconds with no specialist equipment. In combination, these tests demonstrate a similar AUROC on a full range of patients when compared with any of the MV models in this section. MV models might better predict difficult airway if trained on data from additional imaging modalities, such as ultrasound or magnetic resonance imaging. This is not currently an appropriate sole indication for imaging, but these models may be useful for patients who have undergone recent head and/or neck scanning for another reason.

Assistance with Intubation

The achievement of intubation when it is difficult, whether anticipated or not, is a root problem in anesthesiology. Videolaryngoscopes are now widely available in clinical environments to assist with difficult intubation. However, the presence of blood, airway secretions, fog, and technical difficulties such as poor focus can limit their usefulness, especially for inexperienced operators. A decision support system to guide clinicians experiencing these conditions is not yet clinically available.

Machine vision can be used to identify airway anatomy when it is not clearly visible during video or fiberoptic laryngoscopy.^24–26 Carlson et al. performed a manikin-based proof-of-concept study to test the accuracy of machine vision in identifying the glottic opening based on video.²⁴ The best performing model demonstrated accuracy of 81%, sensitivity 70%, and specificity 90%. No AUROC figures were reported. The authors noted that their paper was an initial effort that may eventually lead to a tool that could aid successful intubation.

A further step was taken by Kim et al. who developed a CNN to indicate the location of the glottis in airway images captured by video and fiberoptic laryngoscopy²⁵ during emergency room intubations and otolaryngology examinations. The views were sub-grouped as “good” (n=981) if CL grade 1 or 2a and “bad” (n=219) if the glottis was obscured by CL grade 2b, 3, or 4, poor light, secretion, fog, or poor focus. For “bad” views the approximate location of the glottis was marked by referring to surrounding anatomy. Their best model correctly identified the glottis in 74.5% of the testing data. For identification of the glottis in “bad” views—the most useful clinical application— the accurate prediction rate decreased to 53.1%. This may be because “bad” views provide greater variability of data and the dataset contained fewer samples from this sub-group. The models therefore appear to need a much larger sample size to improve their accuracy for “bad” views. Given that the images were not obtained during routine anesthesia and the dataset contained solely Asian subjects, a broader dataset is needed to validate the models for widespread clinical use. Comparison of a high-performing model with the success of expert practitioners would further demonstrate the clinical utility of using MV to assist with intubation.

Matava and colleagues used a dataset of 775 static images derived from pediatric video laryngoscopies and bronchoscopies to train three CNNs to detect, classify, and label vocal cords and tracheal rings in real time.²⁶ They also utilized transfer learning and developed models with excellent real-time performance on live video feeds, reporting specificities of >0.97 and sensitivities of >0.86.

To improve model performance, we could more accurately define DI rather than using the subjective proxy of CL grade. MV models could be trained either to automatically assess CL grade during videolaryngoscopy or perhaps even to define DI for themselves using an unsupervised ML algorithm. We could also use image inputs in combination with other patient information. The key remains, as with the training of any supervised MV or ML model, in having a large set of labelled, high quality, and varied data.

Although machine vision for assistance with intubation has yet to demonstrate the ability to outperform experienced practitioners in a clinical environment, initial reports show promise.^24–26 This work could lead to a semi-automated design where the MV-guided videolaryngoscope could give direction to the user for optimal positioning to facilitate glottic view. Even further into the future, perhaps the MV model could identify the glottis and give direction to a robotic videolaryngoscope that would correctly position itself and introduce a guidewire, bougie, or endotracheal tube (ETT) with minimal or no human assistance. When integrated with algorithms to detect correct endotracheal placement, this could lead to increased safety and efficiency for DI.

Assessment of Endotracheal Tube Position

Perhaps inspired by the success of machine vision in radiology, over the past two years an explosion of papers report the development of machine vision algorithms to determine appropriate ETT position on anteroposterior chest radiographs.^27–33 These projects aspire to postoperative clinical utility in the intensive care unit but could also be used intraoperatively if chest radiographs are taken.

Early projects used MV algorithms to solve easier problems—differentiating between abdominal and chest radiographs, or the presence or absence of an ETT.²⁷ Despite their simplicity, these algorithms have utility in reducing computational demand by selecting appropriate images for the application of more complex ETT positioning models.

More advanced models were developed to detect both the carina and ETT tip to measure the distance between them and to highlight when that distance is outside clinically acceptable bounds, including when the ETT is positioned in a bronchus.^28–30 Several groups compared the performance of the algorithm to that of healthcare providers, including board-certified radiologists^28,29 and intensivists.³⁰ All demonstrated excellent interrater agreement or non-inferiority of model vs. physician.

Many papers report best results using transfer learning with pre-developed CNN-based algorithms, such as U-Net²⁹ and Inception V3,³¹ that are pre-trained on the ImageNet dataset. Some used “weak” labelling—using the entire image for category labeling—but others reported better results from “strong” labelling,³² either by hand-drawn bounding boxes²⁸ or by first applying a CNN to estimate the location of the carina and then crop the radiograph image to the proximal trachea and bronchus (dimensionality reduction) before applying a second, more specialized algorithm to refine the location and detect the ETT tip.³³ Natural language processing, a text-focused branch of AI, was also used to collect datasets with appropriate images by combing radiology reports for presence or absence of ETT.^28,33 The use of class activation or saliency maps helps both developers and users appreciate that the algorithm is focused on the clinically appropriate section of the image, rather than being driven by spurious information.³¹

Only one study evaluated their model for future clinical application. Brown et al. developed a model to identify if the ETT tip was observed outside a “safe zone” of 3 to 7cm above the carina and then generate an ETT misplacement alert.³⁴ Their algorithm was able to recognize 95% (271/285) of ETTs, 86% of which (233) were appropriately placed. The generated alert messages had a PPV of 83% (265/320) and NPV of 98% (188/192), detecting 40/42 misplaced ETTs in the dataset.

Challenges specific to this domain include negating the effect of distracting artifact from, for example, feeding tubes and sternal wires that can cause difficulties for the algorithm in localizing the ETT tip. In addition, an accurate ground truth may be difficult to ascertain, as interrater agreement between physician interpretations can be significant. Despite these challenges, detection of misplaced ETTs may be one of the first applications of MV used in clinical anesthesia practice as a screening and early alert system applied immediately after the radiograph is taken and before clinical review of the image. Furthermore, the algorithms could be used to automatically populate ETT presence and insertion depth on radiology reports and critical care documentation. Similar uses of MV have been reported for central venous catheter³⁵ and feeding tube placement³⁶ on chest radiographs.

Regional Anesthesia

The utility of ultrasonography for the practice of anesthesiology continues to grow. The advantages of using ultrasound for neuraxial, regional anesthesia, and vascular access are well described.³⁷ However, recognizing and interpreting sonoanatomy remains an expert skill. Low resolution, image artifacts, and infrequent use of ultrasound pose continuing challenges to providers. Investigation into the use of machine vision to enhance the identification of neural and vascular structures is underway.^38,39 Giraldo et al. investigated the role of machine learning in image processing to improve the accuracy of peripheral nerve identification.⁴⁰ Their model was able to identify and differentiate the ulnar, median, and peroneal nerves. This model was also able to differentiate other soft structures, like muscles and aqueous tissues, that might be useful for an anesthesiologist.

Gonzalez et al. also developed a machine vision model to automatically predict and label ultrasound pixel clusters as “nerve” or “background,”⁴¹ demonstrating an accurate performance for segmentation and identification of peripheral nerves. Hadjerci et al. have also been able to demonstrate an accuracy of 89% of the F-score in identifying peripheral nerves.⁴²

Needle tip and entry site localization remain a challenge for central neuraxial blocks.⁴³ Shuang et al. developed an image classifier that differentiates the bone and interspinous region in transverse plane ultrasound images of the lumbar spine, successfully identifying the proper needle insertion site in 45 out of 46 pregnant patients.⁴⁴ .In similar yet unrelated work, a multi-scale and multi-directional feature extraction technique was used to identify the appropriate plane for performing spinal anesthesia.⁴⁵ Pesteie et al. were able to successfully identify ultrasound targets for epidural steroid injection⁴⁵ and facet joint injection using local directional Hadamard feature classification.⁴⁵

Realtime performance will assist with the growth in popularity of machine vision in clinical anesthesia. Smistad et al. were able to demonstrate average runtimes of 38, 8, 46, 0.2 milliseconds for artery detection, tracking, reconstruction, and registration methods, respectively, for a machine learning algorithm to assist with real-time guided local anesthetic block of the femoral nerve.^46,47 Hetherington et al. were able to demonstrate the real-time use of a machine-learning algorithm to identify the vertebral level from a sequence of ultrasound images.⁴⁸ Realtime performance of machine learning models during actual clinical care will be important. Latency due to data pre-processing, inference and graphical depiction of results will need to be minimal to allow adoption and acceptance among anesthesiologists.

Recently, regulatory authorities have approved the clinical use of devices that incorporate MV for nerve identification during regional anesthesia.⁴⁹ The ScanNav Anatomy Peripheral Nerve Block system includes several ML algorithms, each dedicated to a particular block region. ScanNav applies a color overlay on real-time ultrasound images, assisting with the correct identification of key anatomical structures. Non-experts have reported the system useful during both clinical practice and training.^50,51 Other MV systems under evaluation include NerveBlox^52,53 (SmartAlpha, Turkey), and NerveTrack^™54 (Samsung Medison, South Korea). Randomized clinical trials are required to determine the benefit from the use of any MV based system for regional anesthesia, and as of publication are yet to report.

DISCUSSION: CHALLENGES IN DEVELOPMENT AND ADOPTION OF MACHINE VISION FOR ANESTHESIA

Medical imaging is well-suited to deep learning because clinicians undergo intensive training to learn—in ML terms—to map from a fixed input space (i.e., clinical imaging) to an output (i.e., diagnosis). Automated performance of some clinical tasks would gain time for the clinician to use elsewhere. The ability to interpret images with the accuracy of a board-certified physician could also expand access to vital medical care.

Both successes and challenges have been found in applying deep learning to medical imaging tasks. For example, to develop a model for detecting diabetic retinopathy, researchers must hand-label target features onto each retinal image.⁶ This method is time-consuming and can lead to information loss. If humans do not identify small subtleties in the images, the labeling information will not include this information and the MV model will not recognize it. Another fundamental challenge is that training datasets usually only contain an image tagged with a label, without explicit definitions of features. Because the model learns only from features available in the dataset without wider context, the algorithm may use features unrecognizable to humans, which may be irrelevant to the task.⁵⁵

A physician makes clinical judgements based on broad contextual factors to which a MV model does not have access. Another apparent challenge related to the contextless nature of MV models is that models fail to identify similar-but-different conditions. The MV diabetic retinopathy model may miss pathology of other etiologies because that pathology differs in appearance to diabetic retinopathy and so the model does not recognize it as abnormal. Hence, algorithms are not yet a replacement for a physician providing a comprehensive ophthalmological consultation. Therefore there is a need to create expansive multivariate models that allow an algorithm’s prediction to incorporate patient risk profiles, with future studies developing a richer patient representation by incorporating features from EHRs and beyond.⁵⁶ In an early example of the potential of this technology, Isikli Esener et al. demonstrated that their novel proposed multistage classification scheme is more effective than the single-stage classification for computer-aided breast cancer diagnosis.⁵⁷

Capturing images to create a repository for training, testing, and validation carries challenges. With a vast range of photographic conditions and storage formats, there comes a risk that, in the data acquisition process, the model will base predictions on anomalies rather than on salient patient features and thus will make spurious predictions. It can be beneficial for the ML process to downscale images from full to low resolution, especially when using networks pre-trained at low resolution. This process is an example of dimensionality reduction, which is used in CNN development to constrain the image feature space.⁵⁸ These techniques can lead to loss of salient details in the image that may reduce the model’s performance.

Publicly available datasets are frequently large and come without a need to prospectively acquire new images, making them popular sources of data for the development of MV models. These datasets may have inherent sampling biases, as they do not contain image acquisition specifications or hospital process variables. There is a need to develop more intricate methods to mitigate the influence of non-biological signal in these resources.⁵⁹

Small and limited image datasets in healthcare bring another challenge. Although this problem can be improved by using transfer learning and data augmentation techniques, a larger sample of patients is needed to help draw the most robust conclusions. Even studies that use large and complex datasets recommend accessing significantly larger datasets or populations with more events to increase the performance of deep learning models.^5,6 Probably the most important hurdle to overcome is the availability of large, high-quality, diverse, granular datasets obtained through ethical means and with full consent of patients. Building these datasets will take multi-disciplinary partnerships with patient representatives, data professionals, ethicists, and subject matter experts.

There is a critical and currently unmet need to validate and determine the feasibility of MV algorithms in the real-world clinical workflow setting across a full spectrum of case presentations. If real-world model capability is established, clinical trials are then needed to determine if the use of MV models leads to improved care and outcomes compared with current practice. Testing should also include a comparison of model and physician performance to demonstrate both safety and value added by the model to the clinical workflow.

Many challenges to MV in anesthesiology are common to the safe introduction of all AI to our specialty. The technical aspects of model-building and computing power will continue to foster new decision-support solutions.⁵⁶ The future education of clinicians must cover aspects of AI that will allow them to appraise and appropriately use these new tools.⁶⁰ This understanding will also aid acceptance into practice and accelerate widespread clinical use.⁶¹ Finally, deep learning methods have been described as “black boxes.” Especially in medicine, where accountability is important and can have serious legal consequences, it is often not enough to have a good prediction system. This system also must articulate itself in a certain way. Several strategies have been developed to understand what intermediate layers of convolutional networks are responding to.^62–64 Other researchers have tied prediction to textual representations of the image (i.e., captioning),⁶⁵ which is another useful avenue for understanding what a network perceives.

Assessing model generalizability, or the model’s ability to adapt correctly to new, previously unseen data drawn from the same distribution as the one used to create the model, is essential and is achieved by testing it on a new, unseen, gold standard dataset. The regulations that are necessary to avoid systematic errors and performance fluctuations that could harm patients are currently being developed by authorities across the world, including the United States Food and Drug Administration (FDA).⁶⁶

A contemporary clinical anesthesiologist is unlikely to encounter MV-driven decision support systems in daily practice, but over the next 5 to 10 years new uses of this technology will emerge on the clinical stage. It is critically important that, as a specialty, we are prepared with sufficient background knowledge to confidently assess which of these new devices are safe, appropriate, and bring added value to patient care.

Table 3.

Demonstrated Ultrasound Guided Regional Anesthesia Applications of Various Machine Vision–Based Systems

Block category	Peripheral nerve block	ScanNav	NerveBlox	NerveTrack ™
Upper Limb Blocks	Interscalene-level brachial plexus block	√	X	?
	Supraclavicular-level brachial plexus block	√	X	?
Truncal Blocks	Axillary-level brachial plexus block	√	X	?
	Erector spinae plane block	√	X	?
	Rectus sheath block	√	X	?
	Suprainguinal fascia iliaca block	√	X	?
Lower Limb Blocks	Transversus abdominus plane block	X	√	?
	Adductor canal block	√	X	?
	Popliteal-level sciatic nerve block	√	X	?

GLOSSARY

AI

artificial intelligence

ANN

artificial neural network

AUROC

area under receiver operating characteristic curve

CL

Cormack-Lehane

CNN

convolutional neural network

CXR

chest radiograph

DI

difficult intubation

ECG

electrocardiograph

EHR

electronic health record

ETT

endotracheal tube

FDA

United States Food and Drug Administration

IDS

intubation difficulty scale

IV

intravenous

ML

machine learning

MV

machine vision

NN

neural network

NPV

negative predictive value

PPV

positive predictive value

SL

supervised learning

UL

unsupervised learning

US

ultrasound