Harnessing the Power of Artificial Intelligence to Teach Cleft Lip Surgery

Background:

Artificial intelligence (AI) leverages today’s exceptional computational powers and algorithmic abilities to learn from large data sets and solve complex problems. The aim of this study was to construct an AI model that can intelligently and reliably recognize the anatomy of cleft lip and nasal deformity and automate placement of nasolabial markings that can guide surgical design.

Methods:

We adopted the high-resolution net architecture, a recent family of convolutional neural networks–based deep learning architecture specialized in computer-vision tasks to train an AI model, which can detect and place the 21 cleft anthropometric points on cleft lip photographs and videos. The model was tested by calculating the Euclidean distance between hand-marked anthropometric points placed by an expert cleft surgeon to ones generated by our cleft AI model. A normalized mean error (NME) was calculated for each point.

Results:

All NME values were between 0.029 and 0.055. The largest NME was for cleft-side cphi. The smallest NME value was for cleft-side alare. These errors were well within standard AI benchmarks.

Conclusions:

We successfully developed an AI algorithm that can identify the 21 surgically important anatomic landmarks of the unilateral cleft lip. This model can be used alone or integrated with surface projection to guide various cleft lip/nose repairs. Having demonstrated the feasibility of creating such a model on the complex three-dimensional surface of the lip and nose, it is easy to envision expanding the use of AI models to understand all of human surface anatomy—the full territory and playground of plastic surgeons.

Takeaways

Question: Can an AI algorithm be trained to intelligently and reliably recognize the anatomy of cleft lip and nasal deformity and automate placement of nasolabial markings to guide surgical design?

Findings: An AI algorithm was generated that identifies 21 surgically important anatomic landmarks for designing a unilateral cleft repair. Normalized mean error (NME) was calculated for each point, and values were between 0.029 and 0.055, well within standard AI benchmarks.

Meaning: We successfully developed an AI algorithm that can be used alone or integrated with projection-based augmented reality to guide cleft lip/nose repair. This model can be expanded to all of human surface anatomy.

INTRODUCTION

Cleft lip affects one in 700 children globally and is more prevalent in endemic regions. In underresourced areas, cleft burden often surpasses regional capacity to deliver cleft care.¹ In a previous study, the authors used augmented reality (AR) to create a remote yet “hands-on” virtual interactive presence to engage overseas cleft surgical learners.² This AR-based technology platform was consistently used as an adjunct to in-location training to significantly accelerate the transfer of cleft knowledge and skill while building sustainable regional capacity for cleft care.² Although AR plays a central role in remote surgical guidance, it is heavily reliant on the live presence of a remote expert. Artificial intelligence (AI) leverages today’s exceptional computational powers and algorithmic abilities to learn from large data sets and solve complex problems. AI reads, understands, processes, and interprets these data while symbiotically learning from human expertise and real-world scenarios to improve predictive computational capabilities. Thus, AI-driven platforms can augment human capabilities by providing impactful insight and support during medical/surgical diagnosis and decision-making.³

These characteristics make AI ideally suited to play a role in teaching cleft surgery, a procedure defined by both precise anatomic and mathematical relationships and ongoing human experience.³ In cleft lip and nose repair, over 20 anatomic points must be accurately identified and marked along the richly three-dimensional surface of the infant’s small nasolabial region (<10 cm²). Using these important landmarks, several variations in surgical design are possible based on surgeon preference and experience. The aim of this study was to construct an AI algorithm that can intelligently and reliably recognize the anatomy of cleft lip and nasal deformity and automate placement of nasolabial markings that can guide surgical design.

MATERIALS AND METHODS

To develop the AI model for placement of cleft anthropometric markings, we adopted the high-resolution net architecture, a recent family of convolutional neural networks–based deep learning architecture specialized in computer-vision tasks. This architecture has previously been used as the backbone to accomplish tasks, such as object detection, image classification, pose estimation, and even facial landmark detection.⁴ The biggest limitation of these networks is the requirement of large data sets needed to train the algorithm. Given the difficulty in acquiring such quantities of cleft lip images, we first utilized the process of “transfer learning” in which the AI algorithm learns how to detect some anthropometric markings on noncleft photographs to create a general facial detection model. The model is then trained using cleft images with digitally marked anthropometric landmarks. Before training our model, we implemented the standard practice of “augmenting” our dataset. This is a technique to improve the robustness of the model and create new training data from existing cleft images by generating mirror images of each picture.

To select the appropriate sample size needed to train and test our model, we ran experiments using existing facial recognition algorithms. The normalized mean error (NME) for training and testing cohorts was generally found to converge at around 300 images with minimal additional decreases in error appreciated with even four times this number. We, therefore, utilized 345 two-dimensional photographs of infants and children with unilateral cleft lip to train and test our AI cleft model. We divided the aggregate images into those used for training (80%) and those for testing (20%). At the supervision of the senior author (R.M.V.), training images were individually annotated for 21 well-established cleft anthropometric landmarks and points important during surgical design (Fig. 1). These hand-marked points were digitized and used to train our cleft AI algorithm. Nearly all images were provided by Global Smile Foundation, a not-for-profit international cleft outreach organization. Photographs were taken of the full face from a range of angles (frontal to submental). Informed consent was obtained from all individual participants (or a parent) for photography and use of images in research, abiding by the guidelines in the Declaration of Helsinki. Images represented a broad and diverse range of patient ethnicity [Hispanic (n = 245), African (n = 65), and Middle Eastern (n = 35)], assumed gender [male (n = 241) and female (n = 104)], cleft laterality [left (n = 206) and right (n = 139)], and cleft severity [complete (n = 312) and incomplete (n = 33)]. Individual photographs were not associated with numeric age; however, a broad range of ages was represented upon review of the photographs (infancy to adulthood).

Fig. 1.

Comparison of hand-marked versus AI-generated unilateral cleft lip markings. A, Right unilateral cleft hand marked by a fellowship-trained cleft. B, Incomplete unilateral cleft lip hand marked by the same cleft surgeon. C and D, AI-generated cleft markings for each of the corresponding hand-marked images. Anthropometric landmarks for unilateral nasolabial repair are indicated. For each point, c’ corresponds to cleft side and nc’ corresponds to noncleft side. Alare (al) is the most lateral point of each ala, and subalare (sbal) is the most inferior point of the alar base, the highest point of columella (c) lies atop each hemicolumella and is level with each nostril peak. Subnasale (sn) defines the angle between columellar base and upper lip, while pronasale (prn) is the point of maximal nasal projection. Crista philtri superior (cphs) is atop each philtral column at the same horizontal line drawn through sn, crista philtri inferior (cphi) lies at the base of each philtral column (each Cupid’s bow peak). m and l distinctions before these terms correspond to medial and lateral in relation to the cleft. Labiale superius (ls) lies at the midpoint of the upper vermilion (Cupid’s bow trough), stomion (sto) is the point along the vertical facial midline that bisects the free margin of the upper lip, and cheilion (ch) is located at each labial commissure. A triangular flap of vermilion (“Nordhoff’s triangle”) is defined by points (nt1) and (nt2), and the cleft side red line (c’rl) below cleft cphi. Relevant lip and nose measurements for unilateral cleft deformities include heminasal width (sn-al), nasal width (al-al), nasal tip projection (sn-prn), columellar length (sn-c, cphs-c), labial height (sn-cphi, sbal-cphi), and lip width (cphi-ch).

We next tested our algorithm. Each testing image was marked digitally by the senior author and automatically by our cleft AI algorithm. The two-dimensional coordinates of each of the 21 anatomic points generated by the AI algorithm ($\tilde{x},\tilde{y}$) were compared with the two-dimensional coordinates of the human-marked points (x, y). The precision of each point was computed by calculating the Euclidean distance $d=\sqrt[2]{{(x_i^k-x_i^{\sim k})}^2+{(y_i^k-y_i^{\sim k})}^2}$ between the human and AI-generated coordinates, normalized by d_norm (intraocular distance, IOD) to standardize for image size. Normalized error = ${\delta }_i^k=\frac{d\{(x_i^k,y_i^k),(x_i^{\sim k},y_i^{\sim k})\}}{\mathrm{I}\mathrm{O}\mathrm{D}}$.^5,6 The superscript k indicates one of the landmarks, and the subscript i is the image index. The normalized error for each point was averaged across the test cohort to obtain the NME for each anatomic point.

RESULTS

Our cleft AI model was trained to recognize and mark 21 anatomic points representing important anthropometric landmarks for understanding cleft nasolabial anatomy and for designing various types of nasolabial repair. For each point, the NME was calculated and is represented in Figure 2. All NME values were between 0.029 and 0.055. The largest NME was for cleft-side cphi point (Cupid’s bow peak). The smallest NME value was for cleft-side alare point. Our cleft AI model can mark 2D photographs and videos of patients with a range of cleft lip/nose severity (Fig. 1). (See Video [online], which displays how the AI algorithm places anatomic points during real-time video of patient with unilateral cleft lip. Anatomic points are generated and accurately placed over a wide range of viewing angles.) Additionally, our cleft AI model can identify cleft nasolabial landmarks over a wide range of viewing angles, as demonstrated in supplemental digital content 1. (See Video [online].)

Fig. 2.

NME of the 21 anthropometric landmarks in cleft surgery.

Video 1. This video displays the how the AI algorithm places anatomic points during real time video of patient with unilateral cleft lip. Anatomic points are generated and accurately placed over a wide range of viewing angles.

Download video file ^{(25.5MB, mp4)}

DISCUSSION

In Principalization of Plastic Surgery, the pioneering plastic and cleft surgeon Dr. Ralph Millard, Jr., writes, “Progress in plastic surgery requires innovations for improvement of the standard, modification of the routine, solution of problems, and pioneering of new frontiers.⁷” Today, the worldwide burden of orofacial clefting surpasses our capacity to deliver cleft care, leaving many with lifelong disfigurement and disability. While scores of groups participate in overseas cleft outreach, the vast majority do so sporadically and focus on short-term needs. Building long-term cleft capacity in low- and middle-income countries requires consistent connection, partnership, and trust with talented and motivated overseas cleft providers. Previously, our group demonstrated that AR can be used to create a “hands-on” virtual interactive presence to remotely connect a cleft surgeon in the United States with international surgeons seeking to care for children with cleft lip in an endemic region.² By adding a 13-month AR-based curriculum to semiannual on-site training, international surgeons self-reported significant improvement in understanding and performing various aspects of cleft care, including preoperative counseling, principles and techniques of cleft repair, cleft anatomy, operative design and anthropometry, intraoperative decision-making, and operative efficacy. At 30-month follow-up, no child with nonsyndromic cleft required transfer to a distant tertiary care center for nasolabial repair. AR technology accelerates knowledge and skill transfer and helps sustain gains in cleft capacity but is reliant on the presence of a remote expert to teach and proctor cleft surgery.

Here, we aimed to develop a cleft AI model capable of recognizing and automating the marking of nasolabial landmarks important for cleft lip and nose repair. Such AI assistance can be integrated with existing AR platforms or be used independently to guide surgical design. Our cleft AI algorithm accurately and precisely identifies anthropometric landmarks with errors well within the accepted range set forth by standard AI benchmarks.^5,6 The highest NME of all the 21 anthropometric points marked by our cleft AI algorithm corresponds to cleft-side cphi. Unsurprisingly, this is the most subjective of all points when designing a cleft lip repair as the surgeon chooses this point based on anatomy, measurements, tissue characteristics, and individual surgical experience. Of the next six points with highest NME, three (c’nt1, c’nt2, and c’rl) correspond to the three points used to design a triangular flap of vermilion (“Noordhoff’s triangle”) that extends from this very same subjective cleft-side cphi point. Our AI algorithm’s relative inconsistency is, therefore, reflective of inherent human inconsistency. This demonstrates both the limitation and strength of using an AI model to automate cleft lip markings as these algorithms are only as strong as their ability to learn from (and resist) the inherent variabilities of human choice. Such limitations can theoretically be improved by training the algorithm with larger data sets and input from multiple cleft experts.

Using AI to recognize and place cleft markings with high reliability is the first step in equipping overseas learners with the knowledge and skill needed to impact global cleft burden. Our cleft AI model can place two-dimensional markings on a live video feed over an extensive range of viewing angles. (See Video [online].) This reinforces, complements, and automates many aspects of our current AR-based technology platform used to provide remote surgical guidance. A logical next step for this platform is to utilize the AI-generated placement of key anatomic points to automate various design patterns for cleft lip and nose repair. The algorithms to automate these operative design markings are currently being developed; Figure 3 illustrates how the AI cleft algorithm described here establishes the foundation for this forthcoming application.

Fig. 3.

The cleft algorithm’s 21 AI-generated anatomic points can be connected in different patterns to guide various techniques for unilateral cleft lip/nose repair. Illustrated here are (A) Millard’s rotation-advancement (RA) repair, (B) Mohler’s RA modification, and (C) Mulliken’s RA modification.

Previously, our team has demonstrated that surgical markings and designs can be projected onto a patient’s three-dimensional surface anatomy using deformational light technologies to adjust for variations in contour⁸; we similarly anticipate projecting AI-generated cleft markings and repair designs onto the nasolabial surface of children undergoing cleft repair. The intricate contours of the lip and nose introduce challenges related to three-dimensional conformation of projected light. In collaboration with engineering and computer science colleagues, we are using structured light scanning and other projection technologies to solve this challenge (data not shown). At a minimum, the pairing of our cleft AI model with three-dimensional surface projection (projected AR) can automate guidance of basic surgical design. We are currently working on additional AI models that can be integrated with surface projection to assist with navigation and virtual guidance through sequential operative steps. Integrating AI and projection-based AR allows remote experts to use projected light to physically mark and directly interact with a patient’s operative site in real time from anywhere in the world. This can accelerate knowledge and skill transfer from remote expert to all surgical learners in that operating room without requiring obtrusive headsets or other wearable technologies. Remote experts also maintain a human check to computer automation in a procedure that is as measured and mathematic as it is artistic and experiential.⁹ Domestically, this technology can be highly effective for surgical education of medical students, residents, and fellows as well as for collaboration and innovation among surgical experts and leaders.

Our cleft AI algorithm is the first published example of how AI can be used to accurately detect and mark the abnormal surface anatomy of a craniofacial condition. With transfer learning and data augmentation, we achieved this feat by using a very manageable data set. This paves a pathway for development of additional algorithm for other craniofacial disorders. We chose to focus our AI model on cleft lip repair both because of the clear global need for more effective cleft outreach and because cleft lip repair is perhaps the most challenging procedure we could choose to test this technology. Our cleft AI model reliably identifies 21 surgically important anatomic landmarks in the highly three-dimensional surface anatomy of an infant’s lip and nose. Having demonstrated this is feasible, it is possible to envision expanding the use of AI algorithms to understand all of human surface anatomy—from face to foot—the full territory and playground of plastic surgeons. By leveraging such AI algorithms to bind surface anatomy to radiographs (computed tomography, magnetic resonance, etc), AI models can guide dynamic surface projection that illuminates underlying anatomy (normal and abnormal), marks cutting/resection guides, illustrates flap designs, and reliably guides surgical approaches. Adding AI algorithms that recognize wounds, vascularity, and relaxed skin tension lines promises to expand future directions. As “pioneers of new frontiers,” plastic surgeons must be at the forefront of technology that optimizes surgical outcomes and facilitates knowledge and skill transfer—especially when and where there is great need.

ACKNOWLEDGMENTS

The authors would like to acknowledge the significant efforts and assistance of Elsa Chahine, MD, and Beyhan Annan, MPH, Research Fellows for Global Smile Foundation, who collated and standardized the patient images needed to conduct this study. This article abides by the guidelines in the Declaration of Helsinki.

PATIENT CONSENT

Informed consent was obtained from all individual participants for whom identifying information is included in this article by the Global Smile Foundation.