Artificial Intelligence and Ophthalmic Surgery

Structured Abstract

Purpose of review:

Artificial intelligence (AI) and deep learning have become important tools in extracting data from ophthalmic surgery in order to evaluate, teach, and aid the surgeon in all phases of surgical management. The purpose of this review is to highlight the ever-increasing intersection of computer vision, machine learning, and ophthalmic microsurgery.

Recent Findings:

Deep learning algorithms are being applied to help evaluate and teach surgical trainees. AI tools are improving real-time surgical instrument tracking, phase segmentation, as well as enhancing the safety of robotic-assisted vitreoretinal surgery.

Summary:

Similar to strides appreciated in ophthalmic medical disease, AI will continue to become an important part of surgical management of ocular conditions. Machine learning applications will help push the boundaries of what surgeons can accomplish to improve patient outcomes.

Introduction

Artificial Intelligence (AI) has the potential to revolutionize the practice of modern medicine, and its capabilities are continually being realized. AI, and more specifically machine learning (ML), has sparked global interest for a new era of clinical diagnostics and management that can be applied across all fields of medicine. Important applications of machine learning in medicine include early detection of tuberculosis on chest radiographs,¹ identifying cancerous skin lesions as accurately as experts in the dermatology field,² and predicting the onset of sepsis in intensive care unit patients prior to clinical recognition.³

Ophthalmology is no stranger to ML, and the field lends itself well to ML algorithms due to the wide array of imaging modalities utilized in ophthalmic practice. Applications of AI in teleophthalmology may assist in the detection of retinopathy of prematurity, diabetic retinopathy, age-related macular degeneration (AMD) and glaucoma in remote and underserved populations.⁴ Several groups have demonstrated high accuracy for disease detection utilizing deep learning programs in diabetic retinopathy^5–9 and AMD,^10–12 and there is growing literature of ML capabilities for retinopathy of prematurity,^13,14 glaucoma,^15,16 and ocular oncology.^17,18

Perhaps less discussed but equally exciting application of AI in medicine is its role in surgical care. Despite surgery generally considered to be highly effective in treating disease, there is significant space for enhancing patient outcomes utilizing AI for preoperative, intraoperative, and postoperative tools and decision-making. AI-driven tools may become important components of preoperative planning by analyzing clinical imaging, patient characteristics, and timing of surgery to guide the surgical approach. They may provide the surgeon personalized data on potential risks and benefits for each particular eye and predict postoperative outcomes.

Intraoperative AI applications can range anywhere from some assistance (e.g. intraoperative image guidance) to full automation (e.g. autonomous robotic surgery), and major strides have been reached in various points along this spectrum.¹⁹ Shademan et al. demonstrated in vivo autonomous robotic anastomosis of porcine intestine using the Smart Tissue Autonomous Robot (STAR), an AI-driven surgical robot which outperformed expert surgeons in various surgical tasks.²⁰ Other robotic models with variable levels of autonomy include the DaVinci robot (Intuitive Surgical, Sunnyvale, CA) that completely relies on human control, TSolution-One (THINK Surgical, Fremont, CA) for Orthopedic knee replacements that utilizes an automated bone cutter, and Mazor X (Mazor Robotics, Caesarea, Israel) for robotic-assisted spinal surgery.¹⁹

Ophthalmologic microsurgery is technically complex with high stakes and requires a high level of precise movements on awake and non-paralyzed patients. Although generally successful in treating ocular disease, there exists a steep learning curve due to numerous physical, mental, and technical variables that affect surgical consistency. With intraocular surgery performed almost exclusively using an operating microscope with a near-fixed surgical field, a large amount of surgical video data can be used to train ML algorithms for the purposes of intraoperative guidance and future robotic eye surgery.

This review highlights the major advancements in AI and ophthalmic surgery, ranging from surgical education and preoperative planning to instrument tracking and robotic-assisted microsurgery.

Ophthalmic Surgery Education

Due to the learning curve of ophthalmic surgery and the constant room for improvement in how novice surgeons acquire technical skill, novel AI-based and virtual reality (VR) technologies have potential to disrupt ophthalmic surgical education. There is significant value in bolstering surgical training prior to even entering the operating room with live patients, as well as enhancing feedback during and after surgery.²¹

Virtual Reality-based Simulators

The Coronavirus Disease-2019 (COVID-19) pandemic highlighted the importance of finding ways outside of the operating room to acquire surgical skill due to a precipitous drop in elective case volume.²² One such avenue of skill acquisition is the use of virtual reality-based simulators, with the most well-known simulator called the Eyesi Surgical Platform (Haag-Streit, Bern, Switzerland). These VR platforms have become a mainstay in ophthalmic surgical education, and they offer a controlled environment in which improvement on the simulator may translate to improvement in the operating room. One teaching institution found a significant reduction in cataract surgery complication rates after introducing an Eyesi simulator to the resident surgical curriculum,²³ and another study demonstrated improved scores on a previously validated task-specific cataract surgery assessment rubric (Objective Structured Assessment of Cataract Surgical Skill) for novice surgeons in live surgery after using the Eyesi simulator.²⁴ VR-based surgical training may have utility outside of residency education, as Genentech incorporated VR headsets to teach surgeons how to place and refill the port delivery system, an implant currently being studied to deliver anti-vascular endothelial growth factor continuously for retinal disease.²⁵ AI models can learn surgeon decision-making in VR environments which researchers can apply these lessons in real-life scenarios. For example, if a certain maneuver novice surgeons perform consistently leads to a complication during simulation, this may serve as a predictive tool and warn the learner in future environments.

Intraoperative Guidance

One exciting avenue for AI support tools is real-time surgical guidance for novice surgeons in which a “smart” operating microscope can incorporate augmented reality to provide step-by-step direction.²⁰ This system would require rapid identification of the surgical step and associated instrumentation with low latency. Morita et al. utilized a convolutional neural network (CNN) model, known as the Inception V3 model, to accurately recognize the phase of cataract surgery in a surgical video at an average of 96.5% in real-time.²⁶ Useful step-specific scenarios for the novice cataract surgeon may be utilizing the microscope to overlay a pre-specified capsulorhexis size indicator on the anterior capsule. Instrument tracking (Figure 1) coupled with intraoperative imaging such as microscope-integrated optical coherence tomography (OCT) could aid the novice surgeon in depth within the cataract nucleus during phacoemulsification, or ascertaining the forceps distance from the retina during a macular membrane peel. OCT integrated into the instrument tip can also be utilized for retinal surface detection and even tremor compensation.²⁷ Real-time guidance can generate warnings to avoid complications or offer recommendations if the novice surgeon is noted to struggle. Further work on instrument tracking is described below.

Figure 1:

Instrument Tracking in Cataract Surgery. Tool trajectories are mapped using multiple green points on the instruments during capsulorhexis for both the cystotome needle (A) and Utrata forceps (B). Images courtesy of Shameema Sikder, M.D.

Surgical Evaluation

Surgical educators recognize that timely and objective feedback of surgical performance for the novice surgeon is necessary, prompting the emergence of several surgical grading rubrics in cataract surgery and multiple subspecialty surgeries. These rubrics were viewed as a way to standardize assessing surgical skill acquisition, however they can be time intensive for the educator and likely not completely objective if completed concurrently in front of the learner. Automating these structured feedback tools may more consistently track a learning surgeon’s skill level. Kim et al. used deep neural networks to accurately predict if capsulorhexis performance in a surgical video was performed by an expert or novice (as determined by a masked grader utilizing an assessment rubric).²⁸ The authors crowdsourced the annotation of the intraoperative tool tip to map tip velocities for their deep neural network. Their work demonstrated a step-specific evaluation, but future research may expand this to all cataract surgery steps for the novice surgeon. With grading rubrics now being produced for subspecialty surgeries, future work will likely assess phase segmentation in these surgeries. While a vitrectomy may have more variables than an uncomplicated cataract surgery, researchers may analyze specifically the salient portions of a surgery, such as the internal limiting membrane peel maneuver during macular hole surgery. This objective feedback using video-based data may serve to form competency-based milestones in training rather than simply reaching a minimum number of surgical cases in order to meet graduation requirements.

Cataract Surgery

Cataract surgery is the most commonly performed intraocular surgery, and cataract numbers are expected to increase with the aging population. AI technology has been studied for all phases of cataract diagnosis and grading, pre-planning, and surgical management.²⁹

Preoperative Planning

Choosing the correct intraocular lens (IOL) power prior to cataract surgery is one of the main components of preoperative planning. Several IOL formulas exist to aid the surgeon in selecting an IOL for the best refractive outcome, however achieving a postoperative refraction within 0.50 D of emmetropia or slight myopia occurs approximately 70–80% of the time.³⁰ Ladas et al. introduced an IOL “super formula” by combining ideal portions of existing formulas to help the surgeon select an appropriate IOL for both typical and atypical values of axial length, corneal power, and anterior chamber depth.^31,32 The authors report the latest version of the formula incorporates postoperative outcomes to continually improve the formula using an AI-hybrid approach.³² This interest in new AI-driven IOL formulas has surged in the last few years, and existing IOL formulas have also been improved using artificial neural networks.³³ Although not widely explored, ML utilization in pre-planning for other surgeries may prove useful. Feeding pre-operative OCT and visual acuity data into an ML algorithm may guide the clinician on the rate of progression of an epiretinal membrane to decide the optimal timing for surgery. Optimal forceps grab points on the retinal surface from pre-operative OCT, such as rips in the internal limiting membrane, could be overlaid on the retinal surface during surgery to guide the novice surgeon.

Tool and Phase Identification

Video recordings of cataract surgery have become the ideal source of surgical data for step identification and instrument tracking. Initially, surgical tool analysis depended on external markers attached to the instrument. Specifically in laparoscopic surgery, these markers were color-, optical-, or acoustic-based.³⁴ In ophthalmic surgery, motion-based analysis using sensors was limited and only applied to corneal suturing, oculoplastics, or in a simulated environment.³⁵

Current efforts for instrument tracking utilize computer vision-based methods with video recording as the data source. Early algorithms applied motion features or spatiotemporal polynomials, and later methods used conditional random fields or hidden Markov models.³⁶ Current methods apply deep learning using CNNs to recognize surgical instruments. Hajj et al. processed a sequence of consecutive images using CNNs for instrument classification and reported an area under the ROC curve (AUROC) of 0.953.³⁷ In 2017, the CATARACTS challenge was presented to evaluate tool annotation algorithms for cataract surgery.³⁴ The dataset is comprised of 50 cataract surgeries in which 21 surgical tools was manually annotated by two experts. 14 participating teams submitted solutions based on deep learning algorithms, and the winning solution achieved an AUROC of 0.987.³⁴ Although the winning solution did not outperform the manual human grader, it came strikingly close and the authors concluded an automated solution would be a feasible option. When feeding CNN outputs to recurrent neural networks (RNN) in order to take temporal relationships into account, researchers noted an AUROC of 0.996.³⁷ Adding contextual information such as a video stream of the surgical stray could also be considered to improve tool identification.^34,39

Vitreoretinal Surgery and Robotic-Assisted Surgery

Vitreoretinal surgery is a complex and technically challenging field of ophthalmic microsurgery that requires precision movements to manipulate extremely delicate tissue. Physiologic hand tremor has been shown to have a root mean square amplitude of 50–200 microns.⁴⁰ 75% of forces generated by maneuvers during retinal microsurgery measured less than 7.5 mN in magnitude, and only about 20% of events generated at this level could be felt by the surgeon.⁴¹ Coincidentally, this is the near the amount of force required to create a retinal tear during vitrectomy in live rabbits.⁴² Although surgeons are successful in current surgical maneuvers, robotic-assisted surgery may help reduce intraoperative adverse events such as iatrogenic retinal breaks and retinal detachments following macular surgery. Robotic surgery may push the boundaries further for what retinal surgeons are capable of doing, such as retinal vein cannulation for occlusive disease or administering chemotherapy into feeder vessels of intraocular tumors.⁴³

Visual Instrument Tracking

Instrument tracking in retinal microsurgery is difficult due to the wide variety of instrument appearances during changes in illumination, motion blur, shadows or surgical field of view. Several groups have described various computational techniques to overcome these challenges. Snitzman et al. placed instrument detection and tracking in the same framework to account for the target object entering and leaving the field of view regularly.⁴⁴ Alsheakhali et al. modeled the surgical instrument as a Conditional Random Field in which each part of the instrument is detected separately.⁴⁵ Roodaki et al. estimated the instrument tip’s distance to the retina by using the shadow of the instrument in intraoperative OCT images.⁴⁶ Recently in retinal surgery researchers have applied deep learning algorithms for instrument tracking, as Zhao et al. used a CNN with a spatial transformer network for the process of tracking retinal surgical tools in real time without labels.⁴⁷

Deep Learning for Robotic-Assisted Surgery

The DaVinci robot, commonly used in other surgical subspecialties, can be adapted for corneal suturing and pterygium surgery, but was kinematically inadequate for intraocular surgery.⁴⁸ Although robotic-assisted surgery has not been widely adopted in ophthalmology, several ophthalmology-specific machine prototypes are promising. The Intraocular Robotic Interventional Surgical System (IRISS) has a master controller and a manipulator which includes two independent arms that hold surgical instruments.⁴³ The Johns Hopkins Steady-Hand Eye Robot is a surgeon-initiated robot in which the surgeon holds instruments that are attached to robotic arms. The robot aims to smoothen the surgeon movements and also has force-sensing capabilities. The Preceyes Surgical System is also a robotic assistant in which the robotic manipulator holding the instrument is separate from the master controller, yet the surgeon still can use an instrument such as a light pipe in the other hand as well as the standard operating microscope. This system was used in the first live human intraocular robotic surgery.⁴⁸

Converging deep learning and ophthalmic robotic microsurgery is a relatively recent phenomenon, and AI will likely have an increasing role in enhancing the safety, efficacy, and reliability for robotic platforms. One important, early application of AI is helping predict surgeon movements. He et al. utilized an active interventional control framework (AICF) to increase operation safety by actively intervening the operation to avoid exertion of excessive forces to the sclera.⁴⁹ A RNN was used for scleral force modeling, and the authors report AICF may eliminate undesired events in advance by predicting the surgeon’s manipulations. Further work is ongoing for robotics enhanced by AI capable of adapting in real-time, and early iterations will likely still be heavily dependent on human supervision.⁵⁰

Future Implementations of AI in Retina Surgery

There is undoubtedly a role for AI to enhance the safety and efficiency of retinal surgery in order to improve patient outcomes. Deep learning algorithms can provide intraoperative guidance as well as predict and act upon possible adverse events. For instance, a smart microscope could automatically detect intraoperative hemorrhage during a diabetic tractional retinal detachment repair and consequently raise the infusion pressure to tamponade the bleeding. It can subsequently reduce the pressure slowly if the hemorrhage has ceased. Other possibilities include using intraoperative OCT to detect residual subretinal fluid during an air-fluid exchange in retinal detachment repair, or detect and track residual perfluorocarbon bubbles on the retinal surface. In regard to vitreous cutter dynamics, an AI-driven system could automatically modulate the duty cycle of the vitreous cutter port depending on nearby retinal mobility. The video display of a heads-up surgery system could automatically adjust white balance and color filters to enhance membrane staining during macular surgery. Clearly there are limitless possibilities for “smart” instruments and intraoperative guidance parameters to help augment the surgeon’s capabilities.

Conclusion

Analogous to the remarkable achievements in AI and ophthalmic medical disease, AI-driven tools in surgery are advancing at a rapid pace. We will likely continue seeing novel applications of machine learning for all components of surgical management of ocular disease to aid the clinician. Utilizing deep learning in robotic surgery will help augment the surgeon’s capabilities to continue providing advanced surgical care for our patients.

Key Points:

• Deep learning algorithms are increasingly utilized for capturing data from ophthalmic surgical videos.

• Automated identification of surgical instruments and phase segmentation will aid in surgical education, intraoperative guidance, and robot-assisted surgery.

• Machine learning has expanded the possibilities in how surgical instruments and intraoperative tools will improve surgical outcomes.