Advancements in the future of automating micromanipulation techniques in the IVF laboratory using deep convolutional neural networks

Abstract

Purpose

To determine if deep learning artificial intelligence algorithms can be used to accurately identify key morphologic landmarks on oocytes and cleavage stage embryo images for micromanipulation procedures such as intracytoplasmic sperm injection (ICSI) or assisted hatching (AH).

Methods

Two convolutional neural network (CNN) models were trained, validated, and tested over three replicates to identify key morphologic landmarks used to guide embryologists when performing micromanipulation procedures. The first model (CNN-ICSI) was trained (n = 13,992), validated (n = 1920), and tested (n = 3900) to identify the optimal location for ICSI through polar body identification. The second model (CNN-AH) was trained (n = 13,908), validated (n = 1908), and tested (n = 3888) to identify the optimal location for AH on the zona pellucida that maximizes distance from healthy blastomeres.

Results

The CNN-ICSI model accurately identified the polar body and corresponding optimal ICSI location with 98.9% accuracy (95% CI 98.5–99.2%) with a receiver operator characteristic (ROC) with micro and macro area under the curves (AUC) of 1. The CNN-AH model accurately identified the optimal AH location with 99.41% accuracy (95% CI 99.11–99.62%) with a ROC with micro and macro AUCs of 1.

Conclusion

Deep CNN models demonstrate powerful potential in accurately identifying key landmarks on oocytes and cleavage stage embryos for micromanipulation. These findings are novel, essential stepping stones in the automation of micromanipulation procedures.

Introduction

Micromanipulation encompasses the use of minute instruments under high-powered microscopy to perform delicate highly dexterous tasks on biological cells or tissues. In the field of assisted reproductive technologies (ART), micromanipulation is an essential component within the in vitro fertilization (IVF) process. Two high-stakes micromanipulation techniques that are omnipresent in clinics worldwide include intracytoplasmic sperm injection (ICSI) and laser-assisted hatching (AH).

As a fertilization technique, ICSI is a procedure that involves aligning MII oocytes, selecting and immobilizing sperm, and injecting sperm with care to avoid damaging the mitotic spindle. The mitotic spindle is located adjacent to the extruded polar body (PB) and is not visible with brightfield microscopy. Standard practice of the procedure requires aligning oocytes based on location of the PB and injecting sperm 90° perpendicular to the PB, posing significant pressure on the embryologist for proper identification of the PB, penetration with the ICSI needle, and injection of the selected sperm. Successful fertilization depends on a number of factors such as cumulus cell removal and timing of the ICSI procedure [1, 2]. Oocyte lysis or degeneration following ICSI results from both independent biological factors like age-related oocyte quality or gonadotropin stimulation itself [3] and operator-dependent factors such as excessive manipulation, mechanical removal of the cumulus complex, or the final injection procedure. Prospective studies in sibling oocytes evaluating the blastocyst development rate of surplus embryos showed significantly different results among ICSI technicians based solely on volume of cytoplasm aspirated [4]. Currently, training embryologists on the practice of ICSI rely on trainees either practicing on immature oocytes incapable of fertilization or on animal oocytes, such as hamster ova [5]. Although efforts have been made to improve training to adjust for each individual embryologist’s skill level [6], ICSI continues to be highly operator-dependent.

Laser-assisted hatching (AH) is a procedure designed to enable embryo escape from the zona pellucida (ZP). Most commonly, AH is performed on cleavage stage embryos at the 8-cell stage either prior to transfer or in preparation for preimplantation genetic testing (PGT). Studies suggest that AH may increase pregnancy rates among older women with repeat IVF failure and in frozen embryo transfers, however, successful implantation can be exceedingly hard to assess given the large variation of AH technique, timing, and indication-based utilization. AH utilizes a 1.48-µm diode laser that can damage embryos if applied too close to blastomeres. Optimal location of AH involves applying the laser in a region of the ZP that is the furthest way from healthy blastomeres, creating a reliance on operator skill for successful completion without damaging the embryo.

Traditionally, highly skilled embryologists are tasked with performing these complex procedures. Successful performance depends mainly on subjective evaluation with light microscopy to identify the optimal procedure location. Aside from a trained eye to effectively identify the PB or the largest perivitelline space away from healthy blastomeres, embryologists must also concurrently possess dexterous skill in performing the task, leaving these at-times high-stake procedures susceptible to wide variability between embryologists. With all these factors contributing to the success of the procedure, objective standardization, at least in the identification of key morphologic landmarks at the time of the procedure, is key in ensuring continual optimization of these techniques. Artificial intelligence (AI) serves as a promising assistive tool towards the objective standardization of these techniques as an adjunct to the watchful eyes and hands of skilled embryologists. AI is a broad term that describes the use of computational machine learning algorithms trained to perform complex decision-making tasks that mimic human intelligence and information processing. Within image-based fields like embryology, convolutional neural networks (CNNs) have proven to be valuable assets in image-based processing within ART. The goal of our study was to determine whether deep learning frameworks could be developed to accurately identify the optimal orientation of human oocytes and cleavage stage embryos to perform complex micromanipulation techniques such as ICSI and AH.

Materials/methods

Data collection and handling

Retrospective data was collected at the Massachusetts General Hospital (MGH) Fertility Center in Boston, Massachusetts. After obtaining approval through the Institutional Review Board (IRB#2019P001000), time-lapse imaging videos of MII oocytes and cleavage stage embryos were recorded using a commercial time-lapse imaging system (EmbryoScope, Vitrolife). The imaging system used a Leica 20X  objective that collected images at 10 min intervals under illumination from a single 635 nm LED. Each embryo was exported as a video (.avi) using the imaging system software. Videos were processed into each respective image frame for all timepoints post insemination. Each extracted image was 250 × 250 pixels and subsequently cropped to 210 × 210 pixels to remove any potential identifiers present within the frame. Out-of-focus images were included in the datasets and used for both testing and training. Only completely non-discernable images of embryos were removed from the study. Since image collection timepoints across all patients were not consistent we binned them into groups of around 18 min intervals.

MII oocyte image assessment

Following follicular aspiration during oocyte retrieval, all oocytes and associated cumulus complexes were examined under light microscopy. Highly skilled embryologists examined each cumulus complex and isolated all oocytes using standard laboratory techniques for oocyte handling. Following digestion with cumulase, all mature MII oocytes were placed into culture media and imaged using the Embryoscope 2–4 h post oocyte retrieval. Data augmentations were performed by rotating each image in multiples of 30°. Their annotations were also modified to match correct site. Each oocyte image within the training set (n = 13,992 oocytes from 228 fresh cycles) was assessed and annotated by one of four embryologists. Validation (n = 1920 oocytes) and testing (n = 3900 oocytes) were completed with independent data sets not previously introduced to the CNN.

Day 3 cleavage stage image assessment

Cleavage stage embryos at the 8-cell stage were examined and imaged at 66-hours post insemination (hpi) on day 3 of development prior to hatching for either transfer or extended culture with subsequent trophectoderm biopsy for preimplantation genetic testing. Data augmentations were performed by rotating each image in multiples of 30°. Their annotations were also modified to match correct site. Each embryo within the training set (n = 13,908 embryos) was assessed and annotated. Validation (n = 1908 embryos) and testing (n = 3888 embryos) were completed with independent data sets not previously introduced to the CNN.

CNN development

Two CNN models were developed for this study. Each CNN model was trained and tested to evaluate and classify images into 12 classifications. The classifications used in each model resembled the pattern for digits on a clock, spaced 30° apart to either provide an accurate location of the extruded PB in the oocytes (Fig. 1) or provide an optimal location on the zona pellucida (ZP) to perform assisted hatching (Fig. 2).

Fig. 1.

Image of a human oocyte classified as a “7” based on the extrusion of the 1st polar body at 7 o’clock

Fig. 2.

Image of a day 3 cleaved human embryo classified as an 11 based on the largest perivitelline space between the zona pellucida and blastomeres at 11:00

To evaluate optimal ICSI location, we developed a deep convolutional neural network trained with 13,992 annotated images of denuded MII oocytes. We classified the location of the PB and the corresponding location for sperm injection. Accuracy of the CNN-ICSI system was then tuned with a validation set containing 1920 oocyte images to ensure the program training was complete. The developed network was evaluated using an independent test set of 3900 MII oocyte images with known PB and sperm injection location classifications.

A second CNN model was developed focusing on identifying the optimal location for assisted hatching (CNN-AH). The network was trained with 13,908 annotated images of cleavage stage embryos. The optimal location of AH was classified as the greatest distance between the zona pellucida and healthy blastomeres (Fig. 4). Accuracy of the CNN-AH system was then validated with dataset of 1908 cleavage stage embryo images. The trained network was then evaluated using an independent dataset of 3888 cleavage stage embryo images with known AH location classifications.

Fig. 4.

The ICSI receiver operating characteristic (ROC) revealed micro and macro area under the curves (AUC) of 1, which confirmed that the AI can accurately identify the position of the polar body and the corresponding location for sperm injection

Study design and statistical analysis

Each CNN model was tested over three repetitions of running models. When the CNN-ICSI model correctly identified the location of the PB, a “pass” was noted, otherwise, a “fail” was noted. When the CNN-AH model correctly identified the optimal location of AH, a “pass” was noted, otherwise, a “fail” was noted. Initially, the models were created such that instances where the system identified a location that was ± 30°, with respect to the embryologist annotation, was treated as a “pass”. Additionally, we evaluated the AH and ICSI models to assess the accuracy in identifying the embryologist’s annotated location for ICSI or AH with no deviation, where identification of the optimal location had to directly match the embryologist’s annotation. The absolute error was used by the software to calculate accuracy rate for each model. 95% confidence intervals for the absolute accuracy are exact Clopper-Pearson confidence intervals.

Results

The CNN-ICSI model was able to correctly identify the location of the PB and corresponding location for sperm injection with 99% accuracy (95% CI: 98.5–99.2%, n = 3900 oocytes; 3 replicates) (Fig. 3) within a 30° margin of deviation. The receiver operator characteristic (ROC) revealed micro and macro areas under the curves (AUC) of 1, which confirmed the CNN model’s ability to accurately identify the PB position along with the corresponding location for sperm injection (Fig. 4). The CNN-ICSI model was able to correctly identify the location of the PB and corresponding location for sperm injection with no deviation from the embryologist’s annotation with 92.15% accuracy with a 95% confidence interval (CI) ranging between 91.26 and 92.98% (n = 3900) (Fig. 5).

Fig. 3.

ICSI Confusion Matrix Testing: The CNN identified the correct location to perform ICSI with 98.9% accuracy with a 95% confidence interval (CI) ranging between 98.5 and 99.2% (n = 3900). Solid purple squares represent the correct identification of true label by the CNN with the surrounding light purple hatches being the 30° area of freedom. Numbers noted in white boxes denote incorrect CNN identification of the true label

Fig. 5.

ICSI Confusion Matrix Testing (without adjacent grouping): The CNN identified the correct location to perform ICSI with 92.15% accuracy with a 95% confidence interval (CI) ranging between 91.26 and 92.98% (n = 3900). Numbers noted in white boxes denote incorrect CNN identification of the true label

The CNN-AH model was able to correctly identify the appropriate region to apply laser AH on the zona pellucida with 99.41% accuracy (95% CI: 99.11–99.62%, n = 3888 embryos; 3 replicates) (Fig. 6) within a 30° margin of deviation. The ROC revealed micro- and macro-AUC of 1, which confirmed the CNN’s ability to accurately pinpoint the correct location to perform AH (Fig. 7). The CNN-AH model was able to correctly identify the appropriate region to apply laser AH with no deviation from the embryologist’s annotation with 93.54% accuracy with a 95% confidence interval (CI) ranging between 92.73 and 94.30% (n = 3888) (Fig. 8).

Fig. 6.

The deep learning CNN was able to correctly identify the appropriate region to apply laser AH on the zona pellucida with 99.41% accuracy with a 95% confidence interval (CI) ranging between 99.11 and 99.62% (n = 3888). Solid purple squares represent the correct identification of true label by the CNN with the surrounding light purple hatches being the 30° area of freedom. Numbers noted in white boxes denote incorrect CNN identification of the true label

Fig. 7.

The AH receiver operating characteristic (ROC) revealed a micro and macro areas under the curves (AUC) of 1, which confirmed that the AI can accurately pinpoint the correct position to perform AH based on the space between healthy blastomeres and the zona pellucida

Fig. 8.

The deep learning CNN was able to correctly identify the appropriate region to apply laser AH on the zona pellucida with 93.54% accuracy with a 95% confidence interval (CI) ranging between 92.73 and 94.30% (n = 3888). Numbers noted in white boxes denote incorrect CNN identification of the true label

Discussion

Micromanipulation techniques, such as ICSI and AH, have become omnipresent among IVF clinics across the world and play significant roles in improving IVF outcomes. Here, we report a promising application of AI to identify key landmarks and locations for micromanipulation techniques. The accuracy of the developed models in identifying the landmarks was strikingly high, demonstrating that these findings are greater than the likelihood of chance alone. Additionally, our 12-point classification system is an efficient and quantifiable method to measure oocyte/embryo orientation in two-dimensional space. Creating these models is beneficial to both improve real-time micromanipulation techniques and contribute towards future automation.

Currently, highly trained embryologists perform all micromanipulation procedures, leading to high operator dependence in oocyte/embryo evaluation, location selection, and procedure execution. Objective standardization as an adjunct to daily laboratory procedures can assist with quality assurance between embryologists and may streamline the automation of embryology procedures in the future. Significant contributions have been reported towards automating the different components of micromanipulation throughout the IVF process. Robotic ICSI, first reported in 2011[7], was aimed at robotic cell injection with a motion control cell holding device paired with computer vision-based sperm head tracking. Initial efforts focused on oocyte denudation used microfluidics to pass an enzyme-treated cumulus-oocyte complex through a series of microchannels on a chip to strip the cumulus complex [8]. In 2019, the focus shifted to automated sperm selection with multiple deep-learning algorithms [9] developed to assess acrosomal morphology for sperm selection [10, 11]. Zhang et al. [12] reported efforts towards robotic immobilization of motile sperm by using computer vision for sperm head tracking and tail motion anticipation, with a 94.5% consistent immobilization success. Image processing techniques have been proposed for identifying the PB for orientation purposes. The gradient-weighted Hough transform when coupled with elliptical fitting has been shown to detect the PB under different magnifications [13]. However, the accuracy of this rudimentary imaging algorithm drops considerably if there are any morphological issues with the cell [13]. Adaptive Sobel edge detection and Otsu’s thresholding have also been used to identify the PB in mouse oocytes with high success [14]. Integration of this cell detection method with a robotic mouse oocyte rotation system has been proposed as an automated method for performing ICSI (15). While promising strides have been made towards creating a fully automated system for performing ICSI, it is to our knowledge that no such medical device has successfully integrated all components needed for the procedure. The only previous attempt toward automating AH was reported by Wong et al. (16) in two-cell mouse embryos. This study used image segmentation algorithms to identify the maximum distance between the zona pellucida and the blastomeres from 2-cell mouse embryos (16). While novel, this algorithm was not subsequently applied to human embryos or applied at a clinically relevant time period making the algorithm utilizable in IVF lab applications.

Our developed algorithms would be easily integrated into clinics with minimal additional time needed to optimize landmark identification. Specifically, the CNN-ICSI model can pair with previously published robotic ICSI platforms [7] to identify the optimal location of sperm injection prior to robot-assisted procedure performance. Additionally, a major challenge of any micromanipulation technique is visually processing two-dimensional images of three-dimensional biological tissues, which adds complexity to optimal location identification. Our 12-point classification system for polar body recognition allows for added precision for position recognition, whereas if paired with a free-moving robotic ICSI arm, would potentially remove the need to rotate the oocyte prior to fertilization. Also, our system is the first attempt using day 3 cleavage stage human embryos, which allows for additional precision in the CNN training and validation as it examines embryos at a clinically relevant time for AH. These CNN systems would remove at least one component of decision-making within the process, allowing for landmark identification that is not dependent on human operator skill. Ultimately, these algorithms are elegantly designed to utilize current imaging technology to optimize landmark identification, a key step towards automating embryology procedures by using machine learning techniques for improved accuracy. While automation of these processes would be the goal of objective procedure standardization, these models can be extremely valuable now as an additional layer of quality assurance and safety without increasing add-on costs to the IVF process.

The major limitation of this study was the sample sets used to train the CNN models. All images were obtained using only the Embryoscope imaging platform at one timepoint from a single IVF center. This algorithm has not been validated using images from other centers or other imaging platforms, which would be an important goal for future studies to improve generalizability of the algorithm for commercial integration. Future studies should include imaging from multiple imaging platforms and from multiple clinics/geographic regions to better capture a diverse cohort of oocytes and embryos to reduce unintentional bias in model training and validation. We believe that our previously published approach in reliably analyzing lossy medical image datasets of different distributions can be utilized to enhance the robustness and generalizability of the developed CNN-ICSI and CNN-AH reported here [14].

Conclusion

Artificial intelligence algorithms, particularly image-based CNN models, can be used to accurately evaluate and identify the optimal location for micromanipulation techniques such as ICSI and AH. Our described models demonstrate the powerful potential of utilizing CNNs in ART laboratory techniques to complete complex tasks. Developing deep learning technologies to address complex decision-making with precision and accuracy serves as a stepping stone towards the automation of micromanipulation procedures.

Author contribution

All authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by Charles L. Bormann, Victoria S. Jiang, Manoj Kumar Kanakasabapathy, and Prudhvi Thirumalaraju. The first draft of the manuscript was written by Victoria S. Jiang and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Data availability

The datasets, R Studio statistical code, and machine learning algorithms used and/or analyzed during the current study are available from the corresponding author on reasonable request and under a data transfer agreement with Mass General Brigham.

Declarations

Ethics approval and consent to participate

Informed consent was obtained from each individual before participation. Study protocols were approved by the Institutional Review Board (IRB#2019P001000) at Massachusetts General Hospital and Brigham and Women’s Hospital.

Consent for publication

Not applicable.

Conflict of interest

Authors Dr. Hadi Shafiee, Dr. Charles Bormann, Prudhvi Thirumalaraju, and Manoj Kumar Kanakasabapathy wish to disclose a patent, currently licensed by a commercial entity, on the use of AI for embryology (US11321831B2). The rest of the authors declare that they have no competing interests.