Artificial intelligence in time-lapse system: advances, applications, and future perspectives in reproductive medicine

Abstract

With the rising demand for in vitro fertilization (IVF) cycles, there is a growing need for innovative techniques to optimize procedure outcomes. One such technique is time-lapse system (TLS) for embryo incubation, which minimizes environmental changes in the embryo culture process. TLS also significantly advances predicting embryo quality, a crucial determinant of IVF cycle success. However, the current subjective nature of embryo assessments is due to inter- and intra-observer subjectivity, resulting in highly variable results. To address this challenge, reproductive medicine has gradually turned to artificial intelligence (AI) to establish a standardized and objective approach, aiming to achieve higher success rates. Extensive research is underway investigating the utilization of AI in TLS to predict multiple outcomes. These studies explore the application of popular AI algorithms, their specific implementations, and the achieved advancements in TLS. This review aims to provide an overview of the advances in AI algorithms and their particular applications within the context of TLS and the potential challenges and opportunities for further advancements in reproductive medicine.

Introduction

The birth of Louis Brown in 1978, which marked the first successful implementation of in vitro fertilization (IVF), represents a significant milestone in assisted reproductive technology (ART), heralding a new era in the field of reproductive medicine [1]. Over time, advancements such as cryopreservation and preimplantation genetic test for aneuploidy (PGT-A) have been introduced, gradually improving assisted reproductive procedures for the 186 million infertile couples worldwide [2]. However, despite these advancements, accurately predicting the outcomes of an IVF cycle remains challenging, with clinical pregnancy rates varying between 7.6 and 51.1% in the USA [3]. Nevertheless, the emergence of time-lapse system (TLS) technology has provided valuable support for ART and has led to the identification of novel markers within the IVF procedure that can potentially enhance patient outcomes.

Researchers have discovered that TLS technology enables the identification of abnormal changes in embryonic development, such as the spontaneous collapse of the human blastocyst, which can indicate lower quality and higher rates of degeneration and aneuploidy [4]. By analyzing the relationship between oxygen consumption measurements and morphokinetics, researchers can observe the oxygen consumption during embryo cleavages and predict the potential for successful implantation [5]. TLS technology is currently being explored as a promising alternative to blastomere biopsy for PGT-A, as the latter has been found to hinder embryo compaction and blastulation [6].

In the realm of computer science, artificial intelligence (AI) techniques, including machine learning (ML) and neural networks, are being extensively researched. AI involves the design of intelligent computer systems that exhibit human-like characteristics such as language understanding, learning, reasoning, and problem-solving abilities [7]. The rapid progress in AI technology has led to numerous applications in the medical field, including pathology [8], dermatology [9], radiology [10], and oncology [11]. AI is increasingly enhancing diagnoses, prediction-based diagnoses, and clinical care. Recently, the World Health Organization (WHO) defined AI as a supportive tool to improve healthcare decision-making [12]. In the field of ART, AI has proven valuable in predicting embryo cell stages, blastocyst formation, and live births, thereby aiding in the selection of high-quality human blastocysts [13, 14]. Furthermore, AI is anticipated to have additional applications in automating the labeling of work pieces, developing optimal protocols for IVF stimulation, and quality control of culture conditions.

The availability of large amounts of image data has propelled the progress of AI applications in advancing TLS within reproductive medicine. AI’s ability to process vast quantities of image information has alleviated the workload and facilitated shared decision-making with embryologists [15]. However, it is crucial to prioritize the explainability of AI systems to gain a thorough understanding of the decision-making process involved in these applications. By ensuring transparency and explainability, clinicians can mitigate potential risks associated with the use of AI in ART and other critical domains. This review aims to provide an overview of recent AI algorithms, their application in TLS, and the potential future collaborations in this field.

Time-lapse system and AI requirements

Incubation and selection of embryos are critical stages in IVF process. However, traditional grading methods offer limited insights into embryonic morphology at specific time points. Monitoring developmental changes across multiple time points can provide a more comprehensive assessment of the embryo’s potential for successful implantation. Additionally, maintaining stability in the culture medium is crucial for ensuring stable embryo growth while evaluating their morphology in the laboratory setting. In 2008, the introduction of commercially available TLS systems with built-in cameras eliminated the need to transfer embryos between dishes and enabled continuous evaluation without disrupting the culture conditions [16]. In 2010, Wong et al. [17] pioneered the development of an algorithm that marked a significant breakthrough by predicting blastocyst formation on day 2 of embryo culture using TLS. This marked a transition from mere observation of human embryos in culture to harnessing TLS as a valuable tool for selection and prediction. TLS provided objective, precise, qualitative, and quantitative data without disrupting the delicate culture conditions.

However, the routine use of TLS in IVF clinics has stirred controversies. There is a lack of substantial evidence to differentiate between TLS and conventional incubation in terms of live births, miscarriages, stillbirths, or clinical pregnancies [18, 19]. Several factors contribute to this uncertainty. During the 3 to 5-day incubation period, TLS generates a substantial volume of digital images of embryos at regular intervals (typically every 10 to 20 min). Embryologists can review these images to assess division timings, temporal intervals for each cell stage transition, and identify embryos with abnormal cleavage patterns, which may not be suitable for transfer [20]. However, the manual process of reviewing and annotating the vast number of labeled time-lapse datasets is time-consuming and requires expertise and experience. Moreover, manual assessments introduce inter- and intra-individual variations, particularly between different laboratories. To address these challenges and standardize the process, computer-based algorithms with high accuracy and reliability are essential for automation.

Artificial intelligence (AI) tools are exceptionally adept at swiftly processing and analyzing vast amounts of data. Given the intricate nature of TLS techniques and the multitude of variables involved, AI techniques offer an ideal solution. Numerous algorithms have been employed in TLS to classify, predict, and rank reproductive medical data, aiming to improve the efficiency of IVF cycles by reducing failed transfers and miscarriages, ultimately leading to the successful birth of a healthy baby from the transfer of a single, euploid embryo [21].

Engage in AI models

AI technologies, specifically ML and neural networks, have gained widespread adoption in medical applications. These algorithms are designed to perform reasoning and map input-to-output data relationships. However, the training process of these computational models is often opaque, leading to their characterization as “black boxes.” Nevertheless, gaining an understanding of the underlying principles behind these models can reveal their internal mechanisms and enhance decision-making processes [22].

Machine learning

ML has emerged as the predominant approach within the field of AI for the development of valuable software, encompassing applications such as computer vision, image recognition, medical diagnosis, and prediction systems. ML leverages data and algorithms to mimic human learning, progressively improving precision. Through the training process, ML algorithms can automatically discern patterns within vast datasets, enabling them to generate predictions that prove instrumental in optimizing treatment procedures and forecasting pregnancy outcomes [23].

ML encompasses two prevalent models: supervised and unsupervised learning (Fig. 1). Supervised ML employs algorithms to classify or predict outcomes by analyzing new input, learning from labeled input-outcome pairs, and continuously improving its performance [24]. Numerous models have been developed for supervised learning, including decision trees, random forests, support vector machines, gradient boosting, and Bayesian classifiers. Selecting the most appropriate model from this diverse range can be a challenging task. Alternatively, the concept of ensemble learning has been explored in supervised learning, where “weak” classifiers—those with only slightly better than random guessing performance—are combined to achieve higher accuracy than individual classifiers alone [25]. In contrast, unsupervised learning refers to algorithms designed to identify patterns within datasets that lack classification or labeling information [26]. Algorithms employed in unsupervised learning, such as clustering, anomaly detection, and dimensionality reduction, can automatically group, label, and classify data points within these datasets without external guidance or supervision. Clustering, for example, has the ability to compress the entire feature set into cluster codes. This simplifies the handling of large datasets, as ML systems can efficiently process data groups using cluster identifiers. Consequently, the output data from clustering can serve as feature data for downstream ML systems.

Fig. 1.

Supervised and unsupervised learning: supervised learning involves using raw input data that requires human intervention to label or provide explicit guidance before being processed by model algorithms to generate the desired outcomes. On the other hand, unsupervised learning operates without external guidance, automatically classifying data points based on inherent patterns and structures

Neural networks

Neural networks, a subset of ML, draw inspiration from the architecture and dynamics of neuronal networks within the brain [27]. These networks consist of three essential layers: an input layer, one or more hidden layers, and an output layer. By propagating data through these interconnected layers, neural networks can learn from errors and autonomously make decisions without human intervention. Currently, three primary types of neural networks are utilized in the context of TLS: feedforward, recurrent, and convolutional neural networks (Fig. 2).

Fig. 2.

Neural networks: feedforward neural networks (FNN) consist of artificial neural networks (ANN) and deep neural networks (DNN) with one or multiple hidden layers, enabling them to process information in a unidirectional flow. Recurrent neural networks (RNN) possess the capability to learn from both past and future information in time series data. Convolutional neural networks (CNN) leverage convolutional operations to extract essential features from the data, followed by further processing using specific functions to generate appropriate outputs

Feedforward neural network

Feedforward neural networks (FNN) are designed to process information in a unidirectional manner, flowing from the input layer through the hidden layer(s), and ultimately reaching the output layer. This architecture ensures that there are no loops or cycles within the network, maintaining a clear direction of information flow [27]. Artificial neural networks (ANNs), a specific type of FNN, are computational models inspired by biological neural systems, particularly the human brain. ANNs are represented as a graph with nodes referred to as “neurons,” mimicking their biological counterparts.

ANNs offer numerous advantages, such as parallel processing capabilities, the ability to generate outputs with incomplete information, which is often encountered in reproductive data collection, and the capacity to extract hidden features and relationships without imposing restrictions on input and residual distributions. In comparison to traditional forecasting models, ANNs have demonstrated superior performance in predicting treatment outcomes [28]. However, ANNs require data to be translated into numerical values, which can influence their performance. Therefore, it is crucial to design an ANN that closely aligns with the characteristics of the input vectors, necessitating the user’s involvement in determining the representation scheme.

A deep neural network (DNN), also known as deep learning (DL), is an ANN with multiple hidden layers positioned between the input and output layers. The depth of the architecture is determined by the number of layers that perform non-linear operations in the learned function [29]. DL algorithms excel at handling both structured and unstructured data, including images, text, and audio. By automatically learning hierarchical representations of features, DL methods have the capacity to directly extract complex functions from data, eliminating the need for manual feature engineering [29]. However, the successful application of DL requires a comprehensive understanding of the domain and problem at hand to mitigate unintended consequences, such as biased models that discriminate against certain groups. Training DL models also demands significant computational resources, including powerful processors, ample memory, and substantial investments of time and financial resources. Despite these challenges, DNNs offer enhanced performance, efficiency, and accuracy. Notably, studies leveraging this network architecture have successfully constructed robust models for embryo selection in reproductive medicine [30, 31].

Recurrent neural network

The architecture of a recurrent neural network (RNN) can vary, encompassing fully interconnected or partially connected nets, and includes multilayer feedforward networks featuring separate input and output layers [32]. Unlike the independent layers of FNN, RNN architectures possess the ability to learn dependencies both forward and backward in time series data, making them suitable for problems involving sequences and temporal relationships [33]. For instance, predicting the next word in a sentence relies on the context and words that precede it.

A modified version of RNN called long short-term memory (LSTM) was developed exceling at modeling complex sequential data by preserving both short- and long-term dependencies, even in the presence of noise or incompressible input sequences [34]. However, LSTMs may not perform optimally with highly nonlinear or non-sequential data, as their strengths lie in capturing temporal relationships. Additionally, training LSTMs on large datasets can be time-consuming, as they require learning the parameters of LSTM cells, which can be computationally demanding. In the field of medical science, LSTM has demonstrated its superiority in handling temporal information. For example, Liao et al. [14] employed LSTM to learn cell stage information from consecutive frames of TLS videos and accurately predicted blastocyst formation by extracting morphokinetic embryo development parameters demonstrating the effectiveness of LSTM in capturing the temporal dynamics of embryo development. This approach highlights the potential of LSTM in extracting valuable insights from sequential data and its applicability in the field of reproductive medicine.

Convolutional neural network

CNN is a powerful DL model specifically designed for image recognition and classification tasks. It leverages the convolution operator, a fundamental concept in digital signal processing, to transform input information by convolving it with a filter, resulting in a new output signal. This process helps identify and isolate crucial features in the signal while filtering out irrelevant information.

CNNs consist of multiple layers, including convolutional, pooling, and fully connected layers, which collectively extract and process information from input images [35]. In the convolutional layers, convolutions between the input matrix and the filter generate units in a new layer, enabling two-dimensional feature extraction. Pooling calculates an expected value for each spatial region the filter passes through, effectively reducing the size of the image while preserving its essential contours. Once the matrix size is sufficiently reduced, it is flattened into a vector and fully connected between layers, employing the rectified linear unit (ReLU) activation function. The final layer of the CNN network, the last fully connected layer, contains the same number of units as the number of classes in the classification task. It applies the softmax activation function to compute the probability distribution across the classes.

Despite the effectiveness of CNNs in image recognition and classification, they face several challenges. The vanishing gradient problem becomes more pronounced with deeper layers, requiring a substantial amount of training data to prevent overfitting. Operations like pooling can significantly slow down the network, particularly when multiple layers are involved. Training CNNs can also be time-consuming if the computational resources lack robust data processing capabilities. However, the advancements in graphics processing unit (GPU) technology have played a significant role in overcoming these challenges and enabling the application of CNNs in analyzing and identifying features in digital images.

In recent studies, researchers have utilized multilayered CNNs ranging from 5 to 43 layers to classify embryos based on their morphological quality. Notably, the Xception model achieved the highest accuracy of 90.97% in these studies [36]. This research showcases the potential of CNNs in the field of reproductive medicine, where they can contribute to improved embryo classification and selection processes.

Clinical applications of AI technology

The primary objective of AI in TLS is to optimize the selection process of the most viable embryo for transfer, as this significantly impacts the final outcome of a pregnancy. AI technology plays a crucial role in aiding physicians by providing them with informed decision-making support and alleviating the substantial workload in busy IVF laboratories. Numerous research studies have been conducted to automate key steps in the analysis of TLS data, including segmentation, classification, prediction, and the exploration of novel findings related to human embryos. These efforts aim to develop end-to-end pipelines that enable the automated evaluation of raw TLS image sequences, ultimately leading to improved clinical pregnancy outcomes. In Table 1, a compilation of studies is provided, highlighting the application of AI techniques in various aspects of TLS and their impact on different outcomes. The majority of these articles utilize similar input data, such as embryo images, morphokinetic parameters, and clinical information, to develop and train their AI models.

Table 1.

Performance among AI based studies with TLS

Author	Number of samples	Input data	Model	Main advancement	Performance
Automated segmentation/detection
Zhao Xu et al. [37]	24 embryos	Day-one embryo images	The hierarchical fully convolutional network (CNN) the generative adversarial network (GAN)	Automated day-one human embryo segmentation on three distinct features: zona pellucida (ZP), cytoplasm and pronucleus (PN)	The precisions of segmentation were that cytoplasm over 97%, PN over 84%, and ZP around 80%
Leahy, Jang et al. [38]	989 embryos	Embryo images	ResNet101 (CNN) Inception-V3 (CNN) ResNeXt101Mask R-CNN & ResNet50	Automate five key morphokinetic features extraction: Zona pellucida segmentation Fragmentation scoring Stage classification Cell object instance segmentation Pronucleus object instance segmentation	ZP accuracies between 93 and 99% Fragment scoring labels agree 88.9% of the time The developmental stage classifier with 87.9% accuracy mAP of 0.737 and 0.680 on cell object segmentation and pronucleus segmentation
Fukunaga, Sanami et al. [39]	900 embryos	Embryo images	CNN	Automated pronuclei determination algorithms	The sensitivity rates of 0PN, 1PN, and 2PN were 99%, 82%, and 99%
Automated classification
Zabari, Kan-Tor et al. [40]	67,707 embryos	Embryo sequence images	U-Net & ResNet18 CNN	Automatic and standardized morphokinetic annotation of embryo development from time of pronuclei appearance to start of blastulation	R-square 0.994, 97% accuracy
Thirumalaraju, Kanakasabapathy et al. [36]	2,440 embryos	Blastocyst stage images	Inception-V3 ResNet-50 Inception-ResNET-v2 NASNetLarge ResNeXt-101 ResNeXt-50 Xception	Evaluate popular CNN approaches in classifying blastocyst stage based on their morphological quality	Xception 90.97% accuracy
Malmsten, Zaninovic et al. [41]	110,000 embryos	Cleavage stage images	Inception-V3 (CNN)	Up to 8-cell stage classification	98.1% for up to 4-cell and 95.8% for up to 8-cell
Dirvanauskas, Maskeliunas et al. [42]	10 embryos	Embryo images	AlexNet (CNN) Cecoc K-nearest neighbor (KNN) Decision tree (DT) Ensemble Discriminant Naïve Bayes	Automatically cleavage stage (up to 8-cell stage) classification, directly in the incubator machine	93.58% accuracy for the task of eight-cell prediction
Raudonis, Paulauskaite-Taraseviciene et al. [43]	300 embryos	Cleavage stage sequence images	AlexNet (CNN) VGG16 (CNN)	Embryo location detection and up to 4-cell stage classification	At least 90% accuracy in the detection of embryo location. Overall accuracy of over 92% for cell stages classification
Liu, Huang et al. [44]	170 embryos	Embryo sequence images	ResNet50 Multi-task deep learning with dynamic programming	Embryo early development stage (up to 4-cell stage) automated classification from time-lapse videos	86.9% accuracy
Khan, Gould et al. [45]	265 embryos	Cleavage stage sequence images	AlexNet (CNN)	Automatically count the number of cells in the developing embryos up to the 5-cell stage	91.05% average accuracy
Prediction: blastocyst quality
Liao, Zhang et al. [14]	10,432 embryos	Embryo sequence images	DenseNet201 Long short-term memory network (LSTM) The spatial–temporal ensemble model (STEM) Gradient boosting classifier	Predict blastocyst formation and usable blastocysts using TLM videos of the embryo’s first days	78.2% accuracy and 0.82 AUC in predicting blastocyst formation, and achieved 71.9% accuracy and 0.79 AUC in predicting usable blastocysts
Coticchio, Fiorentino et al. [46]	230 embryos	Embryo images	The k-nearest neighbor LSTM The hybrid ensemble classifier	Predicting blastocyst formation at early cleavage stages by detection of cytoplasm movement velocity and AI analysis	AI models with the blind operator classification, resulted in 82.6% accuracy
Giscard d'Estaing, Labrune et al. [13]	891 embryos	Morphokinetic parameters	Logistic regression	A score issued from a ML system with self-improvement capacity able to predict embryo fate	DynScore AUC 0.638 in predicting blastocyst formation
Kan-Tor, Zabari et al. [47]	> 20 000 fresh embryos	Embryo images	DNN U-Net (CNN) Random forest Logistic regression	Early prediction of embryo blastulation and embryo implantation on day 3	Blastulation prediction AUC 0.74 Implantation prediction AUC 0.71
Khosravi, Kazemi et al. [30]	10,148 embryos	Blastocyst stage images	Inception-V1 DNN (STORK) Decision tree	Accurately predict the quality of human blastocysts and help select the best single embryo for transfer	Predicts blastocyst quality with an AUC 0.987
Prediction: implantation rate
Theilgaard Lassen, Fly Kragh et al. [48]	181,428 embryos	Embryo images	3D CNN Logistic regression	Implantation rate	iDAScore v2.0 The AUCs on day 2, 3 and 5 + were 0.861, 0.872, and 0.954 for all embryos and 0.669, 0.621, and 0.707 for KID embryos
Duval, Nogueira et al. [49]	9986 embryos	Embryo sequence images Clinical data	3D CNN Gradient boosting	Multi-centric clinical data to better predict clinical pregnancy outcome	AUC 0.727
Berntsen, Rimestad et al. [50]	115,832 embryos	Embryo images	3D (CNN) LSTM	Predict pregnancy in terms of fetal heartbeat (FH)	Automated iDAScore v1.0 model with AUC 0.67 for KID embryos all embryos with an AUC of 0.95
Bori, Paya et al. [51]	637 patients	Morphokinetic parameters	Artificial neural networks (ANN)	Novel embryo features capable of predicting implantation potential	AUC 0.77 & novel parameters in which blastocyst expanded diameter and trophectoderm cell cycle length had statistically different values in implanted and nonimplanted embryos
Tran, Cooke et al. [52]	10,638 embryos	Embryo sequence images	DNN	Predict the probability of pregnancy with fetal heart (FH) from time-lapse videos	IVY model with AUC 0.93
Milewski, Kuczyńska et al. [53]	610 embryos	Morphokinetic parameters	Artificial neural networks (ANN)	Predicting implantation model based on morphokinetic information	AUC 0.71
Prediction: live birth rate
Yang, Peavey et al. [54]	375 embryos	Morphokinetic parameters	Logistic regression, XGBoost, decision tree, and random forest K-means clustering	Predict clinical pregnancy and live birth rate	Random forest with the best AUC 0.69 and 0.64 in predicting implantation rate and live birth rate. Cluster of slowest morphokinetics had a higher pregnancy rate of 71% accuracy
Huang, Kosasa et al. [55]	101 embryos	Blastocyst images	U-Net architecture (CNN)	Analysis of human blastocyst expansion from the unedited time-lapse image in selecting single blastocysts for transfer	The averaged expansion rate was significantly (P = 0.007) greater in euploid blastocysts that resulted in live births compared with those resulting in failure to give a live birth
Sawada, Sato et al. [56]	470 embryos	Embryo images	Attention Branch Network (ABN) Residual neural network (ResNet56)	A confidence score based on embryo images that is useful for noninvasive selection of embryos that could result in live birth	AUC 0.642
Prediction: ploidy status
Zou, Pan et al. [31]	773 embryos	Morphokinetic parameters Dysmorphism and irregular cleavages Blastocyst quality Clinical data	Decision tree, random forest, adaptive boosting (AdaBoost), Gradient boosting decision tree (GBDT), and gradient boosting (XGBoost) A novel hybrid DNN-LSTM	Euploid blastocysts Implantation prediction	GBDT model had the highest AUC 0.70 in euploidy prediction; AUC 0.78, accuracy 77% for implantation outcome of transferred euploid blastocysts
De Gheselle, Jacques et al. [57]	539 embryos	Morphokinetic parameters Embryonic developmental feature Demographic and clinical data	Random forest classifier Gradient boosting classifier Support vector machine Multivariate logistic regression Naïve Bayes	Euploid blastocysts	The RFC model had the highest accuracy (71%) and AUC (0.75). Morphokinetic features ranked higher with tPB2, PN, and t7 showing the highest relative weight
Lee, Su et al. [58]	690 embryos	Embryo videos	Two-Stream Inflated 3D ConvNet (I3D)	Ploidy status through raw time-lapse video	AUC 0.74 in discriminating between aneuploid embryos (group 1) and others (group 2, including euploid and mosaic embryos)
Huang, Tan et al. [59]	1803 embryos	Blastocyst images & clinical data	3D-resnet50	Ploidy status	Predict euploid on the testing dataset with AUC of 0.80

Automated segmentation/detection

Segmentation is a crucial step in image analysis, particularly in the context of TLS sequences, as it enables the identification and assessment of embryo composition at each stage. However, manual segmentation is time-consuming and requires trained experts, posing challenges for medical workflows. Therefore, recent advancements focus on developing accurate automatic segmentation techniques that can seamlessly integrate into medical practices [37, 38].

In an IVF cycle incorporating TLS, embryos remain within the time-lapse system continuously from the time of intracytoplasmic sperm injection (ICSI) or fertilization until the day of transfer or cryopreservation. The confirmation of successful fertilization, indicated by the appearance of two pronuclei, usually occurs around 16–18 h after insemination or ICSI. Monitoring, recording, and evaluating this event is crucial to ensure normal fertilization. However, the time window for observing the appearance and disappearance of pronuclei is short, and their size and shape can be challenging to discern, particularly since the female pronucleus appears after the male pronucleus. Fukunaga et al. [39] developed a CNN automated pronuclei determination system that achieved high sensitivity rates in determining the number of pronuclei, with rates of 99% for 0PN, 82% for 1PN, and 99% for 2PN. Furthermore, the model accurately evaluated overlapping 2PN cases that are difficult to determine through microscopic observation alone. This high sensitivity can be valuable for embryologists in confirming fertilization.

Several factors, including pronuclei size, zona pellucida (ZP) properties, and embryo fragmentation, correlate positively with embryo development. These characteristics play a vital role in assessing the embryo’s “score” during the early stage. Zhao et al. [37] and Leahy et al. [38] employed CNNs for image analysis to extract valuable features during the pronuclear and cleavage stages, demonstrating consistent results in high-accuracy segmentation (Fig. 3). Leahy et al. [38] further developed a pipeline of five CNNs to expedite the measurement of biologically relevant features that could assist in embryo selection.

Fig. 3.

Segmentation model. An automated segmentation model utilizing convolutional neural networks (CNN) to produce precise pattern annotations for image object segmentation

Morphology assessment remains the most commonly used method for evaluating embryo quality. Traditionally, embryo quality was manually graded based on intrinsic morphological features such as cell size, density, and expansion level. However, there is a growing interest in automating the grading process. Huang et al. [55] utilized a DNN on time-lapse images to analyze human blastocysts, employing an optimized U-Net for the segmentation task. This software accurately identified the irregular shape of the trophectoderm in focused images, enabling precise measurement of the total enclosed area and subsequent analysis of human blastocysts.

Automated classification

The process of assigning labels to unlabeled groups is known as classification. While classifying embryo cells is not a new challenge, there is a continuous need for high accuracy and performance, particularly when aiming for autonomous performance within TLS. Classification involves labeling from a finite set of classes, with two primary types: binary classification, which distinguishes between two classes, and multiclass classification, which identifies three or more distinct classes [36, 42].

In the evaluation of embryo quality, especially in the case of human embryos, images often contain substantial noise and overlapping cells in the background, making traditional cell counting methods based on manually crafted features inadequate for handling large datasets. Consequently, researchers have actively sought automatic and accurate methods for cell number classification in the early stages of embryo development, providing valuable information for embryologists (Fig. 4). Khan et al. [45] proposed a framework for automated cell counting up to the 5-cell stage in human embryo development. Their approach utilized a CNN model consisting of eight layers, including five convolutional and three fully connected layers, with a softmax function generating a distribution across five class labels. The framework achieved an average accuracy of 91.05% for cell classification. Other researchers have also explored the use of various algorithms, such as MTDL and VGG16, for cleavage cell number classification, obtaining comparable results [43, 44].

Fig. 4.

Classification and prediction model. The artificial intelligence model can be applied to various types of input data to generate the desired outcomes in classification and prediction tasks

As the number of cells in an embryo increases, accurately determining the cell count during division becomes more complex and challenging due to overlapping cells, requiring a more robust model and advanced optimization strategies. To address this issue, Malmsten et al. [41] proposed a novel method that utilizes DL to classify embryo images based on different cell division stages, up to 8 cells, specifically for human embryos. Their approach integrated advanced DL models and latent domain knowledge, enabling highly accurate predictions of cell division stages. Initially, the Inception-V3 model was trained using a publicly available dataset of mouse embryos, followed by the development of an optimization strategy using multiple images from the TLS sequence. Subsequently, the model was adapted and applied to a private dataset of human embryos, achieving accuracies of 98.1% and 95.8% for up to 4-cell and 8-cell stages, respectively.

In 2023, Zabari and colleagues [40] developed a CNN-based model that automatically annotates morphokinetic events from pronuclei appearance to the beginning of blastulation with high accuracy and standardization. The model outperformed previous automated morphokinetic annotation tools [60], achieving an R-square of 0.994 and 97% accuracy. Additionally, they utilized unsupervised clustering of high-quality embryos to investigate the morphokinetic heterogeneity of preimplantation development, independent of maternal age and blastulation rate. Comparative analysis of transfer versus implantation rates revealed differences between embryo clusters, primarily characterized by poor synchronization of the third meiotic cell-cleavage cycle. This study is expected to improve the integration of morphokinetic-based decision support tools in IVF treatments and enhance our understanding of preimplantation heterogeneity.

Automated prediction

Prediction methods involve utilizing learned outcomes to create models capable of anticipating values for new data. While classification focuses on categorizing data into classes, prediction aims to fit a shape that closely approximates the data. In the context of embryo transfer, accurate prediction of developmental potential and suitable embryos for cultivation is essential to enhance fertility, minimize economic loss, and reduce pregnancy-related complications. Automated prediction approaches are gaining interest in clinical settings to achieve these goals [13, 46].

In 2019, Khosravi et al. [30] conducted a robust study using an AI approach based on the STORK framework, which employed Google’s Inception-V1 architecture model. This study aimed to predict the quality of blastocysts from a vast collection of over 10,000 human embryos. The model was trained with labeled images of blastocyst stages, and STORK achieved accurate predictions of high or low quality blastocysts with an AUC of over 0.98.

Both Kan-Tor et al. [47] and Liao et al. [14] attempted to build models for predicting the likelihood of blastocyst formation and estimating the implantation rate in the 3-day embryo stage. Kan-Tor et al. [47] utilized an automated segmentation technique with U-Net to segment embryo images, which were then fed into a DNN model. In contrast, Liao et al. [14] combined a cell-counting algorithm with a long short-term memory (LSTM) network to create a temporal stream model. This model converted video data into numerical information and learned the duration and dynamics of cell numbers, enabling the prediction of blastocyst formation based on morphokinetic parameters.

Currently, achieving successful implantation is the ultimate goal, and researchers aim to develop robust, practical, and accurate models for clinical use [49, 53]. Significant progress has been made in supporting the selection of 3-day and 5-day embryos using traditional algorithm-based approaches like KIDScore D3 [61] and KIDScore D5 v.3 [62]. More recently, DL-based models such as iDAScore v1 [50] and iDAScore v2.0 [48] have been introduced. These models analyze time-lapse sequences without user input and provide reliable rankings of embryos according to their likelihood of implantation in both cleavage and blastocyst stages. They improve the consistency of embryo grading and save time compared to conventional evaluation methods. Tran et al. [52] also developed the IVY model, which predicts the likelihood of pregnancy with fetal heart using time-lapse videos, achieving an AUC of 0.93. However, controversies have arisen regarding this algorithm due to imbalances in the training model and classification criteria [63].

The journey from successful fertilization to live birth is a complex process influenced not only by selecting a “perfect” embryo but also by considering the clinical factors of the mother throughout pregnancy. Attempts to predict live birth rates have been made but have not yielded impressive results [54, 56]. Ploidy status, an important outcome for the success of an IVF cycle, is currently confirmed through invasive biopsies using PGT-A. However, with the development of TLS, noninvasive screening for ploidy status is expected to be facilitated by AI [31, 58].

Huang et al. [59] have developed models to predict euploid blastocysts by considering various embryo features and other factors contributing to the success of IVF treatment, including clinical data. The euploid prediction algorithm (EPA) achieved an AUC of 0.8 by optimizing time-lapse videos from specific periods. More recently, De Gheselle et al. [57] utilized a similar approach and analyzed the correlation between features and the target variable in a study on the ploidy prediction algorithm. Their results indicated that morphokinetic features, such as tPB2, PN, and t7, had the highest relative weight, while demographic and clinical parameters, such as sperm characteristics, the woman’s age at the onset of treatment, and the total dose of gonadotropins, also had significant weight.

Challenges and potential future applications AI with TLS

Current limitations/challenges

The application of AI in TLS presents several hurdles, including issues with transparency, small sample sizes, and the absence of external validation. These challenges collectively limit the efficacy of these models in producing reliable outcomes. The lack of transparency arises from the hidden decision-making processes within the model, making it challenging to discern the criteria used to predict outcomes. To build trust and understanding, enhancing transparency in AI models is imperative. On the other hand, AI algorithms heavily rely on large amounts of high-quality data for training and accurate predictions. However, acquiring comprehensive and standardized datasets in the field of ART is challenging due to the unique development patterns exhibited by each embryo, which complicates establishing consistent and standardized AI models. Variations in morphology, cell division rates, and quality further impact prediction accuracy. AI algorithms must accurately analyze and interpret the intricate morphological changes occurring during embryo development, including subtle differences and subjective assessments or uncommon patterns, which can be challenging. Privacy concerns and the limited availability of embryos for analysis in individual centers or research studies raise concerns about data sharing, potentially leading to biased or less accurate predictions. Extensive validation studies and regulatory approvals are necessary to demonstrate AI-based systems’ safety, efficacy, and cost-effectiveness for their widespread adoption in ART centers. Keeping pace with evolving technologies and research findings challenges ensuring optimal performance and accuracy.

Potential future works

Despite the challenges, AI holds significant potential in conjunction with TLS for embryo evaluation in ART. Researchers have conducted numerous studies over the past 5 years to develop and enhance AI-assisted TLS techniques, leading to improved accuracy. Novel markers such as cytoplasmic movements and blastocyst expansion rate have been investigated to select higher-quality embryos [46, 51, 55]. The current emphasis on noninvasive diagnostics presents an opportunity to uncover novel and stable markers for assessing ploidy and implantation rates, consequently enhancing the likelihood of healthier live births as the quality of imaging equipment, including cameras and microscopes, advances. The utilization of improved imaging technology, such as higher-resolution cameras and enhanced time-lapse imaging, offers the potential to obtain more detailed and precise information concerning embryo development. By integrating these techniques with advanced image analysis algorithms, there is a possibility of augmenting the capacity to interpret the intricate morphological changes occurring during embryo development accurately. In subsequent actions, automated segmentation could be employed to assess the relative size of blastomeres and identify dividing planes, which are believed to influence the following development stages. Uneven cleavage, characterized by unequal-sized cells, has been associated with an increased risk of multinucleation and aneuploidy due to an uneven distribution of cytoplasmic molecules such as proteins and mRNAs [64, 65]. In-depth investigations into the field of the embryo culture environment, by leveraging automated evaluation of embryo morphodynamics, AI can analyze and make informed decisions regarding adjustments and modifications to the personalized culture platform. Furthermore, integrating supplementary data, such as multi-omics data, with morphological data offers the potential to yield a more comprehensive understanding of embryo development and augment the precision of predictions.

One potential solution to address the lack of transparency is the implementation of explainable AI (XAI). XAI techniques enable human users to comprehend and trust the results and outputs generated by ML algorithms. Approaches like Local Interpretable Model-Agnostic Explanations (LIME) and DeepLIFT can explain the predictions made by ML algorithms and establish a traceable link between each activated neuron, respectively [66]. This would aid embryologists in understanding and trusting the system while facilitating patient interpretation of the results [67]. Large-scale, prospective validation studies involving multiple clinics and diverse patient populations are essential to provide more robust evidence of AI-based systems’ clinical utility and effectiveness in ART. Collaborative approaches like federated learning (FL) can facilitate data sharing while preserving privacy, benefiting smaller hospitals with limited resources. In FL, a central server creates a shared model and distributes it to clients who use their datasets to train the model and send updated parameters back to the server. By sharing trained parameters, privacy is preserved. Moreover, clients contribute computational power for training, reducing the storage capacity and computational resources required by the server. Smaller hospitals with limited training data and computational resources can also benefit from collaborative FL approaches [68]. Establishing clear regulatory pathways and involving clinicians and researchers in developing and refining AI systems are crucial for their safe and responsible integration into clinical practice. Continuous model improvement based on new scientific knowledge, emerging technologies, and feedback from clinical users should be implemented to ensure the relevance and accuracy of AI systems over time.

Conclusion

This review paper aims to provide a comprehensive overview of AI and its current applications in TLS within the context of reproductive medicine. With the proliferation of large-scale datasets and advancements in precision medicine, the role of AI in this field is poised for significant expansion. Integrating AI into TLS has the potential to revolutionize clinical workflows by automating repetitive tasks and delivering expert-level performance in assessing crucial information related to embryo implantation potential. In order to propel the field forward, it is imperative to develop new methods and techniques for explaining AI outcomes, as well as foster enhanced communication and collaboration among centers to improve the reliability and accuracy of algorithms. As TLS and AI systems continue to evolve, AI-based TLS holds promise for enhancing the likelihood of successful live births through more effective and accurate embryo evaluation and prediction algorithms.

Funding

This work is supported by the National Science and Technology Council, Taiwan [grant numbers MOST110-2221-E-038–001-MY2 and MOST111-2628-E-038–002-MY3].

Declarations

Conflict of interest

The authors declare no competing interests.