Deep learning classification integrating embryo images with associated clinical information from ART cycles

Abstract

An advanced Artificial Intelligence (AI) model that leverages cutting-edge computer vision techniques to analyse embryo images and clinical data, enabling accurate prediction of clinical pregnancy outcomes in single embryo transfer procedures. Three AI models were developed, trained, and tested using a database comprised of a total of 1503 international treatment cycles (Thailand, Malaysia, and India): 1) A Clinical Multi-Layer Perceptron (MLP) for patient clinical data. 2) An Image Convolutional Neural Network (CNN) AI model using blastocyst images. 3) A fused model using a combination of both models. All three models were evaluated against their ability to predict clinical pregnancy and live birth. Each of the models were further assessed through a visualisation process where the importance of each data point clarified which clinical and embryonic features contributed the most to the prediction. The MLP model achieved a strong performance of 81.76% accuracy, 90% average precision and 0.91 AUC (Area Under the Curve). The CNN model achieved a performance of 66.89% accuracy, 74% average precision and 0.73 AUC. The Fusion model achieved 82.42% accuracy, 91% average precision and 0.91 AUC. From the visualisation process we found that female and male age to be the most clinical factors, whilst Trophectoderm to be the most important blastocyst feature. There is a gap in performance between the Clinical and Images model, which is expected due to the difficulty in predicting clinical pregnancy from just the blastocyst images. However, the Fusion AI model made more informed predictions, achieving better performance than separate models alone. This study demonstrates that AI for IVF application can increase prediction performance by integrating blastocyst images with patient clinical information.

Introduction

It is estimated that 1 out of 6 couples present infertility in their lifetime¹. Assisted reproductive technologies (ART) have been established to assist these couples. However, despite more than 40 years since the first IVF baby Louise Brown² was born, the treatment success rate is still below 30% to achieve a healthy live birth³, therefore, the majority of patients may need to go through multiple cycles of treatment before achieving a successful outcome⁴. This leads to significant physical, emotional and financial burden on the patient, therefore improving the success rate of treatment is essential⁵. Infertility treatment plans follow a structured workout to find the correct treatment for each patient⁶. This starts with understanding the patient characteristics such as age, body mass index (BMI), ovarian reserve and sperm parameters, among several variables. Despite this, most of the treatments are not personalised to the individual circumstances, rather, following a standardised treatment plan.

In vitro fertilisation (IVF) and Intracytoplasmic Sperm Injection (ICSI) are the two most important treatments of ART. Through these procedures, multiple embryos are produced with the selection of a single embryo to be transferred to the uterus⁷. Such processes rely on the quality assessment of the embryo which is visually determined by the embryologist. Considering that visual analyses by embryologists could be prone to bias⁸, the use of artificial intelligence (AI) have been introduced⁹ aiming to eliminate the subjectivity associated with these assessments. Indeed, AI has excelled in the medical field through different applications that require assessment, diagnosis and predictions¹⁰. The best example is medical images where Convolutional Neural Network (CNN) has helped diagnose medical conditions in brain, breast, lung and other organs with high accuracy¹¹. AI has also helped in genetic studies making genetic equerries efficient¹².

AI has been also utilised recently using machine learning (ML) to address ART predictions. Mainly used to predict embryo morphology to improve the subjectivity associated with embryo selection and clinical pregnancy prediction¹³. Moreover, there has also been other studies involving AI in other aspects of ART, e.g.: Oocyte assessment, sperm selection, etc^14,15. Part of these studies have used the patients’ clinical data collected during the treatments to make predictions¹⁶, and some have used embryo images paired with clinical data¹⁷. However, we hypothesise that the performance of separated models for clinical data and embryo images could be enhanced by integrating the models together. Indeed, we found that models integrating embryo images and clinical information presented higher accuracies for reproductive outcomes¹³. Salih et al.¹³ found that out of twenty validated studies only four studies combined clinical information and embryo images in their AI models, however, these studies did not integrate clinical information with embryo images for their predictive AI model capabilities^17–20. Thus, we hypothesise that integrating clinical data into models with embryo images at the embryo transfer stage could assist in connecting the features of importance from both clinical and embryo characteristics that are linked to successful reproductive outcomes. This could strengthen the decision making of the model which could become more confident in the prediction¹³.

In this study, we aimed to develop a fully integrated clinical information and embryo morphology model, using both clinical records of patients who underwent to IVF/ICSI treatments and still images of single blastocyst embryos at embryo selection procedures in one system. We tested both model situations separately before combining them together in one single model (fusion model). This gave us three models to compare the performance against each other. Thus, we investigated how different features aid reproductive outcome prediction of clinical interest in ART. Additionally, we also investigated the performance of the system through Machine Learning visualisation to determine how the AI used clinical information and embryo images aid in the decision-making process.

Materials and methods

Data curation

A total of 1503 treatment cycles, which after analysis a total of 1394 IVF/ICSI treatment cycle information was made available along with their corresponding clinical data and blastocyst still images at embryo transfer procedures from 3 IVF groups: Oasis Fertility in India, Fertility Bridge in Malaysia, and iBaby Fertility in Thailand. From a total of 1980 embryo blastocyst images, 1190 of these samples were single embryo transfers. Subsequently, there were 599 samples that were discarded due to artifacts or incomplete data; thus, a total of 1585 images and treatments were used. The samples for clinical pregnancy prediction were 1503 samples (238 multiple embryo transfer and 1265 single embryo transfer), which was then divided into 1048 samples for training, 154 samples for validation and 301 samples for testing. For the live birth outcome predictions there were 1585 samples (230 multiple embryo transfers and 1355 single embryo transfers, where successful cases were connected to the higher scoring embryo morphology images) which were then divided into 1109 samples for training, 159 samples for validation and 137 for blind testing.

The clinical features from records were categorized as either “clinical features”, “treatment features” or “ART and embryo transfer features” depending on what point of the treatment process they were collected (Table 1). The clinical features included patient features, with no personalised data, that would be captured early in the treatment process or prior to the first consultation. This included male and female features to understand the clinical background of the patients. The treatment features were the decisions of what type of treatment was chosen; for instance, IVF or ICSI. The treatment category of fresh embryo transfer was defined as single embryo transfer performed at the end of embryo culture on either day 5 or day 6, while frozen embryo transfer was defined as embryos frozen on either day 5 or 6 that were then thawed later for embryo transfer.

Table 1.

Clinical data incorporated in the AI models.

Information category	Attributes considered
Clinical features	Female and Male age, Infertility Diagnosis, Female and Male BMI
Treatment features	Cycle Number, Treatment Type (IVF/ICSI), Treatment category (Fresh/Frozen), E2 and P4 Prior to Trigger
ART and embryo transfer features	Total Number of Oocytes Inseminated, Total Oocytes Fertilised (2pn), Sperm Motility, Type of media (Single step/Sequential), Embryo Count on Day 5 and 6 (Blastocyst count), Embryo age at Day of Embryo Transfer (Day5 or day6), Number of Embryo Transferred
Clinical outcomes	Pregnancy status (Clinical pregnant/not pregnant), Pregnancy Outcome (Ongoing, delivered, miscarriage)

Note: BMI: Body Mass Index; E2: oestradiol; P4: progesterone.

The ART and embryo transfer features were collected during the ART laboratory processes (Table 1). The embryo quality grading followed the Istanbul consensus²¹, however, as these features required an embryologist to manually determine them, they were not included in any of the final models.

Human ethics

This study was conducted under Monash Health ethics RES-21-0000236L for a low-risk study and was undertaken using data obtained from various IVF groups: Oasis Fertility in India, Fertility Bridge in Malaysia, iBaby Fertility in Thailand, and city fertility in Australia. The project was run in accordance with the National Statement on Ethical Conduct in Human Research (NHMRC, 2007). This prospective study was also approved by the city fertility group, part of the Global CHA IVF partners, as the representative Internal Review Board (IRB), including all the boards of each clinic involved. Consent was acquired by the respective clinic for the unidentifiable samples to be retrieved and utilised for the study.

Reproductive outcomes

The pregnancy status and outcome were used as the output classes of the model; the samples that resulted in clinical pregnancy were labelled as “Pregnant” while the samples that failed to result in pregnancy were labelled as “Not pregnant”. Furthermore, those samples that resulted in a healthy live birth after clinical pregnancy were labelled as “Delivered”, while the ones that resulted in a miscarriage after the pregnancy were labelled as “Miscarriage”. The samples that resulted in a clinical pregnancy, but the pregnancy outcome was not complete at the time of data collection were labelled as “Ongoing”.

AI data sampling and training

All samples were split into 3 sets; 70% were placed in a training set, 10% in a validation set and 20% in a blind test set. The blind dataset was used to simulate new samples that the model is expected to see in a clinical setting. The blind test set was completely hidden with new data, while the validation set assists the model’s accuracy with each step to correct the direction of training the model will take to avoid any overfitting or negative learning curve^22,23. These data sets were split so that the proportion of reproductive outcomes, such as clinical pregnancy and live birth, were distributed evenly across all three sets. During training, each batch was randomly selected with an additional probability (weight) to make each batch approximately evenly balanced. In order to ensure proper data presentation, a random selection of samples was chosen to train the models using an equal number of potential output prediction samples. Furthermore, this weighted batch sampling allowed the models to learn the differences between the classes²⁴. During training, each step of training was temporarily saved for further analysis and supervision. After training was completed, the step that performed best on the validation dataset was selected for evaluation on the test set.

AI models

AI models were built using Python and the open source PyTorch framework²⁵. Three types of models were created to compare the performance of clinical outcome prediction;1.- clinical data model using a multi-layer perceptron (MLP) model;2.- embryo images using a convolutional neuronal network model (Resnet 34) and a 3.- fusion model including both clinical data and embryo images models by a costume made model that integrated these two models (described below).

MLP models

The clinical data models used a MLP²⁶ architecture. Various experiments were performed to justify the performance of the models, these include different hyperparameters of the model: the number of layers, number of neurons in each hidden layer, activation functions, batch sizes, learning rate and optimisation function. The input to the final version of the MLP model had 16 clinical data features (Fig. 1) and fed them through 3 fully connected layers of 16 × 1024 neurons, 1024 × 1024 neurons and finally 1024 × 2 neurons representing a prediction of each class (clinical pregnancy, live birth and miscarriage). The models were fine-tuned through various experimentations to define the hyper parameters of the models. These hyper parameters aided in the accuracy and precision of the models by altering the learning rate and the method used in forward and backward propagation. In order to improve reasonability and predictive power, experimentation resulted in the activation function LeakyReLU²⁷and the optimisation function AdamW²⁸ for training the MLP model. The experiments ran for 400 epochs, batch size of 50 samples, learning rate of 0.001 and the use of cross entropy loss.

Fig. 1.

A.- Fusion model high level architecture. The convoluted images’ model output is concatenated with the clinical information model output to provide an overall probability of the output of the sample. B.- Data collection process through encrypted online database system.

Convolutional neuronal network models

Residual Network (ResNet 34)²⁹ was used as the backbone for the embryo image classification system. ResNet includes skip connections which allows the network to converge more quickly, while stabilising the training. Experiments were run with different variants of ResNet, 18, 34, 50. ResNet34 was determined to be the best performing variant. Experiments were also conducted comparing ResNet pretrained on ImageNet³⁰ with frozen and fine-tuning convolutional weights as well as training from scratch.

Fusion models

The final type of model was a Fusion model. It used a ResNet34 backbone²⁹ to extract the features from the blastocyst images and an MLP backbone³¹for the clinical information. The basic architecture for these backbones were the same as those used in the clinical and image models. However, for both ResNet34 and MLP backbones, the fully connected layer was removed and instead a concatenating layer was formed that connects to a classifier layer for the final output prediction. The image features were extracted by feeding the blastocyst image into its ResNet34 backbone giving 512 outputs. The clinical data features were extracted by feeding the clinical data to the MLP backbone giving 1024 outputs. These data features were then concatenated and fed to a fully connected layer of shape 1536 × 2, giving a prediction of each class (Fig. 1).

Model performance metrics

All the results were performed on the blind test set that was assigned to be completely hidden from the model in the training phase. The F1-Score was calculated as the harmonic mean of the model’s precision and recall. Precision was defined as the number of true positives divided by the number of true and false positives. Recall was the number of true positives divided by the number of true positives and false negatives. As the F1-Score was calculated with the number of false positives and false negatives, it is a better assessment of the performance of these models due to the imbalance of the dataset. The Average Precision was calculated as the weighted mean of precisions achieved by the model at each threshold, with the increase in recall from the previous threshold used as the weight. The Average Precision value ranges from 0 to 1, with higher values presenting better results. The AUC is the area under the ROC curve, which was calculated by plotting the true positive rate against the false positive rate. The AUC value ranges from 0 to 1, with higher values indicating greater performance. However, a completely uninformative model would achieve an AUC score of 0.5. Any model with a lower AUC can be “negatively” informative. Inverting any prediction made by the model would therefore result in better predictions.

Model performance visualisation

Visualisation process has been used to identify how each feature of the input data helps influence the output of the model. For the clinical information model, a back propagation was utilised to assess each clinical feature’s importance in relevance to the output, which was further experimented in two specific situations to compliment the performance. Both tests required filtering of samples to comply, one test was for female samples above 37 years old while samples from females under the age of 30 years old were also compared. Furthermore, a mapping testing tool was used to identify the areas on the embryo image where the model has identified as important locations for the prediction.

ScoreCAM³², a type of class activation mapping that identifies discriminative regions of an image, was used to create saliency maps on the embryo images. These generated saliency maps assign energy to the regions of the embryo images based on their contribution to the model’s prediction for that sample. The areas highlighted in red shows the most important contributions to the model predictions relatively and those highlighted in blue being not employed by the models. Bayesian inference was used to assess whether these heatmap proportions could be used to predict the reliability of a prediction by the model.

In a similar manner, Layer-wise Relevance Propagation, LRP^33,34was used to determine the importance of each of the clinical features in the clinical model’s predictions. LRP works by propagating the prediction of the model backwards through the model’s layers, where the neurons each layer split their relevance among those neurons that feed into them. As the total relevance was conserved in each layer, the final step determines the relevance of each of the input features to the model’s prediction.

For further quantitative assessment of the visualisation heatmaps, masks were created for each embryo image to separate the embryo from the background during test procedures. These masks were used to determine the proportion of the heatmaps assigned to the embryos versus the background by the models. The masks were dilated to create three concentric ring-shaped segmentations of the embryos as well as the remaining central area of segmentation. The segmentations were created such that the outermost contained approximately 40% of the area of the embryo, the second 30% of the area, and the innermost ring and central region each contained 15% of the area. The first ring mostly contains the Zona Pelucida, while the second ring mostly contains the trophectoderm, while the third ring and innermost area mostly contain the cavity and inner cell mass.

Further reliability of prediction was experimented by manually creating masks of the embryos. These masks were dilated to create four regions, background, outer ring of embryo, inner ring of embryo, centre of embryo. The percentage of weight on each region from the saliency maps was used as input for Bayesian inference model³⁵. While ScoreCam was used for assessing the image model, the algorithm of it was adapted for the reliability check. Furthermore, the final values were taken and propagate backwards to the influence of the weights, since Baysian inference is readily available and was the algorithm used to build ScoreCam. As for the fusion model the factor between image and clinical information models was split, and the percentage value for the image was back propagated to derive the accuracy of the maps.

Results

Model performance outcomes

The overall results of the models presented in this study included prediction testing for clinical pregnancy status (classes: Clinical Pregnancy and Not Pregnant) and Live birth (classes: Not pregnant, Miscarriage and Live Birth) (Table 2). Models were divided into three: Models using clinical information, embryo images, and end-to-end fusion model which helped oversee the performance of the generated models over the span of the full sample (Clinical information, embryo image and fused sample of both characteristics). Experiments for prediction of both clinical pregnancy and live birth showed an increase in the overall area under the curve compared to the embryo images model (73% Pregnancy status model, and 80% live birth model); it maintains the overall performance for the live birth model over the clinical information model (88%) while performing better for the pregnancy status model (91%) over the clinical information model (87%).

Table 2.

Comparison of performance metrics for the different models.

Model	AUC (%)	Average precision (%)	F1-Score
Clinical pregnancy prediction
Clinic data only	0.91	0.90	0.78
Embryo images only	0.73	0.74	0.57
End-to-End Fusion clinical and embryo data	0.91	0.91	0.76
Live birth prediction
Clinic Only	0.87	0.77	0.64
Image Only	0.80	0.66	0.37
End-to-End Fusion	0.88	0.77	0.64

Note: F1-Score: Weighted average of precision and recall; AUC: Area Under the Curve.

Training of the models was closely supervised where checkpoints at key iterations of the training were evaluated. Accuracy and training loss which are key points of training evaluation were output at certain iterations to closely monitor the models’ training and confirm that it is going in the right directions. To confirm the correctness of training, the trained model was able to carry out prediction on the fused model with an average accuracy of 88% after the fivefold cross validation (88%, 88%, 88%, 88%, 87%) for the live birth model. Similarly, for the clinical pregnancy fused model an average of 89% was obtained after fivefold cross validation test (91%, 90%, 89%, 88%, 88%), where the best result model was used for further experiments. The average curves can be seen in Supplementary Figs. 1 and 2.

Fig. 2.

ROC (receiver operating characteristic curve) and Precision-Recall graphs for the Pregnancy Status models.

Clinical pregnancy

Figure 2. shows the performance of the model when presented with two class possibility output: Clinical Pregnancy and Not pregnant. The figure shows the three models involved in the system across ROC and precision graphs. The graphs show the Clinical information model scoring 91% accuracy and 90% precision, Images model preforming at 73% accuracy and 74% precision, and fusion model accuracy of 91% and precision of 91%.

Live birth

Figure 3. shows the performance of the model when presented with three class possibility output: Live Birth, Miscarriage, and Not pregnant. The figure shows the three models involved in the system across ROC and precision graphs. The graphs show the Clinical information model scoring 87% accuracy and 77% precision, Images model preforming at 80% accuracy and 66% precision, and fusion model accuracy of 88% and precision of 77%.

Fig. 3.

ROC (receiver operating characteristic curve) and Precision-Recall graphs for the Pregnancy Status models.

Visualisations

Embryo images model

The saliency heatmaps produced by ScoreCAM showed that most of the model’s importance was assigned to the trophectoderm cells of blastocyst embryos (Fig. 4). Additionally, the importance assigned by the model included the overall shape of the embryo. Furthermore, the visualisations indicate specific cases where the AI used flawed decision making by determining the important features in the models’ predictions. For instance, Fig. 4 shows two instances where the image model has assigned most of the importance of the embryo images on the empty background rather than the embryo itself.

Fig. 4.

Saliency Maps through ScoreCAM. A.- Human blastocyst embryo images before embryo transfer at day 5 of development. B.- Model prediction (clinical pregnancy/not clinical pregnancy) and prediction confidence based on visualisation performance. C.- Heatmaps to assess AI visualisation performance on each blastocyst embryo images.

Clinical data model

The average feature importance [Fig. 5] shows that the age of the female and male were the most important clinical characteristic in the model’s predictions, followed by Sperm motility and female partner BMI., from 889 samples. Their high importance indicates the potential for strong performance of a model trained with only basic clinical information. The average importance showed that both female and male patient age were highly important factors to achieve successful live birth. Interestingly, the Embryology images weight played the highest weight compared to the rest of the of the variables. Two more experiments were conducted to check how the female age would affect the feature importance of the model. Although, the sample size used for this study is limited, there is a slight change in the percentage of the feature importance as can be seen in [Fig. 5—B and C].

Fig. 5.

A- Feature Importance of the fusion model including the images and the highly important features of the clinical model for live birth prediction. The chart shows that the images take a high importance in the prediction compared to each feature individually. The clinical information weight outranks the images model in the predictions. B- Feature importance of samples with female age over 37. C- Feature importance of samples with female age under 34.

Fusion model

For the fusion model, both ScoreCAM and LRP techniques were used [Fig. 5—A]. The overall feature importance showed that the most important was the embryo images. However, all the clinical features combined were more important for the model’s predictions than the images alone. This aligns with the relative performance of the models trained with only the image and only the clinical data. It also demonstrates that future AI systems should consider clinical factors in their predictions, as the clinical features of a patient could affect the aspects of an embryo which are most indicative of clinical and live birth outcome potential. The use of these visualisation techniques allows the performance to be assessed on a case-by-case basis in real time. This could allow a CDSS to indicate personalised treatment suggestions to improve the chance for successful implantation.

Reliability of predictions

Bayesian inference was used to assess the reliability of the fusion model’s predictions for both Clinical pregnancy and live birth, whether the model was correct or incorrect, with an accuracy of 68.5% and AUC of 0.72 [Fig. 6]. This indicates that there is a recognisable difference in the produced saliency maps for samples the model makes correct predictions for, and for samples it makes incorrect predictions for. A more sophisticated segmentation of the embryos could further improve this reliability estimation. As these visualisation techniques can be applied to any sample, they allow a user to understand the behaviour of the model and critically assess its prediction.

Fig. 6.

Reliability measurements for area under the curve and confusion matrix.

Discussion

The proposed model sets the building blocks for a clinical decision support system to aid the embryologists in treatment and choosing the right embryo for transfer. This gap was expected due to the inability of predicting clinical pregnancy and live birth considering clinical factors of relevance just by using embryo still images. The current recent studies include limited clinical information integrated into AI models, the incorporation of patients and IVF treatment information into AI models could lead to the creation of various additional suggestions to prompt embryologists and clinicians to support clinical decisions at the time of embryo transfer. The clinical model produced a strong performance in the F1-Score, average precision and AUC metrics, indicating that the clinical information features that AI can utilise increases the chances of correct clinical pregnancy and live birth outcome prediction. Although the image model performed slightly worse in terms of accuracy, the F1-Score maintained the same trend. This means that although the model is making fewer correct predictions for all samples, it was able to learn some distinguishing characteristics of the samples. However, the fusion model achieved even better performance than either the Clinical or Image model could achieve alone. With much higher performance on all metrics compared to the Image only model and higher accuracy, average precision and AUC and a similar F1-Score compared to the clinical information model.

Artificial Intelligence has revolutionised prediction and decision making and has proven the ability to learn from big datasets such as medical imaging in radiography and genomics^36–40. This can also be a great asset in embryo selection and IVF treatment outcome prediction due to the ability to weigh variables and features leading to the proper outcome⁴¹. AI has also shown the flexibility to define and adapt the models created to the proper application. In this study, we have developed an AI model consisting of two essential areas in the field of IVF, a clinical model and an embryo images model, which are then combined to create the final model. As most currently proposed AI systems only use embryo images^13,42–47, the boosted performance of the fusion model presented in this study demonstrated a significant value of including the patients’ clinical data in the model to predict clinical outcomes.

Combining clinical information and embryo images into one AI model has the potential to provide a tailored approach specific to the patients delivering a more personalised treatment. Incorporating this information into AI models provides insights about the patients’ infertility history, which would have implications regarding the prognosis of patients. For instance, number of cycles a patient has attempted ART, plus increased reproductive age in the couple would mean a poor prognosis, which is known to negatively impact live birth outcomes in ART⁴⁸. Sawada et al.¹⁹ presented an approach to the gap by focusing on the time-lapse images of embryo morphology using ResNet and attention models. The highest accuracy reported was 67% when predicting live birth. However, the clinical information utilised was limited to age, method of fertilization and the type of culture media. While the embryo morphology feature was the mode of embryo transfer. In comparison for the same outcome of live birth our AI achieved an 88% prediction accuracy because the model considered more clinical features. Another model that was developed was reported by Loewke et al.¹⁸ were the reported accuracy of 75% was achieved over image segmentation of embryo images. The outcome of the model was over ongoing pregnancy with combined input by choosing the embryos depending on the certain categories. Patil et al.²⁰ also approached the idea of combining clinical information with embryo images however focused on training their model on clinical information. The achieved accuracy was 86% using CNN to predict clinical pregnancy however embryo information was utilised as clinical records in the model. In comparison, our model achieved a 91% accuracy for the same outcome of clinical pregnancy likely due to the use of a larger database and the stronger ResNet architecture. Khosravi et al.¹⁷ developed a model that took in embryo images as an input to rank the embryos on a three-class output. The model performed at 98% accuracy with the embryo grades grouped in their three-class system, however the model was only predicting the grade group unlike our model.

Although the previously discussed models show that there is an interest in combining clinical information with embryo images and/or time-lapse information, a gold standard has not been perfected yet. The AI model presented in this work, accompanied by the visualisation method to assess the AI performance, could lead to future developments for improving the embryo selection process by considering the individual clinical characteristics of patients. Indeed, our findings show how crucial the clinical information is for the embryo selection process, which it could facilitate new clinical strategies for patients during treatments¹³. Similarly, incorporating time-lapse videos in the proposed model as a next step in future studies will facilitate the study of early embryo development and biological milestones that could have influence on pregnancy outcomes.

Considering this approach, Liu et al.⁴⁹ have described a similar approach to the developed study. The study showed two models, patient background information model, and an embryo transfer model, that have been combined via an addition method to arrive at a prediction of live birth positive or negative outcome. A raw data input of 16 clinical patient information was used to train a multi-layer preceptor model, while padded embryo images are used to train the image model that utilises a readily available CNN backbone, EfficientNetV2-S⁵⁰. Although, the study had a generous dataset of 17,580 samples size, this simple model could have had noise when combining unfocused focal planes with focused ones, leading to just a 77% accuracy for binary output (positive/negative) unlike the three-class output presented in this study; Live Birth/Miscarriage/No Pregnancy. Regarding visualisation, embryo heatmaps on both studies have used the backbone of XGradCam⁵¹, however, feature importance used by the developed model has more real-time representation of the importance of clinical information compared to the logistic regression used by Liu et al⁴⁹..

Liu et al.⁴⁹ model demonstrated a significant and important attempt to integrate clinical information with embryo images. Firstly, both inputs in the developed model accepts are raw data, hence the model has access to full availability of the data, compared to processed clinical data form the output of logistic regression and pre-analysis that Liu et al.⁴⁹ utilise on their samples. Secondly, the developed model accepts any raw embryo images where the model performs a minor image size pre-processing to setup the database before initialising the training, compared to adding padding which produces a lot of empty data that overwhelmed the model. Thirdly, we test the model’s output over clinical pregnancy and live birth, including miscarriage which confirms the model’s viability over more lab possible scenarios rather than a limited positive and negative outcome only. Concluding all these points have led the developed model to a better understanding of the samples which in turn led to a better accuracy outcome.

The proposed model combines clinical patient background with embryo images into one model. Although, the results presented showed that there was no drastic increase in the performance of the fusion model when embryo images are added. This could present a strong benefit in IVF laboratories since embryo selection is performed without the consideration of patient background and clinical information⁸. Furthermore, recent studies of AI in IVF applications lack real-time incorporation of clinical information, but in this study, we observed an improvement in the predictive capabilities of critical reproductive outcomes such as live birth. While the clinical model’s strength lies in overall pregnancy and live birth prediction, the fused approach addresses the specific need to integrate clinical and imaging data to better identify viable embryos such that they will increase the likelihood of pregnancy and live birth. Thus, the Fusion model provides a more holistic tool for clinical practice by supporting embryo selection with equivalent accuracy across both types of inputs^13,52.

Personalised approach can play an important role in healthcare, and medical technologies have started taking this approach⁵³. In IVF, recent studies have shown that combining clinical information with embryo images for each study can personalise the decision and prediction per samples^17–20. However, these studies had a limited approach on the integration where the characteristics used relied on either the clinical information or the embryo images¹³; the models used lacked the full integration of both features. In this study, a fused model of both features can aid in various ways: whether to transfer fresh embryos or freeze, checking the number of cycles for suggestions on treatments among several clinical features. Importantly, this study approached training the model in an unbiased practice to include the broader population by including different ethnic groups⁵⁴. However, all the samples used were collected retrospectively. This exposes the model and the data to data shifting, where the older samples in the dataset become more negligible compared to the newer collected data⁵⁵. This can be done through testing the model in a clinical setting and survey it against embryologists to correctly assess the outcome.

Our choice of fusion approach based on a careful evaluation of existing literature⁵⁶, including the potential benefits and limitations of both early, mid and late fusion techniques. While early fusion can enhance feature extraction by integrating clinical data with imaging features⁵⁷, our preliminary experiments indicated that the marginal improvements were a significant concern. On another hand, late fusion outperformed all other fusion methods in our initial setup⁵⁸. Hence, we opted for a hybrid strategy that balances the strengths of both early and late fusion, allowing for a more nuanced integration of clinical data while retaining the integrity of the imaging features. Our findings suggest that this approach helps in capturing complementary information from both data types, essentially important clinical information with high influence on prediction, and embryo image features the reflect a healthy embryo development. Recently, fusion of clinical information with produced images to produced specific outcome predictions could be seen heavily in the medical imaging field^59–63. Patient specific treatment and diagnosis can be seen as a key element in the medical industry to minimise any risk taken by the medical staff and any risk taken towards the patient. Combining the clinical information with produced imaging through a decision support model provides the medical staff with all the information in one display. Importantly, although it could be argued that the integration of images to clinical data does not improve predictive performance as observed in different disciplines^49,59,60,62, this could of benefit in the IVF industry as current AI applications are currently only considering images or timelapse videos without including patient’s clinical data¹³. Thus, considering our current results, future studies could explore if any benefits could be observed by integrating clinical data in the IVF clinical application.

As limitations, a challenge faced during this project was the distribution of samples between “pregnant” and “non-pregnant”. In the provided data, we faced an imbalance of 69% towards positive pregnancy, which it is above what is normally expected in reproductive outcomes through IVF. In order to account for this, various strategies were used during training and evaluation of the produced models such as: dropping samples to have equal percentages or replicating a clinical situation by having less than 30% positive pregnancy samples or balancing the samples to have better representation of all possible cases. Moreover, the collected data included only information from embryos that were successfully transferred. Therefore, our models were unable to account to cycles that fail to achieve embryo transfer at the blastocyst stage, leading to a need for adjustment in the collected samples ratio to achieve the correct database usability as performed in the methodology. Importantly, diversity of the samples can play a critical role in a study of this proportion. Without testing the discarded embryos there was no indication of how much weight the samples could provide for training and prediction²⁴. This is important as the inclusion of samples that were overseen for embryo transfer or were chosen for embryo freezing also contain both important clinical and embryo quality information that can impact reproductive outcomes. This can be addressed in future work by including time-lapse videos of all discarded and transferred embryos⁶⁴. In addition, AI can be data hungry, the models developed and tested in this study suffered from a limited database. This can be addressed using a database with a larger diverse sample size which will potentially introduce more generalisability that will help the AI make more confident decisions. To help redeem the feasibility of the study is to test the concept in a clinical setting against embryologists^13,65–67. The translation of a model at this magnitude would have a massive application in the field of embryology⁶⁸. Moreover, this study will open the possibility to tailor decision making towards patients’ requirement in each the treatment⁶⁹. Although generalisability helps train the AI, the learnt features will suggest precise treatment and direction towards specific cases⁷⁰. By using visualisations and other AI explainability techniques, a clinical decision support system can improve trust in its predictions as well as provide a warning when the system is making flawed predictions⁷¹. There is much more that can be done to improve the interpretability and explainability of this system such as prediction uncertainty estimations and active learning⁷². Due to the potential impact, for an AI system to be adopted in IVF it must have a very high predictive power as well as a high level of trust in its behaviour.

One key area arose with our model was data bias and shifting. Our approach on negating the bias was to shuffle together all the data types and possibilities into one database, as mentioned in the methods and materials. This was to assist the model to generalise the input to any possible data type input and camera/microscope, while setting the ground for future state-of-art model work and architecture such as transformers⁷³. With the model setting the basic requirements of acquiring clinical features, it could be used as the standard of data collection and eliminating the chance of losing or missing any data points to avoid any possible multi-centre bias⁷⁴. Hence the model was required to have the capability to accept any type of image that could be captured by any type and form of image capturing technique, by converting the images to the correct size and imaging clarity. Lastly, for future potential upgrades the model could be used to test continuous learning to improve data shifting⁷⁵.

Although the study results showed promising potential of integrating clinical information with embryo images, future studies requires that AI models are tested against embryologists¹³. Furthermore, prospective implementation studies that can be carried out with embryologists simultaneously, to authenticate the predictions as well as confirm the practical application of AI as a tool in decision making will also facilitate clinical validation of this present work⁷⁶. Comparatively, using blind dataset is a standard practice to derive the performance of the model^77–79, for external validation and performance confirmation is an essential step to confirm the performance of AI models in a clinical setting⁸⁰. Even though the model has a high accuracy against the blind test set, in a clinical setting there could be a performance drop due to the exposure to newer samples⁸¹, thus, it is critical to assess the validity of AI models in a real-world scenario. Interestingly, the multi-modal approach presented in this study could be also explored in the future including biomarkers such as genetics, proteomics, metabolomics and even novel imaging methods such as metabolic imaging^82–84.

Ethical considerations and potential risks

The use of AI in IVF presents promising advancements but necessitates careful consideration of ethical concerns. These concerns include the potential dehumanization of the reproductive process, the introduction of algorithmic bias, a lack of transparency due to AI being a “black box”⁸⁵. Considering this, we introduced a method to assess potential AI issues by having a multi-centre approach, producing visualisation elements of feature importance to detect AI bias. Furthermore, questions surrounding efficacy, patient autonomy and changes to informed consent⁸⁶ are of substantial relevance to be considered in future applications. Importantly, strategies to account for data privacy such as data collection, analysis of sensitive personal and genetic data, equitable access, the grading and moral status of embryos should be carefully considered before AI technologies are clinically introduced⁸⁷. This was also an important point for the study, hence the use of encrypted databases for transferring data from clinics making access limited to principal investigators, as well as providing a common ground for embryo grounding and clinical information were key milestones for the initial study set up.

Conclusion

In conclusion, the integration of two distinct types of data; clinical information and embryo images, into AI models is a potentially worthwhile approach to embryo selection in ART. Embryo images contain a wealth of valuable features that AI algorithms can extract and analyse, such as morphological characteristics, developmental patterns, and cellular dynamics. These features, when interpreted in the context of relevant clinical data, could enable AI models to gain a comprehensive understanding of the factors influencing successful reproductive outcomes. Clinical information, including patient demographics and treatment protocols, could provide a framework for the AI to contextualize and interpret the visual features extracted from embryo images. As observed in this study; by leveraging the synergy between these two data sources, AI models could potentially achieve higher accuracy in predicting clinical pregnancy and live birth rates following single embryo transfer procedures. However, it is important to recognise that the efficacy of these AI models should be further investigated in real life applications by assessing the AI models performance against embryologist in a clinical setting, exposing both AI models and embryologist to new datasets from clinical cases.

Author contributions

M.S. wrote the concept and overseen writing and edits of the manuscript. M.S. and C.A. carried out the experiments and prepared the figures and tables. K.M., E.S., and S.J. helped provide the database for training and testing of the AI. S.R, B.V., H.R., and F.H. assisted in the study direction and purpose. All Authors reviewed and accepted the manuscript.

Funding

Monash Data Future Institute, MDFI-2021-12, MDFI-2021-12, MDFI-2021-12.

Data availability

The datasets generated and/or analysed during the current study are not publicly available due to patient related information but are available from the corresponding author on reasonable request.

Declarations

Competing interests

The authors declare no competing interests.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-025-02076-x.