Stability and reliability of artificial intelligence models in embryo selection for in vitro fertilization

Abstract

Objective:

To evaluate the stability and reliability of artificial intelligence (AI) models and approaches in embryo selection and rank ordering for in vitro fertilization (IVF).

Design:

A laboratory-based study evaluating the stability and consistency of single instance learning models that assess embryos individually, predicting live-birth outcomes based solely on each embryo’s morphological features. Fifty replicate convolutional neural networks with varying initialization parameters were trained and tested across two independent fertility center datasets. Model performance was assessed through embryo rank ordering, critical error rates, and intermodel variability. Interpretability analyses using gradient-weighted class activation mapping and t-distributed stochastic neighbor embedding were conducted to explore decision-making discrepancies among replicate models.

Subjects:

The study utilized retrospective embryo datasets from Massachusetts General Hospital and Weill Cornell Fertility Center, including images from 1,258 patients and 10,713 embryos from Massachusetts General Hospital, and 53 patients with 648 embryos from Cornell.

Main Outcome Measures:

Consistency in embryo ranking (Kendall’s W), frequency of critical errors (instances where low-quality embryos were top-ranked), and intermodel variability across datasets.

Results:

The AI models demonstrated poor consistency in embryo rank ordering (Kendall’s W approximately 0.35) and exhibited high critical error rates (approximately 15%), often ranking lower-quality embryos above viable ones. Significant intermodel variability was observed even among models with similar predictive accuracies (area under curve approximately 60%). When tested on data from a different fertility center, model instability increased (error variance delta: 46.07%²), highlighting sensitivity to distribution shifts. Interpretability analyses revealed divergent decision-making strategies among replicate models, despite identical architectures and training protocols.

Conclusion:

Single instance learning AI models for IVF embryo selection exhibit substantial instability and inconsistency, undermining their clinical reliability. High intermodel variability and critical error rates raise concerns about their suitability for real-world deployment. This study highlights the need for more stable AI frameworks and robust evaluation metrics tailored to the clinical demands of IVF.

INTRODUCTION

Trust is a fundamental driver in the adoption of breakthrough technologies, including solutions based on artificial intelligence (AI). Public opinion is heavily influenced by perceived risks such as bias, data safety, and lack of transparency. An equally critical, although often overlooked, aspect is AI model stability, which can lead to inconsistent and unpredictable performance (1). Such variability in medicine undermines the confidence that clinicians and patients place in AI technologies.

In assisted reproduction, the primary focus has been on creating high-performing models that predict which embryos are most likely to lead to successful outcomes such as pregnancy and live birth. These models analyse static or time-series images of embryos and are used to determine the order in which embryos are transferred clinically (2–6). Model stability and its implications in clinical decisions, such as which embryo is chosen for transfer, have been inadequately studied. Despite AI-based in vitro fertilization (IVF) solutions already being in clinical trials, one study found that eight different commercial AI algorithms exhibited significantly lower agreement with embryologists and with each other when ranking embryos than anticipated (7). In fact, two of the AI algorithms produced orders comparable to random chance. This is concerning, as the choice of algorithm can substantially impact the effectiveness of the AI tool, leading to altered patient outcomes. Although these discrepancies might be attributed to a lack of generalizability between centers, the study’s approach of adapting each model to clinic-specific data before evaluation suggests that there could be other fundamental issues which may be contributing to this variability. It also raises a critical question: how reliable are these AI models, and are we developing and evaluating them appropriately for clinical tasks? To date, studies in IVF have never focused on potential points of failure between models and have limited all evaluations of robustness to generalizability to unseen data.

Therefore, in this study, for the first time, we systematically investigate the stability of single instance learning (SIL) conventional convolutional neural networks trained for live-birth prediction and their implications on the clinical task of embryo rank ordering (Fig. 1A) (Supplemental material for definitions). These models are commonly used in research and commercial platforms. Importantly, this study does not introduce a new model; rather, we evaluate whether existing model frameworks produce consistent embryo rankings when trained repeatedly under slightly different starting conditions (known as seeds).

FIGURE 1.

Development of single instance learning (SIL) models for live-birth prediction and rank ordering. (A) The schematic illustrates the workflow for embryo viability assessment and rank ordering using current deep learning models. Clinically, single embryos from a patient cohort are evaluated by live-birth prediction models, which assign probability scores to each embryo. The embryos are then ranked on the basis of these scores. In this study, model replicates were evaluated to study the reliability of the models and their eventual predicted rank orders. (B) The plot highlights the variability between replicate SIL models in predicting live-birth outcomes of embryos on the basis of their morphological quality. The accuracy and AUC metrics of the models mask the high variability observed in the sensitivity and specificity of predictions (n = 50 seed replicates; 172 embryos). Error bars represent the range. (C) The graph indicates the change in model performance with increasing dataset size. Overall, the model performed similarly with an increase in training data size. The SIL models evaluated in the study utilized 100% of the training data (n = 1,737 embryos with known live-birth outcomes). The reported accuracy here reflects the performance on the test set (n = 172 embryos) of 15 models for all categories except the 100% training data category which reflects the performance of 50 models. The error bars represent the range. (D) The graph compares models trained from scratch with randomized weight initialization to models trained with pretrained ImageNet weights (n = 50 models). The models, regardless of the training strategy, demonstrated similar performance and variability on the test set (n = 172 embryos). The error bars represent the range. AUC = area under the curve.

METHODS

Dataset preparation

The primary dataset comprised 10,713 embryo images from 1,258 patients at the Massachusetts General Hospital (MGH) Fertility Center, collected at 110 ± 3 hours postinsemination (day 5) using embryoscope systems (Supplemental material, available online). We prioritized day 5 embryos as the majority of static image-based AI models in the literature, and nearly all those approaching commercial deployment, are trained and validated on day 5 blastocyst images (4, 5, 8, 9). Prior studies consistently report that models trained on day 5 images outperform those based on day 2 or 3 images in both accuracy and clinical utility (10, 11).

All uses of this dataset were approved by institutional review boards (IRB#2022P002955, IRB#2022P001222) and followed relevant ethical guidelines. An independent test dataset of 648 embryo images from 53 patients was obtained from the Weill Cornell Fertility Center under a data transfer agreement (#2022A016564); only patients with day 5 transfers were included. The MGH data were used exclusively for model training and validation, whereas both MGH and Cornell data were used to evaluate performance on a distinct external cohort. Summary statistics on the datasets and details on model training are provided in the supplementary information and methods, respectively (Supplemental Tables S1 and S2, available online).

The datasets were kept fully separate, and no pooling or retraining was performed. Embryos were labeled on the basis of known transfer outcomes. Those resulting in live birth were marked positive; those that did not were labeled negative. The SIL models were trained using these annotated data. Inclusion in the study was not based on preimplantation genetic testing status, and neither preimplantation genetic testing results, nor embryo grades were used during model training.

Rank variability evaluations

In this study, replicate models were generated by changing the seed, a number that controls how random processes begin during model training. Although the model architecture and data remained identical, changing the seed causes small variations in how the model is initialized. In principle, if an approach is truly stable, these small changes should not meaningfully affect its recommendations. Fifty replicate models using the same architecture and training data were generated, but with different seeds (more details on training in the Supplemental material). Embryo rank orders were generated for patient test datasets from both MGH and Cornell. Only patients with at least four embryos were included in ranking analyses, as smaller cohorts offer limited rank permutations and reduce the interpretability of model disagreements. Additionally, because models cannot meaningfully evaluate empty well images, these images were omitted in rank order evaluations. The final test datasets used by SIL comprised 92 patients for MGH and 49 patients for Cornell. The SIL rank orders were generated using the softmax output, indicating the probability of a live-birth outcome. All embryos within a patient set were evaluated, and rank orders were generated with the embryo having the highest potential for live birth placed at the top.

Kendall’s W, also known as Kendall’s coefficient of concordance, allows to measure agreement among raters and was used to assess the consistency of rank orders generated by different replicate models for each patient (12). A value of 0 indicates no agreement, whereas a value of 1 indicates perfect agreement. Values between 0 and 1 represent varying degrees of agreement, with higher values denoting stronger agreement. Average W values were calculated to assess the overall consistency of each approach across the different test sets.

Transfer rate and live-birth rate evaluations

To evaluate the performance and consistency of the model predictions in clinical settings, the transfer rate and live-birth rate were calculated by aggregating and ranking model predictions for each patient across 50 seeds.

The transfer rate was calculated as the proportion of patients for whom the top-ranked embryo, on the basis of model predictions, was selected for transfer by the clinician. This metric assessed the alignment between the model’s top choice and the clinician’s decision. Specifically, it measured the frequency with which the embryo identified by the model as having the highest potential for a successful outcome matched the embryo actually selected for transfer. When N_match represents the number of patients where the model’s top-ranked embryo was actually transferred, and N_total is the total number of patients, the transfer rate T is given by,

$$T=\frac{N_{match}}{N_{total}}\times 100$$

The live-birth rate was determined by the success rate of live births from the transferred embryos. This metric focused on the outcomes of the embryos selected for transfer and was calculated as the proportion of transferred embryos, identified as top-ranked by the model, that resulted in a live birth. When N_L is the number of live births from transferred top-ranked embryos, and N_TT represents the total number of transferred top-ranked embryos, live-birth rate L is given by,

$$L=\frac{N_L}{N_{TT}}\times 100$$

Known clinical outcomes were available only for embryos that had been transferred. Embryos that were not transferred did not have outcome labels and were excluded from accuracy and live-birth rate analyses. As a result, transfer rate reflects how often the model’s top choice matched the clinician’s selection, whereas live-birth rate captures the proportion of those top-ranked, transferred embryos that resulted in a live birth.

Critical error rate evaluations

A modified Gardner grading system was used for day 5 embryo annotations (3, 13, 14). Briefly, classes 1 and 2 represent arrested and morula stage embryos, whereas classes 3, 4, and 5 represent different grades of blastocysts, with class 5 being the highest quality blastocyst (>3CC). To reduce variability from interlaboratory grading practices, all embryo quality grading for both datasets was performed retrospectively by a single team of embryologists at MGH, and the mode of their annotations was used. These annotations were used solely for evaluating model behavior and were not involved in model training.

In this study, critical errors are defined as instances where a grade 1 embryo (degenerate) is selected as the top pick (highest rank) despite the availability of a grade 3 (early blastocyst) or higher quality embryo in the set. Embryo rank orders were generated for patient test datasets from both MGH and Cornell. In addition to the eligibility criteria, an additional requirement was enforced: each embryo cohort had to include at least one blastocyst. Nongradable embryos and embryos with no mode of quality grade information were also removed. Consequently, the final datasets used by SIL models comprised 83 patients for MGH and 45 patients for Cornell. Critical error rates were determined by calculating the frequency of grade 1 (degenerate or arrested) embryos being ranked as the top embryo for each model when evaluating both the MGH and Cornell datasets. Critical error rate, C, when N_error represents the number of patients with a grade 1 embryo as the top-ranked embryo and total number of patients evaluated is represented by N_patients, can be given by,

$$\text{C}=\frac{N_{error}}{N_{patients}}\times 100$$

RESULTS

Variability in conventional single instance prediction models

Our observations, on the basis of training multiple SIL replicate models to predict live-birth outcomes (n = 50 models) with only varying seed initialization weights, revealed that performance metrics like area under the curve (AUC) and accuracy can obscure substantial variations in the model’s decision-making process (Fig. 1B). The models performed with an average (SD; min-max) performance across accuracy, AUC, F1, sensitivity, and specificity of 58.73% (4.576%, 49.42%–68.60%), 60.02% (4.727%, 49.47%–71.78%), 43.07% (10.22%, 6.061%–60.40%), 46.95% (16.09%, 3.279%–70.49%), and 65.21% (13.55%, 44.14%–98.20%), respectively (n = 50 models; 172 embryos) (Fig. 1B). Studies in the AI-IVF literature generally do not focus on this model variability aspect because, for deployment, the single best-performing model is ultimately selected for clinical use. However, this broad variability is concerning, as it suggests potential instability in the models. Notably, this variability was not significantly impacted by the size of the training dataset or by using pretrained weights (Fig. 1C and D, Supplemental Fig. S1A, available online). Testing with ImageNet weights confirmed these concerns, as retraining and finetuning led to similar levels of variability (Fig. 1D). This suggests that during clinical deployment, AI models may exhibit unpredictable behavior when updated with additional embryo data, raising concerns about their reliability.

Variability in single instance rank ordering

When evaluating the SIL model-generated rank orders using the MGH test set, it was observed that the single instance outcome prediction does not translate to effective rank ordering of embryos (Fig. 2). Models with similar accuracies and AUCs produced completely different rank orders when evaluating 92 patient embryo cohorts from MGH (Fig. 2, Supplemental Table 3). In Figure 2, the rank orders for all 92 patient sets are presented, with models grouped according to the similarities in their generated rank orders. No association was observed with commonly used model selection metrics such as accuracy and AUC on the validation dataset (n = 191 embryos) (Fig. 2). The embryo rank orders generated by the 50 SIL models, when assessed for agreement using Kendall’s coefficient of concordance (Kendall’s W), yielded a low average W value of 0.3571 ± 0.1302 (mean ± SD) across the 92-patient embryo sets and similar results were observed with modified training protocols (Fig. 2, Supplemental Fig. S1B). Using additional data from Cornell, collected on a newer embryoscope model, we further tested these models, without finetuning, which also revealed inconsistencies, with an average Kendall’s W value of 0.3410 ± 0.1398 across 49 patients (Fig. 3A–C, Supplemental Table S4).

FIGURE 2.

Variance of single instance learning (SIL) models in clinical rank ordering. The heatmap visualizes the rank orders provided by the SIL replicate models (n = 50; 92 patients). Each column represents the rank order of a single model, with each row representing a single embryo in a patient set. Embryos are serially listed covering all 92 patient sets and the color indicates the relative rank within each set. Models are clustered on the basis of the similarity of the embryo rank orders. The line graph provides the AUC and accuracy of the respective SIL model. A single patient set, patient #37, is provided in an expanded format to better visualize the rank orders generated by the different SIL models. AUC = area under the curve.

FIGURE 3.

Rank ordering performance of SIL models with the MGH and Cornell test dataset. (A) The images are examples of unprocessed data of embryos available from MGH. The dataset comprises 250x250 images of embryos collected at 110 hpi using an Embryoscope-D system. The well ID information and timestamps were cropped out before utilization in model development and evaluation. (B) The images are examples of unprocessed data of embryos available through Cornell. The dataset comprises 800x800 images of embryos collected at 110 hpi using an Embryoscope+ system. (C) The graph provides a measure of randomness across 92 MGH patients and 49 Cornell patients. Kendall’s W is used to evaluate the consistency of embryo rank orders produced by the different SIL models (n = 50 models). A higher Kendall’s W value indicates greater consistency in the rank orders for each patient. It ranges from 0 (no agreement) to 1 (complete agreement). The 95% confidence intervals [CI] for MGH (mean: 0.3571; CI: 0.3302 to 0.3841) and Cornell (mean: 0.3410; CI: 0.3008 to 0.3811), respectively. (D) The bar graph indicates the critical error rates of each SIL model, with the models sorted on the basis of their performance on the MGH validation dataset. Rank orders that result in the selection of arrested or degenerate embryos when at least one high-quality blastocyst is present in the set is defined as a critical error event. Blue bars indicate the critical error rate when evaluating patient sets from MGH (n = 83) and orange bars indicate the error rate when evaluating patient sets from Weill Cornell (n = 45). The average error rates for MGH and Cornell were 12.4 ± 4.7% (mean ± SD) and 17.29 ± 8.3% (mean ± SD), respectively. MGH = Massachusetts General Hospital

The poorest quality embryos, such as degenerate or arrested embryos, have the lowest chance of success and are associated with a higher incidence of chromosomal abnormalities (15–17). Therefore, in this study, we focused on the frequency at which the poorest quality embryos were incorrectly selected as the top-ranked embryo, despite the presence of at least one viable blastocyst in the cohort (see Methods). We considered these as critical errors and compared their frequencies among the replicate models. The 50 models, when tested with eligible MGH data, yielded an average error rate of 12.41% (range: 3.61%–21.69%) (n = 83 patients). Similar to the lack of correlation between rank orders and accuracy, critical errors were not predicted by model performance in instance-level outcome classification, such as validation accuracy, AUC, F1 score, and loss (Fig. 3D, Supplemental Fig. S2). Evaluations with the Cornell dataset had an average error rate of 17.29% (range: 4.44%–37.78%) across 45 eligible patients (P=.0005; two-tailed unpaired t-test) (Fig 3D). In addition to the increase in the average error rate, the variance between the 50 models was significantly more pronounced when evaluating Cornell data (68.59%²) compared with MGH (22.53%²) (Fligner-Killeen test; med χ² statistic=10.73, P=.0011). Moreover, model error rates did not scale consistently between MGH and Cornell (Δ = 46.07%²); high or low errors at MGH did not reliably predict similar outcomes at Cornell. The increased variance was likely amplified by differences in imaging equipment, laboratory protocols, or patient populations across centers, as current AI models have been demonstrated to adapt inconsistently to changes in clinical environments (14).

Intermodel Interpretability evaluations

Clustering analysis on the basis of the similarity of feature utilization, revealed that feature selection was not correlated with predictive performance and all models highlighted entirely different features and regions of the embryo (Fig. 4A, Supplemental Fig. S3, and S4). For the embryo analyzed in Figure 4A, the average cosine distance between the feature vectors of different model pairs was 1 ± 0.06 (Range: 0.81–1.20), indicating a high degree of dissimilarity among the models. Furthermore, even for the two highest-performing models on the validation set (Model 49 and Model 50), which had the same validation accuracies, the cosine distance between their feature vectors was 1.04.

FIGURE 4.

Local and global interpretability of SIL models. (A) The dendrogram shows SIL models clustered by their feature representations when evaluating a single embryo image. Cosine distance, ranging from 0 (identical) to 2 (maximally dissimilar), is a measure of similarity between two vectors. Similar performing models can use regions that do not overlap with each other in their respective decision-making processes. (B) The t-SNE plot illustrates the feature representations of all embryos from a single patient across 50 SIL models in a 3D space. The lack of clustering indicates that different models use dissimilar features for the same embryos. 3D = three dimensional, SIL = single instance learning, t-SNE = t-distributed stochastic neighbor embedding.

Additionally, we applied t-distributed stochastic neighbor embedding (t-SNE) to cluster the feature associations made by all 50 SIL models across entire patient embryo cohorts (Fig. 4B, Supplemental Fig. S5). The t-SNE analysis revealed that the models struggled to consistently cluster the features for a set of embryos from a single patient, indicating limited overlap in the features each model focused on across all tested embryos. The t-SNEs generated for the MGH patients (n = 97) across all 50 SIL models resulted in an average intra and intercluster distances of 11.74 ± 3.33 and 86.03 ± 27.10, respectively. The intracluster distance reflects how tightly each model groups features within a patient’s embryos; lower values suggest the model is internally consistent. The intercluster distance measures how different the feature groupings were between models; larger values indicate the models saw the same embryos very differently. The high variability, particularly in intercluster distances (SD: 27.10) across patients, suggests that model clusters interact differently with patient-specific data.

DISCUSSION

Reliable AI models have the potential to significantly enhance decision-making in IVF. Current AI tools in IVF use SIL to predict the probability that an individual embryo will implant or yield a live birth (2). They are tuned to achieve the best possible predictive performance, often measured with common statistics such as area under the receiver operating characteristic curve (AUC; how well the model separates embryos that will vs. that will not lead to an outcome) or simple outcome-prediction accuracy. In this study, SIL models trained on day-5 blastocyst images achieved predictive performance comparable with previously reported AI models, which typically demonstrate AUC values ranging between 0.56 and 0.70 when predicting implantation or live-birth outcomes, and therefore our models are representative of current-generation IVF AI tools (5, 8, 18–21).

Clinically, SIL-based individual scores, i.e., the model’s estimated likelihood of live birth for each specific embryo, are used to determine the order of embryo transfer in an IVF cycle (2–6). Although different AI models can show similar overall predictive numbers, the internal reasoning or decision-making pathways they use may differ significantly (7, 22). Although intuitively these differences may seem minor, they can have a substantial impact clinically and it is critical to be aware of these differences and their effect on patient care. No study to date has rigorously tested whether AI models actually provide consistent and reliable embryo rankings, arguably a more relevant measure during routine use. Our findings demonstrate considerable variability among replicate AI models, highlighting an important gap between predictive accuracy and real-world reliability. Our analyses demonstrated that replicate models when using the same training data, model design, and similar-accuracy scores (SD: 4.6%), often made predictions on the basis of different visual patterns within the embryo images. Each model had developed substantially different interpretations of what makes an embryo promising, despite being given the same information and even models with the exact same accuracy used very different features and thus produced very different rank orders. This is similar to having multiple embryologists sort the same set of embryos by quality, each using a slightly different, and sometimes conflicting, set of criteria. Thus, models that appear equivalent by accuracy or AUC are not clinically interchangeable.

Common evaluation metrics, such as accuracy or AUC, provide limited guidance for ranking tasks. A key challenge is the lack of a definitive ground truth, making it impossible to conclusively identify which, if any, of the SIL models provide the correct embryo rankings. This uncertainty complicates model selection for clinical use. Therefore, developing and utilizing clinically meaningful selection criteria is essential. One promising strategy, as employed in this study, involves retrospective evaluation focusing specifically on the top-ranked embryo selections. This method aligns closely with clinical practice, as errors involving top-ranked embryos are substantially more impactful than discrepancies lower in the ranking order. By concentrating on these clinically meaningful top-rank errors, we can better understand the direct consequences of AI variability in patient care.

The critical errors identified in this study reflect model behaviors that would be unacceptable in clinical practice. Embryologists do not typically select degenerate or arrested embryos for transfer when viable blastocysts are present. Yet, the models tested here frequently ranked such embryos highest, despite being trained on outcome-labeled data. These misclassifications are not rare edge cases but systematic failures observed in a substantial fraction of patient cohorts. This inconsistency means that two seemingly identical models could recommend very different embryos as the “best” choice for transfer, potentially impacting patient outcomes. Even as a clinical decision support system, this level of inconsistency could lead to delays to inefficient allocation of clinical resources, disruptions in tightly scheduled IVF cycles, workflow delays and ultimately erode trust in AI guidance, especially because clinicians may not be aware of the internal differences driving these inconsistent recommendations. This issue becomes more pressing when considering the wide diversity across fertility centers, where differences in imaging technologies, patient demographics, and laboratory practices are common (22, 23). Models trained in one clinical environment may perform unpredictably when applied in another setting, complicating standardization and guideline development across the IVF field. Our findings also suggest that relying solely on standard efficiency-focused clinical trials may not be sufficient for validating AI systems in IVF, as these trials typically assess individual models without evaluating their robustness or variability (9, 24). Even a model that performs well in initial clinical studies might yield inconsistent or suboptimal results when retrained or adapted with new data. Given the inherent variability in IVF data, future evaluations should specifically test for the reproducibility and stability of model decisions, ensuring that models remain reliable and clinically useful over time and across diverse clinical settings.

The ability to ensure consistent and unbiased performance across different patient populations is another critical dimension affected by model instability. When AI models demonstrate substantial variability in their decisions, identifying and addressing patterns of unintended bias becomes especially challenging. If a model is inconsistent, it may unfairly disadvantage certain patient groups without clinicians even realizing it. This problem is worsened by the limited transparency often associated with commercial AI tools, where details about training protocols, underlying datasets, and patient demographics are typically undisclosed. Without such information, clinicians cannot confidently determine if a particular AI model fairly represents and adequately serves their patient populations. Therefore, better standardization of model development and evaluation information is necessary to improve trust and reliability in AI-assisted IVF technologies. Given that widely accepted benchmark datasets in IVF are uncommon, the implementation of such datasets could help drive standardization in the field.

CONCLUSION

Our findings highlight the risks associated with unstable models, which can lead to inconsistent and unpredictable results and erode the trust of clinicians and patients. Many IVF AI reports do not examine replicate models, yet our results show that robustness and stability are as important as headline performance metrics. Although AI systems may predict individual outcomes reasonably well, this does not guarantee that they will provide clinically reliable embryo rank ordering (25, 26). This gap can mislead clinicians into believing they are making informed, model-guided choices when rankings are actually unreliable. Taken together with our replicate SIL findings of similar-accuracy models that sometimes placed poor-quality embryos above viable blastocysts, the limitations of current evaluation practices become clear. Clinical adoption should require evidence of ranking consistency, analysis of critical top-rank errors, and proof of robustness and fairness across sites and patient populations.

Supplementary Material

Supplemental data for this article can be found online at https://doi.org/10.1016/j.fertnstert.2025.08.021.