Benchmarking large language models for predictive modeling in biomedical research with a focus on reproductive health

Summary

Large language models (LLMs) are increasingly used for code generation and data analysis. This study assesses LLM performance across four predictive tasks from three DREAM challenges: gestational age regression from transcriptomics and DNA methylation and classification of preterm birth and early preterm birth from microbiome data. We prompt LLMs with task descriptions, data locations, and target outcomes and then run LLM-generated code to fit prediction models and determine accuracy on test sets. Among the eight LLMs tested, o3-mini-high, 4o, DeepseekR1, and Gemini 2.0 can complete at least one task. R code generation is more successful (14/16) than Python (7/16). OpenAI’s o3-mini-high outperforms others, completing 7/8 tasks. Test set performance of the top LLM-generated models matches or exceeds the median-participating team for all four tasks and surpasses the top-performing team for one task (p = 0.02). These findings underscore the potential of LLMs to democratize predictive modeling in omics and increase research output.

Graphical abstract

Highlights

• •Large language models (LLMs) generate R and Python code for omics prediction tasks
• •Top LLM matched or exceeded median DREAM challenge participant performance
• •LLMs allowed model development in minutes starting with tabular data
• •Single-shot prompting reveals strengths and limitations of LLM-based code generation

Sarwal et al. benchmark large language models (LLMs) on four reproductive health prediction tasks from DREAM challenges. Several LLM-generated executable workflows match median participant accuracy and, in one case, outperform top teams. This demonstrates that LLMs can successfully assist with developing predictive machine learning models, while still requiring human oversight.

Introduction

Reproductive health is a critical area of study that encompasses fertility, pregnancy, and childbirth, all of which have profound implications for individual and public health. Among these concerns, preterm birth—defined as delivery before 37 weeks of gestation—remains a major global challenge, affecting approximately 11% of infants worldwide and leading to significant short- and long-term health consequences.¹ Understanding and addressing issues in reproductive health is essential for developing effective prevention strategies and promoting maternal and infant well-being. In recent years, predictive models in reproductive medicine have been developed to estimate the likelihood of adverse pregnancy outcomes, supporting personalized care and informed decision-making.² However, the performance of these models depends heavily on the quality and size of the datasets used, the distribution and types of data available, and the specific methods applied.³ Continued research in reproductive health, supported by robust data and innovative technology, is key to improving outcomes across the reproductive lifespan.

Recent advances in maternal health research underscore the critical importance of accurately estimating gestational age⁴ and understanding the molecular mechanisms underlying placental aging.⁵ However, current clinical tools such as fetal ultrasound remain imprecise, often estimating gestational age with errors of several weeks^6,7 and only 55.1% delivering within 1 week of their due date.⁸ Moreover, there are no reliable predictive tools in practice to assess the risk for preterm delivery or effective treatments to prevent it, besides vaginal progesterone for the subset of women with a short cervix.^9,10,11 Accurate assessment of gestational age and preterm birth risk would enable better evaluation of fetal maturity and targeted interventions to reduce prematurity risk. Precise determination of gestational age is essential for optimizing clinical management, guiding timely interventions, and reducing adverse outcomes for both mother and child. Equally, insights into the “placental clock”—the epigenetic and molecular processes that regulate placental aging—provide invaluable clues to fetal development, maternal adaptation, and long-term health trajectories.¹² Crowdsourced open science initiatives, exemplified by challenges such as the DREAM (Dialogue for Reverse Engineering Assessments and Methods) challenges,¹³ harness global collaboration and diverse datasets, including transcriptomic,¹⁴ microbiome,¹⁵ and methylation data¹⁶ to develop predictive models addressing these critical questions. In particular, DREAM challenges have framed tasks directly aimed at these clinical gaps, using blood transcriptomics to predict gestational age, placental methylation data to model placental aging, and vaginal microbiome profiles to forecast preterm birth risk, highlighting the translational potential of data-driven discovery. While such collaborative approaches offer access to a broad spectrum of data, computational methodologies, and collective expertise, they also face challenges related to coding variability and inconsistent model performance.

Large language models (LLMs) are a promising solution to issues of inconsistency. Cutting-edge systems such as GPT-4, DeepSeek, and others have evolved rapidly, and their applications find critical utility in biological and medical sciences where the challenge of “too much data, too few experts” has long hindered progress.^17,18 By utilizing vast corpora of text-based data—spanning scientific literature, experimental results, and patient records—LLMs are enabling breakthroughs in cancer diagnosis, radiology, electronic health record analysis, hypothesis generation, and more (PathChat,¹⁹ Flamingo-CXR,²⁰ Med-PaLM²¹). Another application of LLMs is to democratize data analysis to non-expert programmers, streamline analysis, and accelerate the discovery and interpretation process in biomedical data sciences. This has been followed by the widespread adoption of LLMs in all parts of the scientific process. Alongside this exciting explosion in LLM usage comes the need for careful and meticulous evaluation, and researchers have highlighted both their potential for breakthrough discoveries and their potential pitfalls.²²

Herein, we utilized the wealth of crowdsourced benchmark data from DREAM challenges to assess the ability of LLMs to support the development of code for building predictive models with omics data, and then assess and visualize model performance. Each DREAM challenge consisted of a research question coupled with training and test data. Teams from around the world submitted solutions to tackle a problem based on training data, which were then evaluated on a blinded test set, mirroring the train-test process in machine learning. We prompted various popular LLMs to mirror a typical workflow of a bioinformatician who writes R or Python code to read data from local files or web-based repositories, to build and evaluate a prediction model for multiple tasks. The data types considered included transcriptomic, epigenetic, and microbiome, spanning several orders of magnitude in terms of problem dimensionality. We systematically assessed the LLMs’ performance in a single-shot setting using 4 prediction tasks: gestational age prediction from blood gene expression or placenta methylation data and preterm or early preterm birth classification from microbiome data. Executing and evaluating the code generated by LLMs, we derived insight in terms of coding language-specific reliability of the LLM-generated code and their ability to match the test set prediction accuracy of participants in the original DREAM challenges. Our study reframes the usage of LLMs in the context of crowdsourced research efforts, highlighting their ability to generate reproducible machine learning pipelines that reduce inconsistencies and improve efficiency in a multi-team collaborative setting.

Results

Study overview

The methodology used in this study is depicted in Figure 1. Briefly, each of the 8 LLMs considered (Table S1) was prompted to generate R or Python code that uses one training dataset that (1) fits a model for the given task, (2) applies the model to the corresponding test set to calculate an appropriate performance metric, and (3) generates a visualization of the result (Table S2). These prompts specified the type and source of data, dimensionality of the feature space, prediction outcome, data partitioning into training and test sets, and required model evaluation metrics (root-mean-square error [RMSE] for continuous outcomes and area under the receiver operating characteristic curve [AUROC] for binary classification). The prediction tasks used to assess LLMs were based on datasets from multiple biomedical domains, each representing distinct analytical challenges (Figure 1; Table S3). The first dataset (Q1) involved transcriptomics data obtained from the Gene Expression Omnibus (GEO), requiring parsing and preprocessing before downstream analysis. The second dataset (Q2) focused on epigenetics, specifically methylation data for the prediction of placental gestational age using regression models. The third dataset (Q3) was used for two classification tasks based on relative microbial abundance. Each of these prediction tasks tested the ability of LLMs to retrieve and organize data, foresee the need for data conversion, identify a suitable modeling framework, and select the appropriate package for fitting those modeling strategies across different programming environments.

Figure 1.

Study flow chart

Eight LLMs were prompted to generate R and Python code to fit and evaluate predictive models for four tasks. Q1: predict gestational age from blood transcriptomics. Q2: predict placental age from DNA methylation data. Predict (Q3A) preterm birth or (Q3B) early preterm birth from microbial relative abundance data. Analysis code was executed, and LLMs were ranked by code functionality and model accuracy. Created with BioRender. See also and Tables S1–S3.

Model evaluation

The test set accuracy metrics obtained by executing the code generated by LLMs for all tasks and programming languages are shown in Figure 2 and Table S4. Comparisons of predicted vs. actual values for Q1 and Q2 and true-positive vs. false-positive rates for Q3 are shown in Figure 3. The empty cells in Figures 2A and 2B signify that the code generated by the given LLM resulted in an error, preventing the completion of the task. Reasons for failures during the execution of LLM-generated code included attempts to load non-existent packages for R/python, inability to download data from the GEO repository or extract required metadata (specifically for python as Bioconductor/R package vignettes provide code examples for GEO data loading and parsing), attempts to select variables that did not exist in parsed datasets, failure to merge feature data with sample metadata before model fitting, and errors in the function calls for model fitting or plotting of results. Overall, among the four LLMs that generated code that successfully completed any task (o3-mini-high, 4o, DeepSeekR1, and Gemini), the R code generated by the LLMs was more successful in completing the four tasks (14/16) compared to Python (7/16). In three instances, LLMs created R code that successfully generated models and test set predictions but failed to produce code to plot results. Due to errors in retrieving gene expression data or parsing its corresponding metadata from GEO (Q1) in Python, none of the LLMs were able to complete this task. However, due to the efficient implementations of high-dimensional models (such as Ridge Regression), the RMSE in Python for task Q2 obtained by 4o and DeepSeekR1 was lower (i.e., higher accuracy) than that of the best-performing team in the Placental Clock DREAM challenge (Q2 RMSE: LLM top 1.12 vs. participant median 2.5 and top participant 1.24 weeks). These results align with previous studies highlighting the effectiveness of high-dimensional regression models in biomedical applications.²³ The top LLM prediction metrics for the other three tasks were comparable to or better than those of the median performance across DREAM challenge participants, but worse than those of the top teams (Q1 RMSE, LLM top = 5.42 vs. participant median 5.4 and top 4.5; Q3A AUROC, LLM top = 0.57 vs. participant median 0.58 and top 0.68; Q3B AUROC, LLM top = 0.59 vs. participant median 0.58 and top 0.92) (Figure 2; Table S4). Top participant models were significantly more accurate than top LLM models for Q1 (p < 0.001) and Q3B (p = 0.008) but not Q3A (p = 0.08), while the top LLM model was more accurate than top participant model for Q2 (p = 0.02).

Figure 2.

Test set prediction results

(A and B) RMSE and AUROC metrics were generated by executing LLM-generated Python (A) and R (B) analysis code. Colors are normalized by dataset, best scores are indicated by the darkest color per column, and the best score per dataset and language is shown in bold. Gray cells indicate that the code generated an error, preventing the completion of the task. Asterisks indicate that the model did not successfully generate a plot of the results. Models listed in Table S1 but not included here did not generate any successful code.

(C) The best and the median model accuracy achieved by teams in the corresponding DREAM challenges.

(D–G) Bar plot representations with 95% confidence intervals of the same performance metrics in (A)–(C) except that median DREAM challenge participant metrics are shown as red dotted lines instead of a bar. p values indicate the significance of differences between performance metrics of the top participant in the DREAM challenges and the top LLM for each task. Underlying data are presented in Table S4, and results for three additional technical replicates for o3-mini-high are included in Table S5.

Figure 3.

ROC curves and prediction scatterplots

(A) Predicted vs. actual gestational ages for all successful LLMs for Q1.

(B) Predicted vs. actual placental gestational ages for the top-scoring LLM for Q2 (left) and lower-scoring LLMs (right subplots). 4o- and DeepseekR1-generated code output identical predictions for (A) and (B) and are plotted together in purple.

(C) Receiver operating characteristic (ROC) curves for all LLM-generated predictions for Q3A endpoint.

(D) ROC curves for all LLM-generated predictions for Q3B endpoint.

In (C) and (D), the three highest scoring models output identical predictions and are represented by the bold green line. Confidence intervals for RMSE statistics were obtained using bootstrap (1,000 iterations), while for the area under the ROC (AUROC), they were obtained using the DeLong method. See also Table S4.

The ability of LLMs to generate R/python code that could be run successfully, resulted in accurate test predictions, and generated a visualization of the predictions was the basis for scoring and ranking of the LLMs (STAR Methods; Table 1). Out of all the LLMs we tested, o3-mini-high received the highest overall score (33), followed by 4o (23), DeepSeekR1 (22), and Gemini2.0 FlashThinking (19) (Table 1). Of note, obtaining code from successful LLMs typically took between a few seconds and 2 min, which is orders of magnitude faster than human participants could develop comparable code and significantly shorter than the 3 months allotted for submissions to the DREAM challenges. In general, LLM generated code was about one page per task and was easy to follow as it often included comments before each code chunk. The most challenging task in terms of data size (Q2, ∼360,000 methylation features and ∼2,000 samples for training) was handled particularly efficiently by the most successful LLMs (DeepseekR1 and 4o), both implementing a different version of ridge regression. The LLM code execution took a few hours rather than dozens of hours needed for fitting models by the top three teams in the Placental Clock DREAM challenge. The impact of the stochastic nature of analysis code generation by the top-performing LLM (o3-mini, high reasoning mode) was assessed by prompting three additional times the model with the same prompt. The success rate of completing the analysis tasks was identical (87.5%) to the one reported in the primary analysis (Figures 2A and 2B; Table S5), yet fluctuations in accuracy were noted as different types of predictive models or tuning techniques were used. The accuracy reported in the initial analysis for the 7 completed tasks in Figures 2A and 2B matched the average accuracy over the 3 trials in 5/7 tasks, exceeded it in 1/7, and underestimated it in 1/7 tasks.

Table 1.

LLM scoring

Model	Python code				R code				Model Score
	Q1 a|b|c|d	Q2 a|b|c|d	Q3A a|b|c|d	Q3B a|b|c|d	Q1 a|b|c|d	Q2 a|b|c|d	Q3A a|b|c|d	Q3B a|b|c|d
o3-mini-high	0	1|2|0|1	1|2|1|1	1|2|0|1	1|2|1|1	1|2|1|1	1|2|1|1	1|2|1|1	33
4o	0	1|2|1|1	0	0	1|2|1|0	1|2|1|0	1|2|1|1	1|2|1|1	23
DeepSeekR1	0	1|2|1|1	0	0	1|2|1|1	1|2|1|0	1|2|0|1	1|2|0|1	22
Gemini	0	0	1|2|0|1	1|2|1|1	0	0	1|2|1|1	1|2|1|1	19
DeepSeekDistill	0	0	0	0	0	0	0	0	0
Llama	0	0	0	0	0	0	0	0	0
Phi-4	0	0	0	0	0	0	0	0	0
Qwen	0	0	0	0	0	0	0	0	0

For each task (Q1, Q2, Q3A, Q3B), models were evaluated for their ability to produce Python and R code that successfully (1) extracted and formatted data (1 point); (2) trained a model, applied it to the test set, and calculated the performance metric (2 points); (3) generated the required plot (1 point); and (4) achieved an accuracy that was within one significant digit of the best model for the endpoint (1 point). The maximum score for a+b+c+d was 5 points (1+2+1+1).

Discussion

Our benchmarking study leveraging omics data from DREAM challenges highlights both the promise and limitations of LLMs in biomedical machine learning applications. Similar benchmarking approaches have been employed to assess LLM performance in other domains, including chemistry and clinical medicine.^24,25 Herein, we found that API-based LLMs such as o3-mini-high and 4o outperformed smaller locally run models in execution success and accuracy, reinforcing the advantages of larger cloud-hosted architectures. However, local models offer increased control and customization, which may be advantageous for researchers seeking data privacy or fine-tuned execution. Prior research has shown similar trade-offs in LLM performance across different computing environments.²⁶

In terms of test set prediction accuracy of models obtained with LLM-generated analysis code compared to human-generated code, our results suggest that in some instances, LLM-generated models can be more accurate and faster to develop. For task Q2, where the LLM-generated model from ∼350,000 methylation features was more accurate than the human counterpart (RMSE = 1.12 vs. 1.24, respectively), neither the LLM nor the human had access to the test set or had any feedback on their model accuracy based on the test set. For this task, the best human-developed model was based on multi-stage random forest models, each of them utilizing thousands of methylation features, and leveraging additional clinical information, such as the obstetrical complication involved in the pregnancy. Such clinical information was not provided to the challenge participants but was publicly available for the training data. In contrast, the best LLM (OpenAI’s 4o) used a Ridge Regression model including only the placenta methylation features. Moreover, LLMs were neither prompted to use the additional clinical information nor made aware of their availability.

For the other three prediction or classification tasks, the human participants in the DREAM challenges had the advantage of either using additional demographics, feature data, and time points in the models (Q3A and B) or receiving and using feedback on their model accuracy based on the test set accuracy (Q1 and Q3A and B). Typically, participants in DREAM challenges can select as their final submission the model that achieved the highest performance among multiple trials, with 3–5 trials typically allowed. LLM performance was assessed in a single trial.

One of the key motivations for integrating LLMs into the biomedical research process is their ability to support computational tasks by allowing prototyping of analysis code for researchers without coding skills and improving the productivity of those with coding skills. While crowdsourcing has democratized access to datasets and allows assessment of models on blinded data while facilitating global collaboration, it also introduces challenges related to reproducibility, coding variability, and quality control.²⁷ LLMs offer a potential solution by generating standardized, executable code, reducing the variability inherent in manually coded solutions, although some level of variability is inherent also to LLM generation due to their stochastic nature. Furthermore, LLMs can automate complex preprocessing steps, ensuring consistency in data handling across multiple modeling approaches, a known limitation of crowdsourced competitions.²⁸

This is not to say that usage of LLMs does not come with its own limitations. For instance, while this study utilized various types of molecular and lab-test data, these data were all in a tabular format and derived in relation to reproductive health. Thus, it is very possible that these same LLMs, when applied to other data types or conditions, may not perform as well. Similarly, this study prompted LLMs to build the predictive models in very specific ways, and it is likely that LLM-created workflows to build more complex models will not do as well. Since LLM outputs are stochastic by nature, a single generation (i.e., one shot, as we did in our study) is insufficient to reliably capture a model’s actual capacity for code generation. In addition, an important caveat is that LLMs—particularly when trained or prompted in similar ways—may converge on a narrow set of model architectures or analytic approaches. Such convergence could inadvertently reinforce suboptimal solutions rather than encourage methodological diversity. The range of scoring we observed across models in this study suggests that, while there is still some variability in how LLMs approach the task, there is a nontrivial risk that future applications could favor reproducibility of “standardized” code at the expense of innovation or optimal task-specific performance. Explicitly recognizing this trade-off is essential as the field considers how best to integrate LLM-generated workflows into biomedical research.

This was illustrated in this study for task Q1 where 3 models achieved the same accuracy (RMSE, 5.4 weeks). Multiple identical models will lead to ties in the ranking of the participating teams and less diversity in the approaches and ultimately less insight into the question formulated in the DREAM challenge. However, the use of LLMs by scientists in their daily analysis tasks has the potential to streamline model development and avoid over-stating the accuracy of models, particularly in reproductive health research, where accurate and reproducible predictive models can significantly impact clinical decision-making and reliable biomarkers derived from omics studies are scarce. A common source for over-stating the accuracy is the feature selection bias, where most predictive features are selected using the full dataset and only then data are split into training and testing.²⁹ None of the four successful LLMs that generated code attempted to leak information between training and test sets. Future research should explore how LLMs can be further fine-tuned for other biomedical applications and using other data types. Additionally, the integration of LLM-generated models into clinical pipelines should be rigorously evaluated to assess their translational potential and ensure alignment with medical best practices. In parallel, there is a clear rationale for modeling with omics features rather than relying only on routinely collected clinical variables. Large, prospective cohorts show that high-dimensional proteomic or epigenomic signatures capture subclinical biology and can improve discrimination and reclassification for future disease beyond age, vitals, anthropometrics, and standard labs. For example, sparse plasma-protein panels trained in the UK Biobank predicted risk across multiple common diseases and, in several endpoints, outperformed (or added to) models built from conventional clinical factors.³⁰

As a result, combining routinely available clinical data with omics can yield the most useful—and clinically interpretable—models. A well-known example is RSClin in early breast cancer, which integrates a 21-gene expression score with clinical-pathologic variables to provide more individualized recurrence and treatment-benefit estimates than either source alone; analogous clinico-omics integrations are increasingly standard across disease areas.³¹

Finally, LLMs can help leverage routinely collected data by unlocking predictive signal in unstructured notes and imaging. Health-system-scale language models trained on electronic health record text (e.g., NYUTron) have improved multiple prospective predictions relative to strong baselines, while vision foundation models (e.g., RETFound) demonstrate label-efficient transfer to downstream retinal disease tasks—both illustrating how foundation/LLM approaches can enhance routine clinical data and facilitate integration with omics when appropriate.³²

This study provides a comprehensive benchmarking of LLMs for biomedical predictive modeling, evaluating their ability to generate machine learning code across multiple data modalities and coding languages. By addressing key challenges associated with code and models generated from crowdsourcing in biomedical research, LLMs offer a promising avenue for improving reproducibility, standardization, and coding efficiency, yet they may lead to identical models generated by challenge participants who adopt LLMs. While API-based LLMs demonstrated superior reliability, locally run models provide opportunities for enhanced control. LLMs offer significant potential in automating machine learning workflows but require careful validation to ensure reproducibility and accuracy in biomedical research. As a result, while more advanced models emerge, capable of handling various kinds of data, continued oversight and validation of these models’ ability to accurately analyze and interpret such data is essential. Ongoing advancements in LLM architectures and prompt engineering strategies are poised to further refine their utility in scientific computing, including efforts to address major public health challenges such as preterm birth.

Our work demonstrates that LLMs can be integrated into researchers’ predictive modeling workflows by rapidly generating code when provided with precise prompts about data structure, variables, and outcomes. Advantages include faster development and model performance comparable to the participant median across coding languages. Challenges include the cost of advanced LLMs and data security and privacy. Future work should explore agentic AI, as opposed to single-shot LLM prompting evaluated herein. Agentic AI can iteratively refine models but will require careful consideration of secure data access and resource management.

Limitations of the study

Several limitations of this study should be noted. Despite assessing eight different LLMs, the range of options in this area is wide and we only covered a subset of the options available to users. Furthermore, although we covered three data domains, other predictive modeling scenarios could be considered within the genomics field (e.g., SNP data, proteomics). The predictive tasks were limited to the cross-sectional study design even when the original data were longitudinal (i.e., task Q3), and hence there is a need to further evaluate the ability of LLMs to handle more complex tasks such as multiple observations per subject, missing data, and multi-class outcomes. The need to keep the analysis task reasonably simple may have also put LLMs in a disadvantage for task Q3 where human counterparts used additional microbiome features and multiple samples per patient. Finally, the reproducibility analysis was limited to the top LLM (o3-mini); it is possible that code generation for the locally run, smaller models could have led to successful task completions if multiple runs were attempted with different temperature settings and seeds.

Resource availability

Lead contact

Requests for further information and resources should be directed to and will be fulfilled by the lead contact, Marina Sirota (marina.sirota@ucsf.edu).

Materials availability

This study did not generate new, unique reagents.

Data and code availability

• •This paper analyzes existing datasets; accession information for these datasets is listed in Table S2.
• •All code is available at https://github.com/vtarca7/LLMDream (https://doi.org/10.5281/zenodo.18134749) and is publicly available as of the date of publication. This includes the code generated by all eight LLMs considered for each of the three DREAM challenges and two programming languages (48 files). Three replication trials of code generation for o3-mini-high for all endpoints and coding languages are also included. Output files including eventual errors and comments displayed by the R and Python interpreters when executing LLM code are also available in the same repository. Original figures generated by LLMs, already summarized in Figure 3, are included also in this repository.
• •Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.

Acknowledgments

This work was funded by the March of Dimes Prematurity Research Center at UCSF and by ImmPort. Research reported in this publication was also supported by the Eunice Kennedy Shriver National Institute Of Child Health & Human Development of the National Institutes of Health under award number R21HD115800 (A.L.T.). The contributions of the NIH author (R.R.) were made as part of his official duties as NIH federal employee, are in compliance with agency policy requirements, and are considered works of the US government. However, the findings and conclusions presented in this paper are those of the authors and do not necessarily reflect the views of the NIH or the US Department of Health and Human Services.

Author contributions

Conceptualization, M.S., T.T.O., and A.L.T.; methodology, A.L.T., G.B., G.S., M.S., and T.T.O.; investigation, R.S., V.T., and C.A.D.; writing – original draft, R.S., C.A.D., N.K., V.T., A.L.T., T.T.O., and G.B.; writing – review & editing, C.A.D., M.S., T.T.O., A.L.T., R.R., and G.S.; funding acquisition, M.S., T.T.O., and A.L.T.; resources, M.S., T.T.O, R.R., and A.L.T.; supervision, M.S., S.B., A.B., A.L.T., and T.T.O.

Declaration of interests

The authors declare no competing interests.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Software and algorithms
R (v. 4.4.2)	R core team33	https://www.R-project.org/
Python (v. 3.11)	N/A	https://python.org/
pROC (v. 1.18.5)	Robin et al.34	https://cran.r-project.org/web/packages/pROC/index.html
LM Studio (v.0.3.16)	Element Labs Inc.	https://lmstudio.ai/
ChatGPT-4o	OpenAI	https://openai.com
ChatGPT-o3-mini (high reasoning)	OpenAI	https://openai.com
Deepseek-R1	DeepSeek	https://www.deepseek.com
Gemini 2.0 Flash (experimental thinking)	Google	https://gemini.google.com
Github repository for this study	This study	https://doi.org/10.5281/zenodo.18134749
Other
DREAM Preterm Birth Prediction Challenge, Transcriptomics Dataset	Tarca et al.14	https://www.synapse.org/Syn1838082
Microbiome Preterm Birth DREAM Challenge Dataset	Golob et al.15	https://www.synapse.org/Syn26133770
Placental Clock DREAM Challenge Dataset	Bhatti et al.16	https://www.synapse.org/Syn59520082

Method details

LLM prompting for R and python code generation

LLMs were prompted using either a web-based API, or locally via LM Studio (v.0.3.16) on a Windows 11 Pro 64-bit system with 192 GB of RAM and 24 CPU cores (Table S1). All LLM prompts, whether executed locally or via APIs, were run using default temperature and seed settings.

R and python code curation

The LLM-generated code was saved and edited by a) adding extra lines of code to enable saving raw predictions (for successful LLMs only) and b) disabling any lines of code that attempted to install R or python packages. Saving raw predictions was done only for aesthetic purposes (Figure 3), whereas disabling code that attempted to install packages was needed to maintain consistency across the R and Python environments. Packages required by all LLMs were first installed before executing the analysis code. No other code edits were implemented to address possible sources of errors during the code execution, and the datasets did not have missing data that the LLMs would need to handle.

Quantification and statistical analysis

R and python code execution and LLM scoring

The code was executed in Python or R, using the same computing system (Red Hat Enterprise Linux 8.10 Ootpa, with 56 logical processors, 250 GiB of RAM) with all R/Python packages required by any LLM already installed. The test set accuracy and generated plots were then scored to enable LLM ranking. Two of the authors, with expertise in R (at the level of package contributor) and in Python, reviewed the outputs generated by the respective interpreters when executing the LLM-generated code, some of which did not run successfully. The scores were assigned as follows: (a) 1 point for successful data extraction and formatting, which included downloading the data from the repository, extracting the required metadata, identifying the relevant variables, and merging of feature data with the corresponding metadata), (b) 2 points for successful model training and application of the model on the test set and metric calculation, which involved training the model on the training dataset, testing it on the validation dataset, and calculating RMSE for regression tasks or AUROC for classification tasks, (c) 1 point for successful generation of the required plot, i.e., a graph illustrating the predictive performance of each model using RMSE (regression) or AUROC (classification), (d) 1 point for achieving highest prediction accuracy (within ±0.02 for AUROC and ±0.1 week for RMSE of the top model for that task).

Statistical analysis

Confidence intervals for test set RMSE statistics for tasks Q1 and Q2 were obtained using bootstrap (1,000 iterations), while for AUROC (Q3A, B) were obtained using DeLong method with the pROC package in R. Comparisons between top human participant model and top LLM model for Q1 and Q2 were performed using paired t-tests on absolute prediction errors, while for Q3A and Q3B using DeLong tests for paired ROC curves. Two-tailed tests were used in all instances with p < 0.05 used to infer significance.

Additional resources

The LLM-generated code is available at https://github.com/vtarca7/LLMDream.

Published: February 17, 2026