Data-reflector: an agentic exploratory data analysis platform for researchers

Abstract

Objectives

Exploratory data analysis (EDA) is foundational to clinical research yet inaccessible to clinician-scientists without programming training. Existing large language model (LLM) tools improve accessibility but struggle with hallucinations. We developed and evaluated data reflector (DR), a hybrid agentic platform pairing natural language interaction with deterministic statistical computation.

Materials and Methods

Data reflector restricts the LLM to intent interpretation, tool selection, and filter parsing while routing all numerical computation through predefined deterministic functions executed locally. We piloted DR with 6 participants of diverse technical backgrounds across 6 clinical (MIMIC-IV, eICU Collaborative Research Database, National Health and Nutrition Examination Survey) and nonclinical (energy, COVID-19, urban mobility) datasets, assessing usability (system usability scale, SUS), task completion time, analytical output correctness against participant-generated manual reference results, hallucination prevention, and independent reproducibility by a second operator. A head-to-head comparison with ChatGPT 5.3 chatbot was also conducted.

Results

Data reflector achieved a mean SUS of 94.58, with reduced task completion time across all evaluable participants. Statistical outputs and generated distribution curves demonstrated complete agreement with participant-generated manual reference results across all evaluable outputs. Independent reexecution by a different operator using independently constructed prompts also demonstrated complete agreement with the original DR outputs across all evaluated tasks. Hallucination-prevention safeguards correctly declined out-of-scope and impossible queries. ChatGPT produced incomplete, internally inconsistent outputs on identical tasks where DR returned complete, accurate results.

Discussion

Data reflector’s architectural separation of LLM reasoning from numerical computation removes dependence on prompt phrasing, session history, and stochastic LLM sampling while preserving conversational accessibility, addressing reproducibility limitations of general-purpose LLM tools.

Conclusion

Data reflector offers a practical pattern for reproducible, accessible artificial intelligence-assisted EDA across heterogeneous data sources in modern clinical and translational research.

Introduction

Exploratory data analysis (EDA) is a foundational step in virtually every data-driven endeavor, serving as the primary mechanism through which researchers characterize data distributions, identify anomalies, assess data quality, and generate hypotheses prior to formal statistical testing.¹ In the context of clinical and translational research, EDA is particularly critical given the heterogeneity and complexity of health data, which routinely spans structured electronic health record (EHR) variables, laboratory values, physiological time series, and administrative codes across multiple linked tables. Despite its importance, EDA remains a significant bottleneck. It is time-consuming even for technically proficient researchers and is frequently inaccessible to clinician-scientists who lack formal training in programming or statistical computing.

The emergence of large language models (LLMs) has generated substantial interest in natural language interfaces for data analysis. General-purpose platforms such as ChatGPT with Code Interpreter and similar tools allow users to upload datasets and request analyses in plain language.² While these tools represent a meaningful advance in accessibility, they share a common architectural limitation where statistical computations are performed through ad hoc, session-specific code generated at runtime. This introduces risks of inconsistency, hallucination, and nonreproducibility that are particularly problematic in research contexts where analytical integrity is paramount.^3–5 A single ambiguous prompt can yield meaningfully different numerical results across sessions, undermining the foundational requirement for reproducibility in scientific inquiry.⁶

Prior work has demonstrated the usability benefits of natural language interfaces for database querying and clinical data exploration.^7,8 However, no existing platform fully addresses both accessibility and reproducibility simultaneously. Reproducibility, the ability to obtain identical results from identical data using the same analytical pipeline, is a foundational scientific requirement, yet it remains largely unaddressed in current LLM-based analysis tools.

We present data reflector (DR), a hybrid agentic EDA platform that integrates natural language interpretation with deterministic statistical computation. In DR, all statistical metrics are computed by a deterministic, locally executed computational layer, while an LLM agent serves exclusively as a reasoning and intent-interpretation layer, receiving only natural language queries, column names, and structural metadata, not raw patient-level data rows. This architectural separation ensures that numerical results are generated through deterministic computational functions rather than directly by the LLM, while semantic interpretation remains dependent on the LLM mediated agent layer. Data reflector supports multitabular analysis, cross-cohort filtering, diverse input formats, and a broad range of output modalities through a conversational interface accessible to users without any programming background.

The primary contributions of this work are: (1) a clinically oriented natural language analytical workflow supporting multitable cohort construction, cross-table filtering, identifier and composite identifier specification, descriptive statistical analysis, visualization, and export, in which numerical results are produced by predefined deterministic functions; (2) a pilot usability evaluation across participants with diverse technical backgrounds using heterogeneous clinical and nonclinical datasets; and (3) an illustrative case study examining the behavior of DR and a general purpose LLM during a structured EDA task.

Methods

Platform structure

Data reflector is a hybrid agentic platform that merges deterministic statistical computation with LLM-based reasoning. Its fundamental distinction from general-purpose LLM-based analysis assistants lies in its computational model: general-purpose tools generate ad hoc code in response to each user request at runtime, making results sensitive to prompt phrasing and nonreproducible across sessions.^7,8 Data reflector instead employs a goal-oriented analysis pipeline in which all statistical operations are executed by predefined deterministic functions, while the LLM acts exclusively as an intent organizer, interpreting user requests, and selecting the appropriate tools, but performing no computation directly. We have provided the details of the platform in Figure 1 and Supplementary Material S1.

Figure 1.

The overall structure of DR platform architecture. Abbreviation: DR, data reflector.

Pilot usability evaluation

Participants

Six participants were recruited representing a range of technical backgrounds: 2 data scientists, 2 postdoctoral research fellows, 1 critical care physician (MD) without any expertise in coding or statistics, and 1 biomedical engineer. This sample size is consistent with established guidance for formative usability evaluations, in which 5-8 participants are sufficient to identify the majority of significant usability issues.^9,10

Datasets

Datasets were intentionally varied in domain and structure to assess DR performance across heterogeneous data types representative of the breadth of data encountered in academic research settings. Clinical datasets included MIMIC-IV,¹¹ the eICU¹² Collaborative Research Database (eICU-CRD), and the National Health and Nutrition Examination Survey (NHANES).¹³ As contemporary clinical research increasingly incorporates nonclinical data sources including environmental, socioeconomic, geospatial, and population-level variables,^14–16 we also evaluated the performance of DR on nonhealth-care datasets including Open Power Systems solar panel data,¹⁷ the DS4C South Korean COVID-19 geographic dataset,¹⁸ and the NYC Taxi dataset.¹⁹

Tasks and procedure

Each participant was assigned a dataset and specific tasks, progressing from simple descriptive statistics to cross-tabular subcohort filtering and distributional visualization (Table 1). Participants completed each task twice: once using DR and once using their preferred manual analytical method. The manually generated results served as the reference for assessment of analytical output correctness and were compared value by value with the corresponding DR outputs using an exact agreement criterion. Task completion time was recorded for both conditions. Following task completion, participants completed the system usability scale (SUS),²⁰ a validated 10-item instrument for assessing usability.

Table 1.

Tasks assigned to each tester.

Participant	Dataset assigned	Tasks
Data Scientist #1	DS4C South Korean COVID-19 geographic dataset	Identify feature types Analyze the data and generate appropriate stats for the dataset (mean, median, IQR, min, max) and save them in pdf and excel format
Data Scientist #2	NYC Taxi Data	Run analysis on the NYC Taxi table with established statistical metrics (median, min, max, mode, mean, SD) using data reflector
MD	eICU-CRDa Data	Analyze the data and generate appropriate stats (mean, median, min, max, mode) for the all the continuous features in all the files in the data
Biomedical engineer	Open Power Systems solar panel data	Identify Feature types and run analysis on the dataset and return the mean, median, IQR, min, max, mode for the features in the file Generate distribution curve for features
Postdoctoral Researcher #1	NHanes 2021-2023	Run analysis on all tables to generate the distribution curves for male and female patients separately. Generate the appropriate stats for each table (mean, median, IQR, min, max, mode, missing rate) for each group and save the results
Postdoctoral Researcher #2	MIMIC IV	Analyze the data and generate appropriate stats (mean, median, IQR, min, max, mode, missing rate) for 2 subgroups using cross-filtration: (1) male patients above 55 years and (2) female patients below 65 years, and return the results in tables in CSV and word formats. Generate distribution curves for each population

^aeICU-CRD, eICU Collaborative Research Database.

Independent reproducibility validation

Following the primary evaluation, M.S. independently reproduced all 6 participants’ assigned analyses using DR. M.S. constructed independent prompts for each task without reference to participants’ original prompt phrasing, using identical datasets and analytical parameters. Outputs from these independent DR executions were compared with the original participant-generated DR outputs on a metric by metric basis across all tasks. Thus, analytical output correctness was assessed by comparison of participant-generated DR outputs with participant-generated manual reference results, whereas independent reproducibility was assessed by comparison of DR outputs generated by different operators using independently constructed prompts.

This validation was designed to test 2 distinct reproducibility properties: (1) prompt-independence, whether differently worded prompts requesting the same analysis yield identical numerical results and (2) analyst-independence, whether a different operator of the platform obtains identical outputs from the same data.

DR and ChatGPT illustrative comparison

To examine how DR and a general purpose conversational LLM handled the same representative EDA task, we evaluated both platforms using a shared NHANES dataset and the same analytical task. We submitted an identical analytical task using NHANES data to both DR and ChatGPT 5.3. The details of the data are provided in Supplementary Material S1. We provided both systems with similar initial prompts and evaluated the outputs for completeness and accuracy.

Hallucination and semantic failure mode evaluation

To evaluate semantic errors in the LLM mediated workflow, we developed a prespecified 60 scenario test suite spanning 6 hallucination and semantic failure mode categories, executed against a purpose built synthetic multitable dataset. The details appear in Supplementary Material S1 and Table S1.

Results

Usability

The overall mean SUS score across all 6 participants was 94.58 ± 7.49 (Table 2). Five of six participants achieved SUS scores of 95 or above, exceeding the published “excellent” usability threshold of 85.²¹ The lowest score (80.0) was recorded by Data Scientist #1, who rated item Q10 (“I needed to learn a lot of things before I could get going with this system”) at 0, suggesting a modest initial learning curve for users most accustomed to their own programmatic workflows. All other participants rated DR at or near ceiling across all items.

Table 2.

SUS scores from testers.

Tester class	Q1	Q2	Q3	Q4	Q5	Q6	Q7	Q8	Q9	Q10	Sum	SUS score
Data Scientist #1	3	3	4	4	4	3	4	4	3	0	32	80
Data Scientist #2	4	4	4	4	4	4	4	4	4	4	40	100
MD	4	4	4	4	4	4	4	4	4	4	40	100
Biomedical engineer	4	4	4	3	4	4	4	4	4	4	39	97.5
Postdoctoral Researcher #1	4	4	4	3	4	4	4	4	4	3	38	95
Postdoctoral Researcher #2	4	4	4	3	4	4	4	4	4	3	38	95
Mean	3.83	3.83	4	3.5	4	3.83	4	4	3.83	3	37.83	94.58
SD	0.41	0.41	0	0.55	0	0.41	0	0	0.41	1.55	2.99	7.49

Task completion time

In this pragmatic usability evaluation, completion times using DR were shorter than participants’ preferred manual approaches across the evaluable participant specific tasks (Table 3). As participants performed different tasks on different datasets and used different manual comparison workflows, these observations are presented descriptively rather than as a controlled estimate of the efficiency benefit of DR. The critical care physician was unable to complete the assigned tasks using manual methods within any practical timeframe, a finding significant finding in itself that demonstrates the accessibility barrier DR is designed to address. Among remaining participants, time reductions ranged from approximately 50% (postdoctoral researcher 2: ∼30-40 min vs ∼70 min) to over 85% (Data Scientist 1: 2:39 vs 23:12 min).

Table 3.

Tasks completion times.

Participant	DRa time (minutes: seconds)	Manual time (minutes: seconds)
Data Scientist #1	2:39	23:12
Data Scientist #2	0:35	<6:00
MD	48:00	Could not complete
Biomedical engineer	3:27	11:32
Postdoctoral Researcher #1	∼60:00	∼180:00
Postdoctoral Researcher #2	∼(30:00-40:00)	70:00

^aDR, data reflector.

DR analytical output correctness

Across all evaluable participants, statistical outputs generated by DR demonstrated complete agreement with the corresponding participant-generated manual reference results (Table 4). Distribution curves similarly demonstrated complete agreement with the corresponding manual reference outputs across all evaluated visualization tasks. Table 4 reports the number of statistical outputs and distribution curves evaluated for each participant. Individual intermediate operations were not scored as separate accuracy endpoints within the participant evaluation. Potential hallucination and semantic failure modes in these intermediate operations were evaluated separately in a prespecified 60 scenario test suite, including dedicated tests of feature interpretation, filter logic, composite identifiers, cross-table linkage, invalid joins, and valid requests (Table S1).

Table 4.

DR analytical output correctness and independent reproducibility.

Participant	Statistical computation accuracy, % (n)a	Distribution curve accuracy, % (n)b	Accuracy with independent reproducibility,c % (n)d
Data Scientist #1	100 (6)	NA	100 (6)
Data Scientist #2	100 (6)	NA	100 (6)
MDe	NA	NA	100 (5)
Biomedical engineer	100 (6)	100 (41)	100 (47)
Postdoctoral Researcher #1	100 (14)	100 (504)	100 (518)
Postdoctoral Researcher #2	100 (14)	100 (488)	100 (502)

^aFor statistical outputs, n is the sum of the statistical outputs metrics count carried out.

^bFor distribution curve correctness, n represents the number of distribution curves compared between participant-generated DR results and participant-generated manual reference results.

^cApplied to both statistical computations and figures distribution curves generation.

^dFor independent reproducibility, n is the sum of the statistical outputs and distribution curves reported in the preceding columns, each compared between the original participant DR execution and the independent DR execution performed by M.S. using independently constructed prompts.

^eFor the MD participant, only agreement between the 2 DR executions could be assessed. No manual reference result was available; therefore, no correctness estimate is reported.

Abbreviations: DR, data reflector; NA, not applicable.

Hallucination prevention

Data reflector’s scope and cohort existence safeguards were illustrated through 2 targeted refusal tests (Figure 2), while broader potential hallucination and semantic failure modes in LLM-mediated operations were evaluated separately using the 60 scenario test suite reported in Table S1. In the first test, a request was submitted for a t-test on the loaded dataset. As this falls outside DR’s defined analytical scope, DR declined to execute the task and responded: “This falls outside the scope of this EDA platform. I can only perform descriptive exploratory data analysis (distributions, summary statistics, and filtered subgroup comparisons). I cannot perform inferential statistical tests or predictive modelling” (Figure 2A). In the second test, DR was asked to analyze Middle Eastern patients within the NHANES dataset, a demographic category not captured in the data. DR executed the cross-tabular filter, identified a zero-size matching population, and responded: “I couldn’t run the EDA because the filter 'Middle Eastern patients’ matched 0 rows or could not be resolved in the loaded data” (Figure 2B). In both cases, DR declined to generate results, rather than fabricate them.

Figure 2.

Hallucination prevention. (A) Task hallucination prevention. (B) Population metrics hallucination prevention.

Figure 2.

Continued.

Independent reproducibility validation

Independent reproducibility validation was completed by M.S. across all 6 participants’ assigned tasks, using independently constructed prompts on the identical datasets employed in the primary evaluation. Independent reexecution demonstrated complete agreement between M.S.’s DR outputs and the original participant-generated DR outputs across all evaluated tasks and outputs (Table 4). This analysis assessed reproducibility rather than serving as the reference for correctness. For the 5 participants with manual reference results, the participant-generated DR output, participant-generated manual result, and independently generated M.S. DR output were concordant. This held for both descriptive statistics (mean, median, IQR, minimum, maximum, mode, missing rate) and complex cross-tabular subcohort analyses requiring multitable joins and natural language filter interpretation, as well as distribution curve generation (Figure 3).

Figure 3.

Task reproducibility. (A) M.S. prompt 1. (B) M.S. prompt 2. (C) Postdoctoral Researcher #2 prompt. (D) M.S. prompt for postdoctoral researcher.

Figure 3.

Continued.

Figure 3.

Continued.

Figure 3.

Continued.

Hallucination and semantic failure mode evaluation

The 60 scenario test suite evaluated DR across ambiguous and similar feature interpretation, natural language filter logic, composite identifier handling, cross-table linkage, invalid join rejection, and valid analytical requests (Table S1). Data reflector satisfied the prespecified expected behavior in all 60 scenarios (Table S1).

DR vs ChatGPT

We provided similar opening prompts to both ChatGPT and DR (Figure 4A and B), specifying the data and analytical intent. As DR has been developed for exploratory data analysis, we prompted: “Use demo file as the main cohort file, and use SEQN as the main identifier.” For ChatGPT, a general purpose tool, we provided more detailed framing: “I would like to perform exploratory data analysis on these data files. Use demo file as the main cohort file (Master List of Participants), and use SEQN as the main identifier. In the next steps, I will ask you specific analysis questions.” We then submitted the same analytical prompt to both platforms: “Run analysis on CBC table for white male patients above 64 years old.” Data reflector correctly interpreted the task and returned accurate results (Figure 4C). ChatGPT’s initial response provided a CSV file that included mean, SD, median, 25 and 75 percentiles, minimum and maximum values, but omitted the mode (Figure 4D and E). When we asked it to display the results in the frontend GUI, it generated a table that omitted the mode and rendered for the standard deviation, 25th percentile and minimum values as “…”. In response to our follow-up prompt, “Show the full results, some metrics show up as …”, ChatGPT regenerated the table with values populated for previously truncated SD, minimum, and 25th percentile fields; however, the resulting values were incorrect and did not match those in the originally generated CSV file (Figure 4F and G).

Figure 4.

Performance of DR vs ChatGPT. (A) ChatGPT Data upload and instructions. (B) DR instructions. (C) DR test. (D) ChatGPT first run. (E) ChatGPT first run CSV file results. (F) Incomplete frontend results from ChatGPT. (G) ChatGPT frontend first run results. (H) ChatGPT second run. (I) ChatGPT second run clarification. (J) ChatGPT third run initial results. Abbreviation: DR, data reflector.

Figure 4.

Continued.

Figure 4.

Continued.

Figure 4.

Continued.

Figure 4.

Continued.

Figure 4.

Continued.

Figure 4.

Continued.

Figure 4.

Continued.

Figure 4.

Continued.

Figure 4.

Continued.

To probe whether these errors could be corrected through alternative prompting, we created a second ChatGPT conversation branch from the same opening prompt (Figure 4A), using a different analytical prompt: “Run analysis on CBC table for white male patients above 64 years old and return the main statistics by identifier in a table.” But then ChatGPT misinterpreted “by identifier” as a request to compute metrics per individual SEQN, producing a result inconsistent with the intended population-level summary (Figure 4H). We then provided a within branch clarification specifying the exact metrics required, “No, I mean run analysis on CBC table for white male patients above 64 years old, and return the mean, sd, median, q1, q3, minimum, maximum, mode, and missing rate by patient id in a table.”; however, ChatGPT again failed to produce the correct output (Figure 4I).

When we used a more specific analytical prompt (Figure 4A), “Run analysis on CBC table for white male patients above 64 years old, and return the mean, SD, median, Q1, Q3, minimum, maximum, mode, and missing rate by patient ID in a table,” after the same opening instructions, the ChatGPT returned a table that it described as containing the results (Figure 4J). The table displayed “…” in place of existing values. Only after we explicitly requested the full table did ChatGPT returned fully populated and correct values with this prompting scheme.

Discussion

In this study, we developed and evaluated DR, a hybrid agentic platform that decouples natural language understanding from statistical computation in exploratory data analysis. Across 6 participants spanning a deliberately heterogeneous range of technical backgrounds and across 6 datasets drawn from clinical and nonclinical domains, DR achieved a mean System Usability Scale score of 94.58, with 5 of 6 participants exceeding the established “excellent” usability threshold of 85.²¹ Data reflector also reduced task completion time relative to manual methods in every evaluable case, and enabled task completion for a critical care physician participant who could not complete the analysis using available manual approaches. Statistical outputs and distribution curves demonstrated complete agreement with participant-generated manual reference results across all evaluable outputs. Separately, independent reexecution by a second operator using independently constructed prompts demonstrated complete agreement with the original DR outputs, supporting reproducibility across operators and prompt formulations. Complementary testing using the 60 scenario suite evaluated intermediate behaviors including feature interpretation, filter logic, composite identifier handling, and cross-table linkage against prespecified expected behavior (Table S1). Targeted hallucination-prevention tests confirmed that requests outside the platform’s defined analytical scope and queries against nonexistent subcohorts were correctly declined rather than fabricated. In this illustrative comparison using a shared NHANES cohort and the same analytical task, DR returned complete and accurate analytical outputs on the initial response. The evaluated ChatGPT interactions initially produced incomplete outputs, and selected follow-up interactions produced displayed values that were inconsistent with the originally generated CSV results. These findings illustrate differences observed between the 2 platforms for this representative analytical task. During the reported evaluation, DR used OpenAI API version 5.2 (gpt-5.2-2025-12-11).

These findings extend a growing literature documenting both the promise and the limitations of large language models as data analysis tools. Earlier work has shown that ChatGPT and related tools can render data science workflows accessible to non-programmers and produce results that, with carefully constructed prompts, approximate those of conventional statistical software.² However, this performance is consistently dependent on prompt sophistication. In a health care-focused validation study using a National Inpatient Sample subset, ChatGPT achieved 32.5% accuracy on basic prompts, 81.3% on intermediate prompts, and 92.5% on advanced prompts when computing inferential statistics, demonstrating that analytical correctness scales with the technical sophistication of the user constructing the prompt.²² Our direct comparison against ChatGPT illustrates this dependence concretely, where a typical user-style prompt produced a CSV with omitted statistics, a frontend table with truncated values, internal inconsistencies between the CSV and the displayed table, and persistent misinterpretation of analytical intent across follow-up clarifications.

Several recently described platforms have applied conversational interfaces to clinical and biomedical data analysis, but few have been systematically compared on the architectural choices that determine reproducibility and accessibility. We compared DR against 24 widely available platforms and frameworks across general-purpose conversational LLMs (eg, ChatGPT,²³ Claude,²⁴ Gemini²⁵), specialized data-analysis assistants (eg, Julius AI,²⁶ PandasAI,²⁷ EDA-GPT,²⁸ Data Interpreter²⁹), enterprise analytics platforms (eg, Tableau Next,³⁰ BigQuery³¹), and biomedical AI agents (eg, AI-HOPE,³² EHRFlow,⁷ BioChatter,³³ BRAD³⁴), examining data handling, exploratory analysis, AI/agent functionality, and deployment and security based on their publicly available documentation (Tables S2A, S2B, and S2C). Data reflector was one of a small number of platforms in this comparison that combined a conversational natural language interface with deterministic local computation in which all numerical results are produced by predefined functions rather than LLM-generated runtime code, and the only such platform among the biomedical agents we examined. This architectural separation, restricting the LLM layer to intent interpretation, tool selection, and natural language filter parsing while routing all statistical operations through deterministic backends, means that DR’s deterministic numerical computation produced identical statistical outputs across 3 semantically equivalent paraphrased requests (Table S3). This evaluation assessed output invariance for a fixed analytical request, while broader semantic failure modes were evaluated separately using the 60 scenario test suite in Table S1. Unlike many widely used chatbot platforms, DR also does not impose vendor-defined file size, computation, or session caps when deployed locally (Table S2C). Several systems occupy an adjacent design space to DR and provide important architectural context. Data Copilot³⁵ similarly constrains natural language driven analysis through designed analytical interfaces, although its workflow incorporates generated code as an intermediary. InsightPilot³⁶ maps natural language analytical requests to a bounded set of designed analysis actions for automated data exploration. Data formulator³⁷ separates high-level analytical intent from lower-level data transformation and visualization operations, providing another relevant example of constrained natural language-driven analysis. Recent work examining LLM agents across biomedical domains further highlights grounding, privacy, safety, and evaluation as central considerations for biomedical agent systems. These approaches share aspects of DR’s separation between high-level natural language intent and constrained downstream operations while differing in analytical scope, execution mechanisms, and intended applications.³⁸

The accessibility implications of this architecture are particularly relevant for clinician-scientists. Electronic health record-based research increasingly requires iterative exploratory analysis to characterize data quality, refine variable definitions, and generate hypotheses, and this work is most effectively performed by researchers with domain expertise in the underlying data. Yet exploratory analysis remains inaccessible to most clinicians without programming training, who typically depend on collaborators or data analysts to translate analytical questions into code. Our finding that the critical care physician participant could not complete the assigned tasks using available manual methods illustrates this barrier directly, while the high SUS scores and substantial time reductions observed across users with diverse technical backgrounds suggest that DR’s natural language interface meaningfully extends analytical access without sacrificing computational integrity. Notably, our evaluation deliberately spanned both clinical (MIMIC-IV, eICU, NHANES) and nonclinical (Open Power Systems solar panel data, DS4C South Korean COVID-19 geographic data, and the NYC Taxi dataset) datasets, reflecting the increasing integration of traditionally nonhealth care data into modern biomedical research.^14–16 Consistent performance of DR across these heterogeneous data sources supports its applicability to the multisource analytical workflows that are becoming standard in clinical and translational science. Data reflector incorporates safeguards for cross-tabular operations. Users specify the identifier or combination of identifiers used for cohort linkage, including composite identifiers when appropriate. For cross-table filtering, the specified identifier is used to derive the applicable set of IDs, and statistical calculations for a requested feature are performed within the table containing that feature using the filtered identifier set rather than on an expanded joined representation, thereby reducing the risk of duplicated observations resulting from many to many relationships. When identically named candidate features occur across multiple tables, DR identifies the ambiguity and requests clarification regarding the intended feature source before proceeding. Natural language filters are mapped to the relevant feature and corresponding values represented in the dataset rather than relying exclusively on exact string matching, allowing semantically equivalent representations to be incorporated into cohort definitions. Data reflector’s architectural separation prevents direct LLM generation of numerical results by routing statistical computation through predefined deterministic functions. This property does not eliminate the possibility of semantic errors in LLM-mediated operations such as feature interpretation, filter parsing, identifier selection, or tool selection. We therefore evaluated these potential failure modes separately using a prespecified 60 scenario suite encompassing ambiguous features, negation, composite identifiers, similar feature names, invalid joins, and valid requests that should not be refused (Table S1). Each scenario was evaluated once and within a continuing analytical session; consequently, this evaluation establishes whether DR produced the prespecified expected behavior in these scenarios but does not assess stability across repeated executions, and some scenarios incorporated context from the currently loaded table or preceding request.

It is important to acknowledge the limitations of this study. First, this was a formative usability pilot involving 6 participants from a single academic medical center. While this sample size is consistent with established guidance for identifying the majority of usability issues,¹⁰ it is not designed to support inferences about clinical workflow adoption. Larger, multi-institutional evaluations across diverse practice settings and user populations will be required to assess generalizability, workflow integration, and adoption. Second, DR’s analytical scope is intentionally bounded to descriptive exploratory analysis, including summary statistics, distributional visualization, and cross-tabular sub cohort filtering. It does not currently include inferential statistical tests, such as t-tests, chi-squared tests, or predictive modeling. Extending these capabilities while preserving deterministic guarantees is an important direction for future development. Third, although DR’s computational layer operates entirely locally and does not transmit raw patient-level data to external services, agentic modules do transmit column names, small numbers of sample values, natural language task descriptions, and filter expressions to the OpenAI API. These elements may raise data governance concerns in highly restrictive EHR environments. This design choice trades a limited disclosure surface for the conversational reasoning required for natural language interaction and may not be appropriate for all data-governance contexts. Institutions should review this workflow under local data governance policies before deployment with sensitive clinical datasets. Fourth, the comparison with ChatGPT was conducted as a head-to-head evaluation on a single representative task rather than a quantitative benchmark. The observations are illustrative and not intended as an exhaustive characterization of general-purpose LLM behavior, which continues to evolve rapidly. Finally, as each participant was assigned a different dataset and task set and used their preferred manual analytical approach, participant background, task complexity, dataset, and comparison method were not independently controlled. Accordingly, observed differences in completion times are descriptive. Future controlled evaluations in which multiple participants perform identical representative tasks using standardized comparison workflows would provide a more direct assessment of comparative efficiency.”

In conclusion, DR pairs a natural language interface with deterministic statistical computation to deliver reproducible, accurate, and accessible exploratory data analysis. By restricting the LLM to intent interpretation and routing all computation through predefined deterministic functions, DR addresses both the reproducibility limitations of LLM-authored analytical code and the programming barrier that excludes most clinicians from exploratory analysis, offering a practical solution for AI-assisted exploratory data analysis across the increasingly heterogeneous data sources of modern clinical and translational research.

Author contributions

Moein Sabounchi (Conceptualization, Data curation, Formal analysis, Validation, Visualization, Writing—original draft, Writing—review & editing), Nimay Hazare (Writing—review & editing), Christopher Capone (Validation, Writing—review & editing), Mateen Jangda (Validation, Writing—review & editing), Wonsuk Oh (Validation, Writing—review & editing), Pushkala Jayaraman (Validation, Writing—review & editing), Shashank Gupta (Validation, Writing—review & editing), Ashwin Sawant (Writing—review & editing), Zachary Kuschner (Validation, Writing—review & editing), Prathamesh Parchure (Writing—review & editing), Prem Timsina (Writing—review & editing), Ben Kaplan (Writing—review & editing), Aditi Vakil (Writing—review & editing), Roopa Kohli-Seth (Writing—review & editing), Patricia Kovatch (Writing—review & editing), Alexis Zebrowski (Writing—review & editing), Benjamin S. Glicksberg (Writing—review & editing), Pulkit Agrawal (Writing—review & editing), Bruce Darrow (Writing—review & editing), Lisa Stump (Writing—review & editing), Alexander W. Charney (Writing—review & editing), Girish Nadkarni (Funding acquisition, Project administration, Writing—review & editing), and Ankit Sakhuja (Conceptualization, Data curation, Formal analysis, Funding acquisition, Project administration, Supervision, Writing—original draft, Writing—review & editing)

Supplementary material

Supplementary material is available at JAMIA Open online.

Funding

This study was supported by National Institutes of Health (NIH) grants K08DK131286 (A.S.) and R01DK133539 (G.N.N.). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

Conflicts of interest

G.N.N. is a founder of Renalytix, Pensieve, Verici and provides consultancy services to AstraZeneca, Reata, Renalytix, Siemens Healthineer, and Variant Bio, serves a scientific advisory board member for Renalytix and Pensieve. He also has equity in Renalytix, Pensieve, and Verici. A.S. is a consultant for Roche Diagnostics Corporation. All remaining authors have declared no conflicts of interest.

Data availability

The datasets utilized in the validation of Data Reflector are publicly available and can be accessed after completion of any necessary training and execution of applicable data use agreements. MIMIC-IV version 2.2 is available through PhysioNet at https://physionet.org/content/mimiciv/2.2/. The eICU Collaborative Research Database is also available through PhysioNet at https://physionet.org/content/eicu-crd/. The DS4C South Korean COVID-19 geographic dataset is available through Kaggle at https://www.kaggle.com/datasets/kimjihoo/coronavirusdataset. The NYC Taxi Data are available through Kaggle at https://www.kaggle.com/code/headsortails/nyc-taxi-eda-update-the-fast-the-curious and through the NYC Taxi and Limousine Commission trip-record data portal at https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page. The Open Power System Data time series solar panel data are publicly available at https://data.open-power-system-data.org/time_series/2020-10-06/ and through DOI: https://doi.org/10.25832/time_series/2020-10-06. The National Health and Nutrition Examination Survey August 2021-August 2023 datasets are publicly available through the CDC/National Center for Health Statistics at https://wwwn.cdc.gov/Nchs/Nhanes/ContinuousNhanes/default.aspx? Cycle=2021-2023. The synthetic multitable dataset created specifically for the 60-scenario hallucination and semantic failure mode evaluation reported in Table S1 is publicly available through Zenodo at https://doi.org/10.5281/zenodo.22129835