Can AI write your code? A case study of chatgpt’s statistical coding capabilities for quantitative research

Abstract

Background

Recent advancements in Artificial Intelligence (AI), particularly in large language models (LLMs) like OpenAI’s ChatGPT, have extended its applications well beyond simple dialogue generation. ChatGPT has shown potential in supporting data-driven decision-making. ChatGPT has gained traction in academia for its ability to generate code for data analysis, providing robust support for programming languages. This study aims to evaluate ChatGPT’s ability to generate code for causal inference and data analysis.

Methods

This study evaluates ChatGPT4.0 Pro’s performance in coding Difference-in-Differences (Diff-in-Diff ), Inverse Probability Treatment Weighting (IPTW), and Regression Discontinuity (RD) using problem sets and reference code from “Causal Inference: The Mixtape”. The evaluation was conducted in Python, Stata, and R. Researchers provided structured prompts and feedback, and a fourth researcher replicated all tasks to assess consistency. Primary outcomes included accuracy, efficiency, error output, editing needs, and inter-user consistency.

Results

ChatGPT generated accurate code and results in R and Python for most tasks. However, it struggled with IPTW and performed less reliably in Stata. Errors were often related to data management or figure generation. Although ChatGPT could replicate correct results, the structure and syntax of its code varied across users and sessions.

Conclusions

ChatGPT shows strong potential as a supportive tool for econometric coding tasks in health economics, especially in Python and R. However, its output still requires human interpretation and validation. As generative AI continues to evolve, these tools hold promise for streamlining research tasks but remain supplementary to skilled human researchers in quantitative research.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13561-025-00714-8.

Background

In recent years, Artificial Intelligence (AI) including large language models (LLMs), and generative AI has seen extensive advancement. OpenAI’s ChatGPT is one example of a partially autoregressive model that uses natural language processing to create dialogue. Its reach has extended well beyond basic dialogue creation. In healthcare, it can support patients’ medical consultations [1–7], clinical decision-making [2, 8–12], disease diagnosis, detection and classification [13–19], and other forms of clinical support [20–22]. Additionally, ChatGPT has become increasingly valuable in academic contexts due to its robust programming support and ability to produce code for data analysis [23–25].

The ability of ChatGPT to generate code for data analysis has been investigated in both qualitative and quantitative research across various fields [26–29]. For instance, Korinek (2023) analyzed how ChatGPT can assist economists in research and found that it can significantly enhance productivity by automating various small-scale tasks, including complete data analysis and coding tasks, especially in Python and R [28]. Similarly, Huang et al. (2024) demonstrated that ChatGPT-4 could generate SAS, SPSS, and R code to perform descriptive statistics, intergroup analysis, and correlation analysis [29]. However, both studies emphasize that ChatGPT remains limited in handling more advanced and complex methods which still require human oversight.

Econometric methods such as Difference-in-Differences (Diff-in-Diff), Inverse Probability Treatment Weighting (IPTW), and Regression Discontinuity (RD) are widely employed in empirical research for causal inference [30, 31]. However, the extent to which ChatGPT can generate accurate and replicable code for these methods remains underexplored. Existing studies often evaluate ChatGPT’s coding performance based on subjective researcher judgment, rather than against standardized reference code. While studies such as those cited above have started this discussion, they have not assessed common methods including IPTW nor have assessed all programs used in this research including Stata. Therefore, this study aims to assess ChatGPT’s ability to produce code for Diff-in-Diff, IPTW, and RD by applying it to problem sets from established econometrics textbooks and evaluating the outputs against their corresponding reference implementations.

Methods

Data and problem sets

To evaluate ChatGPT- 4.0’s (pro, paid for version) accuracy and efficiency in coding econometric analyses, we utilized publicly available data and problem sets from “Causal Inference: The Mixtape” by Cunningham [32]. The Mixtape introduces causal effects and demonstrates how to use econometric techniques with detailed coding examples in R 3.6.0, Stata 18, and Python 3.13. We chose to use The Mixtape due to its focus on conceptual and applied exercises that align with health economics research. The examples focus on essential components of quantitative and econometrics research including data cleaning, regression methods, and general assumptions underlying statistical analysis. We selected problem sets focused on Diff-in-Diff, IPTW, and RD as the basis for testing. The coding examples and corresponding outputs from The Mixtape served as the benchmark for assessing ChatGPT’s performance in terms of both accuracy and efficiency.

To assess the extent to which ChatGPT’s statistical coding capabilities can support quantitative research, we followed a three-step process: (1) Prompting ChatGPT with problem sets and asking it to generate code to perform the relevant econometric analyses (2). Providing more comprehensive prompts to replicate full coding tasks from The Mixtape, including econometric analyses, data management, and the generation of figures (3). Executing the generated code in R 3.6.0, Stata 18, and Python 3.13, and recording the outputs for evaluation.

Problem set: Difference-in-Differences

In 1970, five U.S. states legalized abortion before the nationwide legalization following the Roe v. Wade decision in 1973. I want to examine whether early abortion legalization affected adolescent health outcomes. Please write (R/STATA/Python) code to perform a Diff-in-Diff analysis to estimate the impact of abortion legalization on gonorrhea incidence among adolescent females aged 15–19. Instead of using a simple post-treatment indicator, use year-by-treatment interactions to capture the dynamic treatment effects over time. The variable repeal indicates whether a state legalized abortion in 1970 (repeal = 1) or in 1973 (repeal = 0).

Problem set: IPTW

In the mid-1970s, the National Supported Work (NSW) Demonstration was a randomized job training program designed to help disadvantaged individuals—including former offenders, high school dropouts, and recipients of public assistance—transition into the formal labor market. In this problem, I am evaluating the impact of the NSW program on participants’ post-treatment earnings. The program randomly assigned eligible individuals to either receive job training (treat = 1) or not (treat = 0). Please write code in [R/Stata/Python] to perform an IPTW analysis using estimated propensity scores, to adjust for differences between the treatment and control groups and estimate the Average Treatment Effect on the Treated (ATT).

Problem set: RD

In U.S. congressional elections, whether a Democrat or Republican wins can often come down to a narrow vote margin. When the Democratic vote is close to 50%, the outcome is considered quasi-random. Please write code in [R/Stata/Python] to estimate the effect of Democratic electoral victory on the legislator’s future liberal voting behavior (score) using a sharp RD design. Use the running variable lagdemvoteshare (Democratic vote share in the prior election) and define treatment as democrat = 1 if the Democrat won. Restrict the sample to close elections where lagdemvoteshare is between 0.48 and 0.52. Cluster standard errors by id.

Generating code

To evaluate ChatGPT’s coding capabilities in econometric analysis, we recruited four researchers with advanced expertise in econometric methods—two of whom hold PhDs and two who are current PhD candidates. Three researchers were each assigned to work with one programming language (R, Stata, or Python), following structured prompts to interact with ChatGPT. A fourth researcher independently replicated the entire process across all three languages to validate results and assess the consistency of ChatGPT’s responses.

To standardize the evaluation, we collaboratively developed and refined a set of prompts designed to guide ChatGPT through the coding tasks. These prompts were constructed to decompose each problem set into logical steps and ensure uniform background information was provided across all tasks. Three researchers independently drafted initial versions of the prompts, which were then iteratively refined through group discussions and pilot testing. During this refinement process, we compared ChatGPT’s initial responses to identify ambiguities, missing context, or inconsistent interpretations, and revised the wording, sequence, and specificity of instructions accordingly. The final versions balanced clarity and generalizability, providing ChatGPT with sufficient structure while avoiding overfitting to any one problem type. Example prompts are available in the supplementary material for Python, but all the prompts followed the same structure and asked for the same information. Broadly, prompts were designed to guide GPT through the structure of three common causal inference frameworks. Each prompt sequence began by specifying the policy or program context followed by instructions to prepare and clean the dataset (import, exclusions, factor coding, generation of covariates), estimate treatment effects with appropriate control variables or weights, and produce standard graphical outputs (event-study plots, propensity score distributions, RD plots with fitted lines). Additional steps included creating interaction terms, applying clustered standard errors, trimming weights, and validating continuity assumptions were prompted as needed. This standardized prompt structure ensured consistency across analytic designs while allowing for method-specific implementation details.

No datasets were uploaded to ChatGPT during this study. After ChatGPT generated the code, the researchers executed the code locally in the corresponding programming environment. They then provided immediate feedback based on the output. If an error occurred, the error message was reported back to ChatGPT for troubleshooting.

Primary outcomes

This study evaluated five primary outcomes: accuracy, efficiency, error, editing, and consistency. Accuracy was assessed by directly comparing the results generated by the ChatGPT-written code with the corresponding benchmark outputs published in the Mixtape materials for each econometric method. Accuracy was defined in a binary manner: if the ChatGPT-generated results matched the benchmark, accuracy was recorded as “Yes”; otherwise, it was recorded as “No.”. Efficiency was assessed by comparing the number of code commands used by ChatGPT with those in the standard code, and the average difference in command count was calculated separately for each programming language as well as across all tasks. Human oversight was required when ChatGPT-generated code produced syntax errors or incorrect results. Intervention typically occurred after around ten unsuccessful attempts, underscoring the continued need for researcher involvement to ensure accuracy and validity. To assess error output, we documented any instance where the ChatGPT-generated code produced an execution error. The proportion of prompts that resulted in such errors was then calculated. Editing was evaluated in cases where the code did not yield an error but required clarification, additional context, or adjustment to achieve the correct output. We computed the average percentage of prompts that necessitated manual intervention or revision.

Finally, consistency was assessed through a replication procedure. A fourth researcher independently repeated all coding tasks using the same prompts across R, Stata, and Python, employing a new ChatGPT account with no prior conversation history. This was done to evaluate the consistency of ChatGPT’s outputs when used by different individuals under identical prompt conditions. If the regenerated code was similar in logic and structure as determined by the fourth researcher, it was interpreted as evidence of internal consistency in ChatGPT’s code generation process.

Results

Overall, ChatGPT was able to generate accurate results in Python, Stata, and R that were consistent with the output produced by the standard reference code. However, instances of error outputs and required prompt editing emerged during the process, indicating that researcher supervision and iterative feedback were necessary to achieve correct results. The consistency of ChatGPT’s output was further validated by a fourth researcher who independently repeated all coding tasks across languages using identical prompts (see Table 1). Nevertheless, even when the final results were consistent, the structure and content of the code generated by ChatGPT often varied across sessions and users (Additional File 2).

Table 1.

Summary of ChatGPT Coding Capabilities

Program	Analysis	Total Prompt		Accuracy 4	Error output 5		Prompt Editing 6
Python	IPTW1		9	Yes	33%		11%
	Diff-in-Diff2		14	Yes	50%		14%
	RD3		11	Yes	0%		0%
Stata	IPTW1		16	Yes	31.25%		6.25%
	Diff-in-Diff2		21	No	76%		20%
	RD3		12	Yes	8.3%		0%
R	IPTW1		18	Yes	39%		17%
	Diff-in-Diff2		18	Yes	67%		22%
	RD3		12	Yes	0%		0%

1: IPTW Inverse-probability-treatment-weighting

2: Diff-in-Diff Difference-in-differences

3: RD Regression discontinuity

4: Accuracy: Indicates whether the code generated by ChatGPT produced results identical to the standard code

5: Error Output: The proportion of prompts that resulted in code execution errors (number of error outputs/total prompts)

6: Prompt Editing: The proportion of prompts that required manual revision or clarification to achieve correct output (number of edited prompts/total prompts); “Total Prompt” refers to the total number of prompt–response exchanges (including initial and corrective prompts) required to obtain a final, functioning code for each analytic method

For the econometric analysis tasks, ChatGPT successfully generated correct code for IPTW, Difference-in-Differences, and Regression Discontinuity using both Python and R, producing results consistent with the Mixtape benchmarks after prompt editing. In contrast, ChatGPT produced erroneous code for IPTW across Python, Stata, and R. Diff-in-Diff analysis presented moderate difficulty, with observed error rates of 50% in Python, 76% in Stata, and 67% in R, alongside respective prompt editing rates of 14%, 20%, and 22% (see Table 1). Many of these issues stemmed from challenges in data management and figure generation (Table 2). ChatGPT exhibited higher error and editing rates in tasks involving data management and especially figure generation. Among the three platforms, Stata posed the greatest difficulty for ChatGPT. Importantly, even when errors occurred, ChatGPT was able to produce accurate results after receiving appropriate guidance from researchers.

Table 2.

Evaluation of ChatGPT Coding

		Number of Total Prompt			Error output			Prompt Editing
Program	Analysis	Econometric analyses	Data Manage	Generate Figure	Econometric analyses	Data manage	Generate Figure	Econometric analyses	Data Manage	Generate Figure
Python	IPTW	4 (44%)	2 (22%)	3 (33%)	3 (75%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)	1 (33%)
	Diff-in-Diff	1 (7%)	5 (36%)	8 (57%)	0 (0%)	1 (25%)	6 (75%)	0 (0%)	2 (40%)	1 (13%)
	RD	3 (27%)	5 (45%)	3 (27%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)
Stata	IPTW	6 (38%)	9 (56%)	1 (6%)	2 (33%)	3 (33%)	0 (0%)	1 (15%)	0 (0%)	0 (0%)
	Diff-in-Diff	4 (19%)	5 (24%)	12 (57%)	2 (50%)	2 (40%)	12 (100%)	2 (50%)	2 (40%)	1 (5%)
	RD	3 (25%)	5 (42%)	4 (33%)	0 (0%)	0 (0%)	1 (25%)	0 (100%)	0 (100%)	0 (100%)
R	IPTW	4 (22%)	14 (78%)	0 (0%)	1 (25%)	6 (43%)	0 (0%)	0 (0%)	3 (21%)	0 (0%)
	Diff-in-Diff	4 (22%)	7 (39%)	7 (39%)	0 (0%)	6 (86%)	6 (86%)	0 (0%)	3 (43%)	1 (14%)
	RD	4 (37%)	6 (50%)	2 (33%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)

Figure 1 depicts notable variations in the quantity of code generated by ChatGPT in IPTW, Diff-in-Diff, and RD. Specifically, ChatGPT generated 14 additional commands of Python code and 27 more commands in R compared to a human for the IPTW problem set. Similarly, in the Diff-in-Diff analysis, ChatGPT produced 8 more commands in Python and 12 more in R. In RD, the output increased by 1 command for Stata and 11 commands in R. At times, ChatGPT was slightly more efficient in generating code. In Stata GPT used with 12 fewer commands in Diff-in-Diff and 4 fewer in IPTW. For RD, ChatGPT suggested 1 fewer line of code for RD in Python. Comments were not included in the count of lines of commands.

Fig. 1.

Comparing the Number of Commands Between Mixtape and ChatGPT NOTE: Abbreviations: Difference-in-differences (Diff-in-Diff); Inverse probability treatment weighting (IPTW), Regression discontinuity (RD)NOTE: Abbreviations: Difference-in-Differences (Diff-in-Diff); Inverse Probability Treatment Weighting (IPTW); Regression Discontinuity (RD)

Discussion

The findings of this study underscore both the potential and limitations of using ChatGPT for applied econometric coding and research support across multiple statistical programming environments. Consistent with prior studies, ChatGPT was generally capable of generating code that produced accurate results; however, the process was not entirely autonomous [33]. Errors and the need for prompt refinement were common, highlighting the importance of human oversight in ensuring reliable outcomes. To the best of our knowledge, this is the first study to systematically evaluate ChatGPT’s econometric coding capabilities across Stata, R, and Python, directly comparing its output to standard textbook code.

Rather than expecting identical code outputs, our assessment of reproducibility aimed to evaluate the consistency and reliability of ChatGPT’s logic and structure when given the same prompts. While the generated code differed in syntax across users and sessions, the resulting analyses and interpretations remained largely consistent, underscoring and reinforcing ChatGPT’s potential as a coding assistant rather than a deterministic programming tool. In practice, human programmers also produce functionally distinct code, and full uniformity is neither expected nor necessary. However, awareness of this variability remains important for reproducibility, documentation, and version control in collaborative projects.

ChatGPT proved most useful for automating straightforward and modular tasks, such as producing summary statistics, formatting variables, and applying econometric methods. However, ChatGPT struggled with more complex tasks involving multiple steps, such as calculating IPTW or generating figures based on Diff-in-Diff regression estimates. These issues appear to stem, in part, from ChatGPT’s inconsistent handling of internal memory within a single session. For instance, in the Diff-in-Diff problem set using Python, ChatGPT introduced errors by referencing variable names inconsistently and failing to link regression outputs to figure generation. Nonetheless, when users provided explicit regression results, ChatGPT was generally able to complete the figure correctly, indicating that guided intervention can mitigate some limitations.

Across platforms, ChatGPT performed most reliably in Python, followed by R, and showed the least accuracy in Stata. In both R and Python, ChatGPT was able to generate code that produced accurate results on the first attempt for most problem sets. In contrast, Stata required more detailed prompts and often struggled with producing publication-ready tables and figures, necessitating greater user involvement. This made sense given that Stata has fewer online resources for coding support, whereas Python and R are open-source languages with extensive user communities and publicly available documentation. These findings are consistent with informal reports and prior studies, which also found that ChatGPT’s performance varies across programming languages. In Prander et al. (2025), researchers found that there were limitations and ChatGPT could not easily code in R but faced more difficulties in Stata [34]. This discrepancy may be due to the different languages being tested, and ChatGPT may be more appropriate as a coding tool for certain statistical languages over others [35]. In addition, researchers’ coding skills and experience also play a role when comparing the efficiency of ChatGPT’s code to human-written code. Many of our analyses required researchers to ask ChatGPT to correct code or errors manually. In other fields such as computer science, researchers also describe needing to fix errors in code due to specific types of mistakes that permeated throughout different analyses [36]. These results should be seen as exploratory given the small number of standardized problem sets. Future research involving broader and more complex problem sets will be essential to validate and extend these observations.

While generative AI offers substantial promise for streamlining econometric workflows, it is not yet capable of independently conducting quantitative research. Effective use still depends on researchers’ coding proficiency and methodological expertise to interpret outputs, identify errors, and ensure analytical integrity. Therefore, ChatGPT should be viewed as a complement to, and not a replacement for, expert human judgment.

At this stage, generative AI is great for learning code and coding support but is not equipped to independently perform econometric and quantitative research. Therefore, we argue that current versions still require human expertise for interpretation and validation. While these tools can streamline workflows, they lack the domain-specific knowledge and critical thinking needed for rigorous analysis [37]. Moreover, the prompts used in this study may not generalize to other contexts, as ChatGPT’s outputs varied even with identical inputs due to its probabilistic nature and contextual sensitivity. Researchers replicating similar methods should manually verify and refine outputs to ensure accuracy. Finally, while ChatGPT can automate routine analytical tasks, relying on AI-generated code without expert validation may produce biased or misleading results. Human oversight remains essential for both technical accuracy and ethical integrity in health-related research.

A major limitation of this study is the overall generalizability of our findings because the data, problem sets, and code used as standards are publicly available for specific tasks. ChatGPT might have been previously trained on these materials, potentially overestimating its performance. Despite ChatGPT possibly being trained on these materials, our research still observed multiple errors, specifically in Stata. Additionally, the updates and training on Generative AI are ongoing. We accessed ChatGPT 4.0 pro version for all prompts within the same month to minimize this bias. Nevertheless, as new versions are released, future researchers might obtain different results.

A future concern is that OpenAI recently announced that further training will rely upon open source chat forums such as Reddit or Stack Exchange [38]. Since many of these forums contain outdated or inaccurate code, it is possible for ChatGPT’s coding prowess to fluctuate rather than consistently improve over time. Additionally, the researchers in this study are very familiar with the statistical methods and coding languages used, which may impact the generalizability to researchers of differing levels of technical expertise. Finally, we did not formally assess prompt quality, which may have affected performance. While prompt-engineering techniques could improve results, our approach reflects the typical experience of researchers with limited LLM expertise, underscoring the importance of clear prompt design for reliable outcomes.

Given the limited scope of this study, several future research questions should be assessed. For example, although this study focused on ChatGPT, it is important to acknowledge that the broader landscape of large language models for code generation is evolving quickly. Alternative tools such as Windsurf, Cursor, and Codex have demonstrated strong potential for professional-grade statistical programming. Future research should examine not only ChatGPT but also these emerging platforms, as comparative evaluations across LLMs will help determine which models are most effective and reliable for different econometric tasks. Additionally, it would be useful to conduct a deeper analysis of prompts, errors and solutions. Furthermore, although our study did not identify major readability issues in the ChatGPT-generated code, we recognize that code brevity and reduced command usage may compromise interpretability, particularly for less experienced researchers. As such, the readability and clarity of AI-generated code should be carefully evaluated in future studies to ensure accessibility and reproducibility.

Conclusions

ChatGPT has demonstrated the ability to generate accurate statistical code and outputs across multiple econometric methods, particularly when guided by human oversight to ensure reliable outcomes. However, while the outputs were generally correct, the structure and content of the generated code varied across users and sessions and were often less efficient than human-written code. Generative AI presents exciting opportunities to enhance and support research, but it cannot replace human researchers in its current form. Human expertise remains essential when using ChatGPT to perform quantitative research.

Abbreviations

AI

Artificial Intelligence

ATT

Average Treatment Effect on the Treated

Diff-in-Diff

Difference-in-differences

IPTW

Inverse Probability Treatment Weighting

RD

Regression Discontinuity

Authors’ contributions

DW helped conceptualize the project, designed the methods, ran the experiment for Stata, analyzed the data, and drafted the manuscript. ET designed the methods, ran the experiment for Python, analyzed the data, and drafted the manuscript.TT helped conceptualize the project, designed the methods, ran the experiment for R, analyzed the data, and edited the manuscript. DX helped conceptualize the project, designed the methods, validated results, analyzed the data, and edited the manuscript. NM helped conceptualize the project, design the methods, and edit the manuscript. LS supervised the project. All authors read and approved the final manuscript.

Funding

The authors received no financial support for this research.

Data availability

All data generated or analyzed during this study are included in this published article and its supplementary files.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.