Three large language models demonstrate competitive performance in Traditional Chinese Medicine national medical licensing examinations over two years

Abstract

Large language models (LLMs) are increasingly integrated into medical education and assessment. However, their performance in high-stakes, non-English medical licensing examinations—particularly within culturally distinct medical systems such as Traditional Chinese Medicine (TCM)—remains insufficiently evaluated. This study aimed to systematically compare the performance of Gemini 3 Pro, DeepSeek V3.1, and GPT-5.2 in the Chinese National Traditional Chinese Medicine Licensing Examination (TCMLE) over two consecutive years (2023 and 2024), and to assess their potential implications for TCM education. All original examination questions from the 2023 and 2024 TCMLE (600 questions per year), encompassing all official question types and examination units, were independently input into each model in Chinese. Model responses were evaluated based on accuracy. Comparative analyses across models, question types, and examination units were conducted using chi-square tests, with statistical significance set at P < .05. DeepSeek V3.1 demonstrated significantly higher overall accuracy than Gemini 3 Pro and GPT-5.2 in both 2023 (87.1%) and 2024 (86.7%) (P < .001 for all comparisons). Gemini 3 Pro exhibited moderate and relatively stable performance across both years, whereas GPT-5.2 achieved the lowest overall accuracy despite a modest improvement from 2023 to 2024. Notably, DeepSeek V3.1 showed particular strength in structured and clinically oriented question formats and in foundational knowledge units. Linguistic and cultural alignment plays a critical role in LLM performance on specialized medical licensing examinations. Locally optimized models such as DeepSeek V3.1 may serve as valuable auxiliary tools in TCM education, particularly for examination preparation and knowledge reinforcement, although careful human oversight remains essential.

Introduction

The rapid evolution of large language models (LLMs) has expanded their influence into specialized professional domains, including medical education, assessment, and high-stakes licensing examinations¹. Trained on vast and diverse textual corpora, LLMs demonstrate advanced capabilities in semantic comprehension, knowledge integration, and contextual reasoning, offering new opportunities to support medical learning, self-assessment, and examination preparation^2,3.

Among widely deployed models, the Generative Pre-trained Transformer (GPT) series developed by OpenAI has established performance benchmarks across multiple medical examinations, including the United States Medical Licensing Examination (USMLE)^4,5. Its latest iteration, GPT-5.2, further enhances reasoning capacity and multilingual support. However, its training data remain predominantly English-based, which may limit its effectiveness in linguistically nuanced and culturally specific medical systems such as Traditional Chinese Medicine (TCM).

Google’s Gemini series, particularly Gemini 3 Pro, emphasizes multimodal integration and cross-domain reasoning, demonstrating strong performance in scientific and logical problem-solving tasks^6–8. Despite these strengths, its performance in TCM-specific contexts—where clinical reasoning is grounded in classical literature, syndrome differentiation, and culturally embedded diagnostic frameworks—has not yet been systematically evaluated.

In parallel, the DeepSeek series, developed in China, has attracted increasing attention for its open-source accessibility, cost efficiency, and close alignment with Chinese linguistic and cultural contexts^9–11. DeepSeek V3.1 incorporates extensive TCM classical texts, modern educational materials, and domestic medical regulations into its training corpus. In addition, it employs advanced techniques such as chain-of-thought prompting and a mixture-of-experts architecture to enhance reasoning accuracy and transparency in Chinese-language professional tasks.

Although the application of LLMs in standardized medical examinations has been actively explored, systematic comparative studies focusing on TCM licensing examinations remain scarce¹². The National Traditional Chinese Medicine Licensing Examination (TCMLE) assesses not only factual knowledge but also diagnostic reasoning, classical theory application, and integrative clinical decision-making within a distinct medical paradigm. As a high-stakes national licensure examination, the TCMLE provides a rigorous and ecologically valid benchmark for evaluating LLM adaptability to specialized, culturally grounded medical knowledge systems.

Accordingly, this two-year study aims to comprehensively evaluate and compare the performance of Gemini 3 Pro, DeepSeek V3.1, and GPT-5.2 in the 2023 and 2024 TCMLE. By analyzing model accuracy across question formats and knowledge domains, we seek to delineate their strengths and limitations in TCM-specific educational contexts and to inform the responsible integration of LLMs into TCM education and AI-assisted learning tools.

Methods

Data collection

Original test questions from the 2023 and 2024 Chinese National Traditional Chinese Medicine Licensing Examination (TCMLE) were collected for analysis. Each annual examination consisted of four units, with 150 multiple-choice questions per unit, yielding a total of 600 questions per year. According to official examination regulations, each question was equally weighted as one point, and a total score of 360 or higher was defined as the passing threshold.

The TCMLE includes several standardized question formats: A1-type questions (single-best-answer), A2-type questions (case summary–based single choice), A3/A4-type questions (case analysis with sequential subquestions), and B1-type questions (standard matching). These formats are designed to assess not only factual medical knowledge but also clinical reasoning, syndrome differentiation, and application of traditional Chinese medical theory. All TCMLE questions are text-based; the examination does not include image-based items.

Three large language models (LLMs)—Gemini 3 Pro, DeepSeek V3.1, and GPT-5.2—were evaluated using the complete set of examination questions from both years.

Study design

Model testing was conducted between December 12 and December 20, 2025. All models were accessed through their respective official platforms using standard user interfaces. Examination questions were entered in their original Chinese language format, and all model responses were generated in Chinese.

Each question was submitted only once to each model to reflect real-world, single-attempt examination conditions. To reduce stochastic variability while preserving reasonable response diversity, the temperature parameter was fixed at 0.7 for all models. Following the methodological framework of Liu et al.¹³, specific model functions were configured. Thinking mode (reasoning enhancement) was enabled for all three models: DeepSeek V3.1 used its official “deep thinking” mode, GPT-5.2 used “thinking mode”, and Gemini 3 Pro used its default reasoning-enhanced configuration. Web search was disabled for all models. For DeepSeek V3.1, this was achieved via the official platform setting that disables internet search. For GPT-5.2 and Gemini 3 Pro, which lack a direct disable option, an explicit instruction “Do not perform any web search when generating responses” was added at the beginning of each session. Memory was disabled via user-accessible settings, and each question was submitted in a new, independent chat session except for about 30 sets of sequential questions (about 90 sub-questions) that required contextual continuity, which were entered within the same session. The 600 questions from each examination year were input sequentially in the original examination order.

At the beginning of each chat session, the following prompt was entered into each model: “You are a professional traditional Chinese medicine practitioner. Please answer the following question from the National Traditional Chinese Medicine Licensing Examination. Select the most appropriate answer from the options. Do not perform any web search. Provide only the final answer (e.g., ‘A’).” For the sequential questions that required contextual continuity, the role instruction was given at the beginning of the session and not repeated for each sub-question unless a new session was initiated. Only one final answer was permitted for each question. A response was considered correct only if the model explicitly selected the official correct option. Incorrect selections, incomplete responses, or refusals to answer were all classified as incorrect.

Model performance was evaluated by comparing responses with the official answer keys. Accuracy was calculated at the overall level and further stratified by examination unit and question type. Comparative analyses across models were conducted to assess their reliability and practical utility in applying TCM knowledge. Based on observed performance patterns, potential implications for the integration of LLMs into TCM education, clinical reasoning training, and decision-support applications were explored.

Statistical analysis

All data were organized and managed using Microsoft Excel (Microsoft Corp., Redmond, WA, USA). Descriptive statistics were calculated as percentages to summarize model accuracy and pass rates. Visualizations were generated in Excel to facilitate interpretation of performance differences.

Inferential statistical analyses were conducted using SPSS version 23.0 (IBM Corp., Armonk, NY, USA). Differences in accuracy among the three models were evaluated using the chi-square test. Pairwise comparisons were performed to assess categorical differences between models. Additional chi-square analyses were applied to examine performance variations across examination units and question types. A two-sided P value of < .05 was considered statistically significant.

Error pattern analysis

To move beyond overall accuracy comparison and understand the nature of model errors, we performed a qualitative error analysis following the framework of Liu et al.¹³. All incorrect responses across the three models (Gemini 3 Pro, DeepSeek V3.1, and GPT-5.2) for the 2023 and 2024 TCMLE were collected. Two authors independently reviewed each incorrect response and classified it into one of four predefined error categories based on iterative review:

1. Terminology misinterpretation: misunderstanding of classical TCM terms or syndrome differentiation.

2. Incomplete clinical reasoning: failure to integrate multiple clinical clues, particularly in case-based A2/A3/A4 questions.

3. Cultural or classical text reference errors: incorrect recall or application of TCM classical literature.

4. Over-selection or under-selection: selecting too many or too few options in B1 matching or multiple-answer items.

Disagreements were resolved by consensus. The absolute number of each error type were calculated for each model.

Ethics statement

Ethical approval was not required for this study, as it did not involve human participants, patient data, or any identifiable personal information. All examination questions analyzed were obtained from publicly available sources.

Results

Overall performance

In the 2023 TCMLE, Gemini 3 Pro, DeepSeek V3.1, and GPT-5.2 achieved overall accuracy rates of 84.5%, 87.1%, and 76.3%, respectively. In the 2024 TCMLE, the corresponding accuracy rates were 79.5%, 86.7%, and 71.7%. Overall performance differed significantly among the three models in both examination years (all P < .001).

Post hoc pairwise comparisons showed that DeepSeek V3.1 consistently outperformed both Gemini 3 Pro (P = .001) and GPT-5.2 (P < .001) in each year. Gemini 3 Pro also achieved significantly higher accuracy than GPT-5.2 (P = .002), although its performance remained inferior to that of DeepSeek V3.1. Detailed performance metrics are summarized in Tables 1 and 2.

Table 1.

Overall performance comparison of Gemini 3 Pro, DeepSeek V3.1, and GPT-5.2 in the 2023 Traditional Chinese Medicine Licensing Examination.

2023 TCMLE							Pairwise comparisons
Accuracy	Gemini 3 Pro	DeepSeek V3.1	GPT-5.2	Cramer’s V	c	P	Gemini vs. DeepSeek		DeepSeek versus GPT		Gemini versus GPT
							c	P	c	P	c	P
Total	84.5% (507/600)	87.1% (523/600)	76.3% (458/600)	0.111	24.883	< .001	1.446	.23	24.369	< .001	12.381	< .001
Type of questions
A1	84.2% (160/190)	83.2% (158/190)	76.8% (146/190)	0.070	2.847	.24	0.078	.78	2.735	.10	2.460	.12
A2	82.4% (150/182)	85.2% (155/182)	74.7% (136/182)	0.100	5.543	.06	0.514	.47	5.238	.02	2.688	.10
A3/A4	88.4% (76/86)	93.0% (80/86)	76.7% (66/86)	0.176	5.218	.07	0.974	.32	6.280	.01	3.250	.07
B1	85.2% (121/142)	91.5% (130/142)	77.4% (110/142)	0.128	9.690	.008	2.124	.15	10.801	.001	2.648	.10
Unit of questions
Unit 1	89.3% (134/150)	88.0% (132/150)	81.3% (122/150)	0.080	4.821	.09	0.134	.71	4.678	.03	3.258	.07
Unit2	78.0% (117/150)	83.3% (125/150)	80.0% (120/150)	0.045	1.521	.47	1.377	.24	0.551	.46	0.200	.66
Unit3	87.3% (131/150)	88.0% (132/150)	72.0% (108/150)	0.162	19.643	< .001	0.034	.85	14.459	< .001	10.887	.001
Unit4	83.3% (125/150)	89.3% (134/150)	72.0% (108/150)	0.161	19.421	< .001	2.279	.13	14.852	< .001	5.561	.02

Table 2.

Overall performance comparison of Gemini 3 Pro, DeepSeek V3.1, and GPT-5.2 in the 2024 Traditional Chinese Medicine Licensing Examination.

2024 TCMLE							Pairwise comparisons
Accuracy	Gemini 3 Pro	DeepSeek V3.1	GPT-5.2	Cramer’s V	c	P	Gemini vs. DeepSeek		DeepSeek versus GPT		Gemini versus GPT
							c	P	c	P	c	P
Total	79.5% (477/600)	86.7% (520/600)	71.7% (430/600)	0.151	41.114	< .001	10.960	.001	40.930	< .001	9.970	.002
Type of questions
A1	84.5% (186/220)	88.2% (194/220)	64.5% (142/220)	0.256	43.099	< .001	1.235	.27	34.048	< .001	23.188	< .001
A2	74.9% (167/223)	84.8% (189/223)	75.8% (169/223)	0.097	6.341	.04	6.054	.01	4.852	.03	0.059	.81
A3/A4	75.8% (72/95)	83.2% (79/95)	75.8% (72/95)	0.081	1.237	.54	1.185	.28	1.185	.28	0.000	1.00
B1	83.9% (52/62)	93.5% (58/62)	75.8% (47/62)	0.200	7.441	.02	2.682	.10	6.254	.01	0.973	.32
Unit of questions
Unit 1	88.7% (133/150)	90.7% (136/150)	60.7% (91/150)	0.296	39.433	< .001	0.300	.58	39.113	< .001	33.333	< .001
Unit2	74.0% (111/150)	82.0% (123/150)	75.3% (113/150)	0.072	1.551	.46	2.882	.09	2.057	.15	0.053	.82
Unit3	80.7% (121/150)	87.3% (131/150)	76.0% (114/150)	0.121	6.569	.04	2.254	.13	6.120	.01	1.000	.32
Unit4	74.7% (112/150)	86.7% (130/150)	74.7% (112/150)	0.121	6.569	.04	6.120	.01	6.120	.01	0.000	1.00

To contextualise the LLMs’ performance, we compared their scores with human examinee data. The official passing threshold for the TCMLE is 360 points (60%) for both years, as set by the National Administration of Traditional Chinese Medicine. Detailed official statistical reports (e.g., mean scores, pass rates) are not publicly released; therefore, human examinee data are derived from Yikao Bang, a third-party medical examination platform that compiles and analyses examination results. According to Yikao Bang’s statistics, the 2023 TCMLE had a mean score of approximately 388.80 points (64.8%) and a pass rate of about 72%. In 2024, the mean score was approximately 357.85 points (59.6%), below the passing threshold, with a pass rate of about 48%. Converting the LLMs’ accuracy percentages to equivalent scores (accuracy × 600), DeepSeek V3.1 achieved 522.6 points (2023) and 520.2 points (2024); Gemini 3 Pro achieved 507.0 and 477.0 points; GPT-5.2 achieved 457.8 and 430.2 points, respectively. Thus, all three LLMs substantially outperformed the average human examinee in both examination years, maintaining scores well above the passing threshold even when the human mean fell below it.

Performance by question type

When stratified by question type, distinct performance patterns were observed across the models (Figs. 1 and 2).

Fig. 1.

Comparison of model accuracy by question type in the 2023 Traditional Chinese Medicine Licensing Examination. Accuracy rates of Gemini 3 Pro, DeepSeek V3.1, and GPT-5.2 are shown for A1-type (single-best-answer), A2-type (case summary), A3/A4-type (case analysis), and B1-type (standard matching) questions. DeepSeek V3.1 demonstrated consistently higher performance across all question types, with the largest advantage observed in B1-type questions.

Fig. 2.

Comparison of model accuracy by question type in the 2024 Traditional Chinese Medicine Licensing Examination. Accuracy rates of Gemini 3 Pro, DeepSeek V3.1, and GPT-5.2 are presented across four official question formats. Performance trends in 2024 mirrored those observed in 2023, with DeepSeek V3.1 maintaining superior accuracy, particularly for structured and clinically oriented question types.

In the 2023 TCMLE, the accuracy of each model varied across question types (Table 1). For A1-type questions (single-best-answer), Gemini 3 Pro achieved 84.2%, DeepSeek V3.1 83.2%, and GPT-5.2 76.8%, with no significant pairwise differences.

For A2-type questions (case summary), DeepSeek V3.1 (85.2%) outperformed GPT-5.2 (74.7%; P = .02), while Gemini 3 Pro (82.4%) did not differ significantly from either. For the more complex A3/A4-type questions (case analysis),, DeepSeek V3.1 (93.0%) significantly outperformed GPT-5.2 (76.7%; P = .01), with Gemini 3 Pro (88.4%) in between. For B1-type questions (standard matching), DeepSeek V3.1 (91.5%) again outperformed GPT-5.2 (77.4%; P = .001), while Gemini 3 Pro (85.2%) did not differ significantly from either.

In the 2024 TCMLE, the accuracy of each model varied across question types (Table 2). For A1-type questions, DeepSeek V3.1 achieved the highest accuracy (88.2%), followed by Gemini 3 Pro (84.5%) and GPT-5.2 (64.5%). Pairwise comparisons indicated statistically significant differences among all models (P < .001), except between DeepSeek V3.1 and Gemini 3 Pro (P = .28). For A2-type questions, DeepSeek V3.1 again demonstrated superior performance (84.8%), whereas Gemini 3 Pro (74.9%) and GPT-5.2 (75.8%) exhibited comparable accuracy. The difference between DeepSeek V3.1 and Gemini 3 Pro was statistically significant (P = .01), while no significant difference was observed between Gemini 3 Pro and GPT-5.2 (P = .81).For A3/A4-type questions, DeepSeek V3.1 achieved an accuracy of 83.2%, marginally higher than that of Gemini 3 Pro and GPT-5.2 (both 75.8%). However, none of the pairwise differences reached statistical significance (all P > .05). For B1-type questions, DeepSeek V3.1 demonstrated a pronounced advantage, attaining an accuracy of 93.5%, compared with 83.9% for Gemini 3 Pro and 75.8% for GPT-5.2. The difference between DeepSeek V3.1 and GPT-5.2 was statistically significant (P = .012), whereas the remaining pairwise comparisons were not.

Across the 2023 and 2024 TCMLE, DeepSeek V3.1 consistently outperformed both Gemini 3 Pro and GPT-5.2 in overall accuracy, achieving 87.1% and 86.7%, respectively. Gemini 3 Pro maintained relatively stable performance (84.5% in 2023, 79.5% in 2024), whereas GPT-5.2 showed the lowest accuracy in both years (76.3% and 71.7%). Notably, all three models exhibited a modest decline in accuracy from 2023 to 2024, coinciding with the reported decrease in human examinee mean scores over the same period (from approximately 64.8 to 59.6%). DeepSeek V3.1 demonstrated particular strength in structured question formats (A1 and B1) and foundational knowledge units, whereas its advantage over Gemini 3 Pro was less pronounced in complex case-based questions (A3/A4). GPT-5.2 consistently lagged behind the other two models across most question types and units, highlighting the challenges of cross-cultural and cross-linguistic transfer for English-centric LLMs in the TCM domain.

Performance by examination unit

Analysis across the four examination units revealed model-specific performance variations (Figs. 3 and 4).

Fig. 3.

Comparison of model accuracy across the four examination units in the 2023 Traditional Chinese Medicine Licensing Examination. Model performance is shown for Unit 1 (foundational TCM knowledge), Unit 2, Unit 3, and Unit 4. DeepSeek V3.1 and Gemini 3 Pro demonstrated strong performance in Unit 1, while GPT-5.2 exhibited substantially lower accuracy. DeepSeek V3.1 achieved the highest overall accuracy across most units.

Fig. 4.

Comparison of model accuracy across the four examination units in the 2024 Traditional Chinese Medicine Licensing Examination. Accuracy distributions across the four examination units are displayed for Gemini 3 Pro, DeepSeek V3.1, and GPT-5.2. Consistent with the 2023 results, DeepSeek V3.1 maintained superior performance across multiple knowledge domains, with particularly strong advantages in Units 3 and 4.

In the 2023 TCMLE, model performance varied across the four examination units (Fig. 3). In Unit 1 (foundational TCM knowledge), Gemini 3 Pro (89.3%) and DeepSeek V3.1 (88.0%) achieved comparable accuracy, whereas GPT-5.2 (81.3%) lagged behind. In Unit 2, DeepSeek V3.1 (83.3%) slightly outperformed GPT-5.2 (80.0%) and Gemini 3 Pro (78.0%), though none of the pairwise differences reached statistical significance. In Unit 3, DeepSeek V3.1 (88.0%) and Gemini 3 Pro (87.3%) performed similarly and both significantly outperformed GPT-5.2 (72.0%; P < .001 for both comparisons). In Unit 4, DeepSeek V3.1 (89.3%) achieved the highest accuracy, significantly outperforming both Gemini 3 Pro (83.3%; P = .02) and GPT-5.2 (72.0%; P < .001).

In the 2024 TCMLE (Fig. 4), DeepSeek V3.1 consistently achieved the highest accuracy across all four units. In Unit 1, DeepSeek V3.1 (90.7%) and Gemini 3 Pro (88.7%) both outperformed GPT-5.2 (60.7%; P < .001 for both comparisons). In Unit 2, no statistically significant differences were observed among the three models (DeepSeek V3.1 82.0%, GPT-5.2 75.3%, Gemini 3 Pro 74.0%; all P > .05). In Unit 3, DeepSeek V3.1 (87.3%) significantly outperformed GPT-5.2 (76.0%; P = .01), while Gemini 3 Pro (80.7%) did not differ significantly from either. In Unit 4, DeepSeek V3.1 (86.7%) significantly outperformed both Gemini 3 Pro (74.7%; P = .01) and GPT-5.2 (74.7%; P = .01).

Across the two examination years, DeepSeek V3.1 demonstrated the most consistent and highest performance, ranking first in three of four units in 2023 and all four units in 2024. Gemini 3 Pro showed moderate performance, with notable strength in Unit 1 (foundational knowledge) but weaker performance in Unit 4 (clinical integration) in 2024. GPT-5.2 consistently underperformed, particularly in Unit 1 (where its accuracy dropped from 81.3% in 2023 to 60.7% in 2024) and Unit 4 (72.0% in both years). These patterns suggest that domain-specific training (DeepSeek V3.1) enhances performance across diverse knowledge areas, whereas English-centric models (GPT-5.2) struggle most in units requiring classical TCM theory and culturally embedded clinical reasoning.

Error pattern analysis

To understand the nature of model errors, we categorized all incorrect responses (n = 685 across the three models and two examination years) into four predefined error types (see Methods). The distribution of error patterns across Gemini 3 Pro, DeepSeek V3.1, and GPT-5.2 is presented in Table 3.

Table 3.

Distribution of error patterns across LLMs.

Error pattern	Gemini 3 Pro	DeepSeek V3.1	GPT-5.2
Terminology misinterpretation	31	24	58
Incomplete clinical reasoning	66	41	88
Cultural or classical text reference errors	48	35	72
Over-selection or under-selection	71	57	94

DeepSeek V3.1 made substantially fewer errors in terminology misinterpretation (24 vs. 31 for Gemini 3 Pro and 58 for GPT-5.2) and cultural/classical text reference errors (35 vs. 48 and 72), reflecting its TCM-specific training and better linguistic-cultural alignment. However, it still showed a considerable number of over-/under-selection errors (57), suggesting occasional difficulty with instruction adherence in complex matching or multiple-answer questions.

Gemini 3 Pro had the highest number of incomplete clinical reasoning errors (66), indicating relative weakness in multi-step integration of clinical information in case-based scenarios.

GPT-5.2 consistently had the highest error counts across all four categories, particularly in terminology (58), clinical reasoning (88), and cultural/textual errors (72), consistent with its English-centric training and limited exposure to TCM-specific knowledge.

Discussion

Principal findings

This study provides a systematic, two-year comparative evaluation of Gemini 3 Pro, DeepSeek V3.1, and GPT-5.2 using the 2023 and 2024 Chinese National Traditional Chinese Medicine Licensing Examination (TCMLE). Across both examination cycles, DeepSeek V3.1 consistently demonstrated superior accuracy and stability, outperforming Gemini 3 Pro and GPT-5.2 in overall performance, across question types, and across examination units. Gemini 3 Pro exhibited moderate and relatively stable performance, whereas GPT-5.2, despite modest improvement from 2023 to 2024, remained the lowest-performing model overall.

Interpretation in the context of the TCMLE

The TCMLE is among the most culturally and conceptually distinctive national medical licensing examinations worldwide. Beyond assessing factual knowledge, it emphasizes classical TCM theory, syndrome differentiation, and integrative clinical reasoning within a uniquely Chinese medical paradigm. As TCM constitutes a core pillar of China’s health care system, the TCMLE serves both as a professional gateway and as a benchmark of clinical competence. Evaluating LLM performance in this context therefore offers a rigorous test of their adaptability to specialized, non-Western medical knowledge systems.

Role of linguistic and cultural adaptation

The superior performance of DeepSeek V3.1 is most plausibly attributable to its extensive Chinese-language and TCM-specific training corpus, which integrates classical medical texts, contemporary educational resources, and localized clinical guidelines. Such linguistic and cultural alignment likely facilitates more accurate interpretation of TCM terminology, conceptual frameworks, and diagnostic logic. In contrast, GPT-5.2—despite its advanced general reasoning capabilities—remains predominantly trained on English-centric data, which may constrain its effectiveness in culturally embedded examinations.

This interpretation is consistent with prior literature demonstrating that region-specific training substantially enhances LLM performance in corresponding licensing contexts. Takagi et al. reported that GPT-4 markedly outperformed GPT-3.5 on the Japanese Medical Licensing Examination, underscoring the importance of linguistic adaptation¹⁴. Similarly, Liu et al. showed that Chinese-trained LLMs achieved significantly higher accuracy on Chinese medical examinations compared with models dominated by English-language training data⁵.

Comparison with prior work on GPT models

GPT-5.2’s comparatively lower accuracy further illustrates that state-of-the-art general reasoning alone is insufficient for mastering the semantic depth and cultural embeddedness of TCM knowledge. This finding corroborates earlier studies by Wang et al. and Zong et al., in which GPT-3.5 failed to meet passing thresholds on Chinese medical licensing examinations due to discrepancies in medical policy, epidemiology, and cultural context^15,16. Fang et al. explored cross-lingual strategies by translating Chinese questions into English before inputting them into ChatGPT but observed no significant improvement, highlighting persistent challenges in cross-cultural medical reasoning¹⁷. This finding is consistent with broader evidence from systematic reviews showing that LLMs consistently underperform on non-English or culturally specific medical examinations compared with English-language ones^18,19.

Although Gemini 3 Pro demonstrated stable cross-year performance, its accuracy consistently lagged behind that of DeepSeek V3.1, particularly in examination units emphasizing classical TCM theory and syndrome differentiation. This suggests that while general-purpose and multimodal models possess strong logical capabilities, they may lack the domain-specific depth required for highly specialized medical disciplines such as TCM.

Performance gap between TCM and western medical licensing examinations

A key novelty of the present study is the demonstration that state-of-the-art large language models—despite achieving near- or above-90% accuracy on multiple Western and general medical licensing examinations—exhibit substantially lower performance on the Traditional Chinese Medicine Licensing Examination. Recent studies have reported that advanced LLMs such as GPT-5, Gemini 2.5 Pro, and DeepSeek-R1 achieve accuracy rates exceeding 90% on the United States Medical Licensing Examination (USMLE), the Japanese National Medical Examination (JNME), and the Chinese National Medical Licensing Examination (CNMLE)^13,18,20,21. For example, Liu et al.¹³ found that Gemini 2.5 Pro achieved 97.2% accuracy on the JNME, with near-perfect performance on general medical knowledge questions. In contrast, even the best-performing model in our study (DeepSeek V3.1) reached only 87.1% and 86.7% accuracy on the 2023 and 2024 TCMLE, respectively, and GPT-5.2 dropped to approximately 71–76%.

This performance gap of approximately 10–20 percentage points is not merely a matter of language proficiency. Rather, it reflects the unique challenges posed by TCM’s classical literature, syndrome differentiation, and culturally embedded diagnostic frameworks—domains that are poorly represented in predominantly English-centric and biomedical training corpora. The locally optimized DeepSeek V3.1, which incorporates TCM-specific texts into its training, performed better than GPT-5.2 but still fell short of its own reported performance on general medical examinations. These findings suggest that “textbook-level medical knowledge” in LLMs, as defined by Liu et al.¹³, is domain- and culture-dependent.

Thus, the comparatively poorer performance observed in the TCM examination constitutes a central contribution of this study, highlighting the need for culturally adapted training data and careful validation before deploying LLMs in non-Western or traditional medicine educational contexts.

Comparison with human examinee performance

To help readers unfamiliar with the TCMLE interpret the practical meaning of the reported scores, we compared the LLMs’ performance with human examinee data.

The official passing threshold is 360 points (60%). Human mean scores, derived from a third-party platform (Yikao Bang) because no official detailed statistics are publicly released, were approximately 388.8 points (2023) and 357.9 points (2024). All three LLMs achieved equivalent scores substantially above the human means and the passing threshold in both years (see Results for exact conversions).

Several caveats apply. First, the human data are from a third-party source and should be interpreted as descriptive benchmarks only. Second, LLMs were not subject to the same testing conditions as human test-takers (e.g., time limits, fatigue, psychological stress). Third, the TCMLE primarily assesses factual recall and basic clinical reasoning, not the full range of competencies required for clinical practice. Therefore, superior LLM performance on this examination reflects strong knowledge retrieval capacity rather than clinical competence, and should not be overinterpreted as readiness for unsupervised clinical deployment.

Insights from question-type and unit-level analyses

Stratified analyses by question type further elucidated model characteristics. DeepSeek V3.1 performed particularly well on A1-type (single-best-answer) and B1-type (standard matching) questions, which primarily assess structured knowledge retrieval and well-defined clinical associations. In contrast, all models demonstrated lower and more variable performance on A3/A4-type (case-based) questions that require multi-step reasoning and contextual integration. This limitation has been consistently reported in prior evaluations of LLMs across medical examinations^4,16, including studies of the Chinese National Medical Licensing Examination¹⁶.

At the unit level, DeepSeek V3.1 showed notable strengths in Unit 1 (foundational TCM knowledge) and Unit 4 (clinical integration), indicating balanced proficiency in both theoretical understanding and applied reasoning. These findings reinforce the conclusion that model performance is strongly associated with the depth and relevance of domain-specific training data, as previously emphasized by Liu et al.⁵.

The error analysis provides critical insights beyond simple accuracy rankings. DeepSeek V3.1’s lower error counts in terminology misinterpretation and classical text references underscore the importance of domain-specific and culturally aligned training corpora for specialized medical licensing examinations. Conversely, its notable over-/under-selection errors (57 incorrect items) suggest that even well-aligned models may struggle with instruction adherence in complex question formats, such as B1 matching questions.

Gemini 3 Pro’s predominant errors in incomplete clinical reasoning (66 errors) indicate that while its factual recall may be adequate, it lacks robust multi-step integration of clinical information—a finding consistent with the error patterns reported by Liu et al.¹³ for LLMs on the Japanese National Medical Examination, where models frequently failed to prioritize appropriate actions in questions with complex contextual information.

GPT-5.2’s consistently high error counts across all categories, especially in terminology (58 errors) and cultural/textual references (72 errors), highlight the persistent challenge of cross-cultural and cross-linguistic transfer for English-centric models.

These error patterns also have practical implications for TCM education. When students use LLMs as auxiliary tools, they should be particularly cautious with Gemini 3 Pro’s answers to case-based questions, and with DeepSeek V3.1’s answers to matching or multiple-answer questions. Future model fine-tuning should prioritize clinical reasoning chains and instruction-following accuracy, in addition to knowledge retrieval. As noted by Liu et al.¹³ qualitative error taxonomies are essential to move from “which model is better” to “why and under what conditions models fail.” Our analysis follows this methodological precedent.

Educational implications

From an educational perspective, these findings suggest that DeepSeek V3.1 may serve as a valuable auxiliary tool in TCM education, supporting knowledge consolidation, formative assessment, and simulation-based learning when used under appropriate supervision. Its strong performance in structured and clinically oriented domains indicates potential utility in diagnostic reasoning exercises and exam preparation.

However, LLMs should not be viewed as substitutes for formal instruction or independent clinical judgment. Instead, they may function best as complementary tools that enhance learning efficiency while preserving the central role of human expertise in medical education.

Ethical considerations and limitations

Despite these promising findings, several limitations must be acknowledged. LLMs are inherently fallible and may generate incorrect responses with high confidence—a phenomenon reported for multiple GPT-based models¹⁶. Such errors pose potential risks in both educational and clinical settings. In addition, the stochastic nature of model outputs and observed inter-year variability underscore the need for continuous validation prior to deployment in high-stakes environments. Over-reliance on LLMs may also impede the development of independent clinical reasoning and critical thinking, which remain core competencies in medical training.

Ethical and practical concerns—including data privacy, algorithmic transparency, and potential bias—require careful consideration. Prior studies have shown that LLMs may inherit inaccuracies or biases from heterogeneous training data²², highlighting the necessity of robust governance frameworks to ensure responsible integration into medical education and clinical workflows²³.

Study limitations and future directions

This study has several methodological limitations. First, the analysis was primarily descriptive and comparative; predictive modeling was not employed to examine how factors such as question difficulty or cognitive demand influence model accuracy. Future research should incorporate regression-based or machine learning approaches to identify determinants of LLM performance in TCM assessments.

Second, only text-based questions were evaluated; image-based items that are integral to modern TCM diagnostics, such as tongue and pulse assessment, were not included. However, it should be noted that the TCMLE is a purely written examination, and according to the official syllabus, it contains no image-based questions (e.g., radiographs, CT images, or photographs of tongue or pulse findings). Therefore, evaluating LLM performance on visual diagnostic tasks was not possible using this authentic examination dataset. Given that LLMs generally underperform on image-based medical tasks^24–26, future studies should address this gap by adopting multimodal evaluation designs.

Third, the rapid evolution of AI technologies limits the temporal generalizability of these findings. Newer model versions may exceed the performance reported here. Finally, the focus on two consecutive examination cycles with a fixed sample size of 600 questions per year may limit extrapolation to other examination periods or broader TCM knowledge domains.

Future studies should incorporate repeated sampling designs or multi-user simulation frameworks to quantify output variability and its impact on real-world educational use.

Conclusion

This two-year evaluation demonstrates that DeepSeek V3.1 significantly outperforms Gemini 3 Pro and GPT-5.2 on the Chinese Traditional Chinese Medicine Licensing Examination, particularly in structured question formats and foundational knowledge domains. The findings underscore the importance of linguistic and cultural alignment when deploying large language models in specialized, non-English medical education settings. While locally optimized LLMs such as DeepSeek V3.1 show promise as supportive tools for TCM education and clinical reasoning, their application should be accompanied by continuous validation and robust ethical oversight.

Author contributions

Dr. Huang had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Concept and design: Huang, Xue, Yao, J Pan. Acquisition, analysis, or interpretation of data: All authors. Drafting of the manuscript: Huang, Du, Y Pan, Ng. Critical revision of the manuscript for important intellectual content: All authors. Statistical analysis: Huang, Du, Y Pan, Ng, Ding. Obtained funding: Huang, Xue. Administrative, technical, or material support: Ng. Supervision: Xue, Pan.

Funding

This work was supported by the following educational and curriculum development grants: Shanghai Municipal Key Course Development Program for Higher Education (2025); “AI+” Curriculum Development Program of Shanghai Jiao Tong University (2024); Shanghai–Ottawa Joint School of Medicine English-Taught Curriculum Development Program (2025).

Data availability

All data generated or analyzed during this study are available from the corresponding author upon reasonable request.

Declarations

Competing interests

The authors declare no competing interests.