What the performance of multimodal LLMs on a national licensing exam teaches us about occupational therapy education

Abstract

Background

Multimodal large language models (LLMs) are increasingly adopted in occupational therapy (OT) education, offering new opportunities for learning and assessment. However, their ability to handle professional licensing examinations remains unexplored. This study aimed to provide educational insights for OT educators by analyzing how multimodal LLMs perform on the Japanese National Examination for Occupational Therapists (JNEOT), particularly focused on visually-based questions.

Methods

The complete 199-item 2025 JNEOT (consisting of 177 text-only questions and 22 visually-based questions) was presented to OpenAI o3, Gemini 2.5, GPT-4.5, and Claude 3.7 in a zero-shot manner. Official answer keys served as the reference standard. Overall correct response rates and subscores were calculated for text-only versus visually-based and general versus practical items. We examined differences using Cochran’s Q, and significant omnibus results were followed by Bonferroni-corrected McNemar tests.

Results

All models achieved passing-level scores (82.4–91.0%) on text-based questions but showed a marked decline on visually based items (50.0-68.2%). No significant inter-model differences were found for visual tasks. Ten OT-specific questions were answered incorrectly by all models.

Conclusions

These findings suggest that LLMs can effectively reinforce foundational OT knowledge but remain limited in visual interpretation and contextual reasoning. For educators, the results offer practical guidance on using LLMs to enhance theoretical learning while maintaining OT educator-guided approaches for reflective and experiential competence development.

Trial registration

Not applicable.

Introduction

Artificial Intelligence (AI)-driven technologies have catalyzed a paradigm shift in medical and healthcare professional education [1, 2]. Among these technologies, large language models (LLMs) have emerged as particularly transformative tools for medical education, particularly due to their remarkable capabilities in natural language processing tasks such as medical translation, medical question-answering, medical dialog summarization, and medical imaging analysis [3, 4]. As LLMs become ubiquitous, medical and healthcare educators urgently need to understand their specific strengths and limitations to guide curriculum development and student assessment. Such understanding is essential for fostering AI literacy and ensuring that these technologies enhance rather than undermine professional judgment and human-centered care [5]. Despite their transformative potential, LLMs present several challenges that directly impact educational implementation, including ethical concerns, AI hallucinations, information bias, privacy issues, and data bias, all of which require careful evaluation by educators [6, 7]. Recently, national examination questions consisting of multiple-choice questions (MCQs) have been widely used as standardized benchmarks to assess LLM performance in the medical [8–11] and dental fields [9, 12–14]. It has been reported that the accuracy of LLMs differs between English-speaking and non-English-speaking countries, and validation of LLM performance in national exams across various languages is currently underway [8, 10]. Previous research has shown that current LLMs can achieve a passing-level accuracy rate on these examinations [15–17]. However, their performance tends to be lower for visually-based questions compared with text-based questions [13, 14, 18], suggesting limitations in multimodal reasoning that remain relevant for medical and healthcare education.

As a field of rehabilitation medicine, occupational therapy (OT) focuses on community-based practice, client-centered and occupation-based approaches to enhance an individual’s ability to perform activities of daily living and participate in society [19]. Given this practical orientation, OT curricula focus on developing students’ ability to integrate theory with practical experience in clinical settings, requiring the integration of multiple forms of clinical reasoning, including procedural, interactive, conditional, and narrative reasoning [19, 20]. According to the World Federation of Occupational Therapists (WFOT) Human Research Project 2020, Japan ranked second globally in the number of practicing occupational therapists as well as in the number of WFOT-approved education programmes [21], making it a key context for examining AI applications in OT education. Despite growing interest in using LLMs such as ChatGPT in OT education [22, 23], systematic evidence remains limited. Research exists on the Korean NEOT using ChatGPT [23] and Japanese physical therapy examinations using multimodal LLMs [24], yet comprehensive analysis in OT remains lacking. Notably, no study has investigated how differences between English- and non-English-language contexts affect the educational use of LLMs, leaving a critical gap in understanding how AI can inform global OT curricula.

Herein, this study aimed not merely to evaluate model performance but to derive educational insights for OT educators by analyzing the performance of four multimodal LLMs on the latest JNEOT to reveal insights about OT education and assessment. Beyond assessing accuracy, we examined response characteristics across different types of questions (text-based vs. visually-based; general questions vs. practical questions) to identify which domains of knowledge are well supported by AI and which require continued educator-guided instruction. Through this analysis, we sought to clarify both the potential and limitations of AI in OT education, and to inform the development of LLMs-based learning tools that support OT students and educators, particularly for non-English-speaking countries.

Materials and methods

Dataset

This study used the 60th JNEOT from February 2025 as a dataset. The JNEOT questions and answers are published on the Ministry of Health, Labour and Welfare of Japan (MHLW) website and are available to anyone free of charge [25]. This exam consists of 200 MCQs, each of which requires the selection of a certain number of correct answers from five options. The questions comprised 40 practical and 160 general questions. One hundred of those questions consisted of the common basic subjects of physical therapy and OT (anatomy, physiology, kinesiology, pathology, clinical medicine, and rehabilitation medicine). Another 100 questions consisted of the specialized subjects of OT, and 40 of specialized subject questions were practical questions. The exam includes visual materials such as clinical photographs, medical images (computed tomography, magnetic resonance imaging, and illustrations). The number of questions in the 60th JNEOT is shown in Table 1.

Table 1.

The number of questions in the 60th Japanese National examination for occupational therapists

	General (Basic & Clinical) questions	Practical (Case-based) questions	Total
PT and OT common questions	100	-	100
OT specific questions	60	40	100
Total	160	40

PT physical therapists, OT occupational therapists

One question that MHLW had officially withdrawn from scoring due to validity concerns was excluded from the dataset. As a result, a total of 199 questions were scored, including 177 text-only questions and 22 visually-based questions. The specialties of the questions were determined by two researchers (K.S. and S.A.) involved in education in the field of OT. Disagreements were resolved through consensus discussion with a third researcher (Y.M.).

No ethical approval or institutional review was required for this study, as it relied on exam questions published by the MHLW. All visual materials used in this study were originally supplied by the MHLW in PDF format and were converted to JPEG format for this research.

Multimodal large language models and prompting

Four multimodal LLMs were used in this study to examine their performance on the JNEOT. The chosen models were OpenAI o3 (OpenAI Global, San Francisco, CA, USA, launched April 16, 2025), ChatGPT-4.5 (GPT-4.5; OpenAI Global, launched February 27, 2025), Gemini 2.5 Pro Experimental (Gemini 2.5; Google, Mountain View, CA, USA, launched March 25, 2025), and Claude 3.7 Sonnet (Claude 3.7; Anthropic, San Francisco, CA, USA, launched February 24, 2025). Each LLM can process both textual and visual data in parallel, which enables the use of a variety of question formats. All questions were input into OpenAI o3 between May 3 and 4, 2025. The other three models received the same questions between March 27 and 30, 2025.

In this study, a zero-shot approach [26] was employed. No specialized prompt design or extra instructions were incorporated. The original Japanese text for each question, along with its answer choices, was entered into the prompt window exactly as provided. If a question included figures, tables, or images, those materials were presented to the multimodal LLMs in their original form, without any additional explanation.

Statistical analysis

To assess differences in correct response rates between the four models, we used Cochran’s Q test. When this omnibus test revealed significant differences, we performed post hoc analysis using McNemar’s test for all possible pairwise comparisons. The Bonferroni correction was applied to these six pairwise comparisons to maintain an overall family-wise error rate of 0.05. All analyses were performed using IBM SPSS Statistics 27 (IBM SPSS, Inc., Armonk, NY, USA).

To summarize the overlap in correctly answered items across models, we created a descriptive UpSet-style set-intersection plot from a binary (correct/incorrect) matrix of 199 items and four models.

Results

Overall performance across the entire study

Table 2 presents the correct response rates and 95% confidence intervals for the four LLMs. Gemini 2.5 Pro produced the highest overall correct response rate with 91.0% (95% CI 86.1–94.6), followed by OpenAI o3 with 88.4% (83.2–92.5), GPT-4.5 with 83.4% (77.5–88.3) and Claude 3.7 Sonnet with 82.4% (76.4–87.4). A Cochran’s Q test confirmed a significant difference between the four models (p < 0.001). McNemar tests with Bonferroni adjustment showed that Gemini 2.5 outperformed GPT-4.5 (p = 0.011) and Claude 3.7 (p = 0.002); its advantage over OpenAI o3 was not significant (adjusted p = 1.000), and no other pairwise contrast reached significance (Table 3).

Table 2.

Correct response rates (%) and 95% CIs of the four LLMs

	OpenAI o3	GPT4.5	Claude 3.7	Gemini 2.5
All questions	88.4 (83.2–92.5)	83.4 (77.5–88.3)	82.4 (76.4–87.4)	91.0 (86.1–94.6)
Text-only questions	91.5 (86.4–95.2)	87.0 (81.1–91.6)	86.4 (80.5–91.1)	93.8 (89.2–96.9)
Visually-based questions*	63.6 (40.7–82.8)	54.6 (32.2–75.6)	50.0 (28.2–71.8)	68.2 (45.1–86.1)

^*Includes one or more images, illustrations, figures, or tables

Table 3.

P-value between overall correct response rates of four LLMs

		OpenAI o3	GPT4.5	Claude 3.7	Gemini 2.5
All questions	OpenAI o3	-	p = 0.226	p = 0.076	p = 1.000
	GPT4.5	-	-	p = 1.000	p = 0.011
	Claude 3.7	-	-	-	p = 0.002
	Gemini 2.5	-	-	-	-
Text-only questions	OpenAI o3	-	p = 0.420	p = 0.249	p = 1.000
	GPT4.5	-	-	p = 1.000	p = 0.039
	Claude 3.7	-	-	-	p = 0.019
	Gemini 2.5	-	-	-	-
Visually-based questions*	OpenAI o3	-	N.S.	N.S.	N.S.
	GPT4.5	-	-	N.S.	N.S.
	Claude 3.7	-	-	-	N.S.
	Gemini 2.5	-	-	-	-

^*Includes one or more images, illustrations, figures, or tables

N.S., Not siginificant (Cochran’s Q test) Because Cochran’s Q test did not indicate a significant difference among the samples, no multiple comparisons were performed.

Influence of question format

Of the 199 questions, 177 were text-only and 22 included one or more clinical images, radiographs, or illustrations. When the input was text-only, the correct response rate was high for all models (Tables 2 and 3): Gemini 2.5 answered 93.8% of the items correctly, OpenAI o3 91.5%, GPT-4.5 87.0%, and Claude 3.7 86.4%. Cochran’s Q again indicated heterogeneity (p < 0.001); after correction, significant differences were observed in the pairwise comparisons between Gemini 2.5 and GPT-4.5 (p = 0.039), as well as between Gemini 2.5 and Claude 3.7 (p = 0.019). For the visually-based questions, the ranking changed little, Gemini 2.5 68.2%, OpenAI o3 63.6%, GPT-4.5 54.6%, Claude 3.7 50.0%, but Cochran’s Q was not significant (p = 0.104), so no individual contrasts survived adjustment (Tables 2 and 3). The figures show a general decrease of 23–36% points when image interpretation was required.

Performance within specialties

Analysis of the nine specialties recognized in the national curriculum revealed uniformly high scores in the basic sciences (Table 4). All models exceeded 94% in anatomy, physiology, kinesiology, pathology, clinical psychology and clinical medicine, with both Gemini 2.5 and OpenAI o3 achieving perfect scores in several categories. OT specific content, on the other hand, was more difficult: across the 81 text-based OT questions, Gemini 2.5 scored 87.7% and OpenAI o3 scored 85.2%, while GPT-4.5 and Claude 3.7 fell below 80%. The gap widened when images were added, with correct response rates for OT visuals ranging from 63.2% (Gemini 2.5) to 47.4% (Claude 3.7). The UpSet plot analysis revealed that all four LLMs answered ten questions incorrectly. Four of these questions were text-based, and six were visual-based (Fig. 1).

Table 4.

Correct response rates (%) of four LLMs by specialty and by question format (All / Text‑only / Visually‑based)

Specialty	Questions (n)	OpenAI o3	GPT-4.5	Claude 3.7	Gemini 2.5
All questions
Total	199	88.4	83.4	82.4	91.0
Occupational therapy	100	80.0	73.0	71.0	83.0
Anatomy	20	100	95.0	95.0	100
Physiology	18	100	94.4	94.4	100
Kinesiology	10	100	80.0	80.0	100
Pathology	7	100	100	100	100
Clinical psychology	6	100	100	100	100
Clinical medicine	37	94.6	97.3	94.6	97.3
Rehabilitation medicine	1	0	0	100	100
Text-only questions
Total	177	91.5	87.0	86.4	93.8
Occupational therapy	81	85.2	77.8	76.5	87.7
Anatomy	18	100	100	100	100
Physiology	18	100	94.4	94.4	100
Kinesiology	9	100	77.8	77.8	100
Pathology	7	100	100	100	100
Clinical psychology	6	100	100	100	100
Clinical medicine	37	94.6	97.3	94.6	97.3
Rehabilitation medicine	1	0	0	100	100
Visually-based questions*
Total	22	63.6	54.6	50.0	68.2
Occupational therapy	19	57.9	52.6	47.4	63.2
Anatomy	2	100	50.0	50.0	100
Kinesiology	1	100	100	100	100

^*Includes one or more images, illustrations, figures, or tables

Fig. 1.

Distribution of correct and incorrect answers for each LLM in OT-specific questions. Upper panel: Bar graph displays the number of questions correctly responded exclusively by text-only (green) or visual-based (purple) questions, and the number of questions incorrectly responded by all models for text-only (light gray) and visually-based (dark gray) questions. Lower panel: UpSet plot showing the intersection of correct responses among models (OpenAI o3, GPT4.5, Claude3.7, and Gemini2.5). The horizontal bars positioned on the left side indicate the total number of correct responses achieved by each individual model, while the linked dot patterns show which specific model combinations produced correct answers for particular questions

A representative example of a visually-based and practical OT question answered incorrectly by all LLMs

One of the practical questions (P06) presented at the 60th JNEOT, it was a visually-based question including OT evaluation and treatment approach for motor function impairment in cerebrovascular disorders. This question presented OT evaluation data for a 52-year-old man with right hemiparesis (Brunnstrom recovery stage: upper extremity III, finger III, lower extremity V) and required the selection of the most appropriate upper-limb training from five options. The correct answer was “4. Passing a ring behind the back,” whereas several LLMs, including ChatGPT, incorrectly selected “3. Catching a ball with both hands.”

Discussion

This study evaluated the performance of several multimodal LLMs in the JNEOT, with all models showing high correct answer rate. While text-based questions showed a higher correct answer rate, the correct answer rate for visually-based questions decreased with all models. These differential performance patterns provide insights into both AI’s current capabilities and its limitations in assessing the full spectrum of occupational therapy competencies, with important implications for the integration of AI in OT education and assessment.

Text-based performance and international relevance

The models demonstrated robust performance on text-based questions (86.4–93.8%), with particularly high accuracy in foundational subjects where all models exceeded 94% (Table 4). This strong performance suggests that LLMs can serve as reliable supplementary tools for the self-study of biomedical and fundamental knowledge in OT education. Students can use LLMs to check and reinforce their understanding of basic science concepts at any time, which may enhance learning efficiency. For educators, these findings imply that LLMs may help reduce the burden of routine instructional tasks allowing them to devote more time to teaching higher-order skills and practices. Compared to existing international studies, this study significantly expands the evidence base of LLMs in OT education. Lee et al. [19] evaluated ChatGPT3.5 on the Korean NEOT using text-only questions, whereas our study is the first multimodal analysis to test four state-of-the-art LLMs using both text and visual materials. These findings are internationally relevant because foundational biomedical knowledge is universal across health professions education, and similar challenges may arise in non-English-speaking contexts, particularly in regions such as South Korea and Taiwan, where OT curricula and licensing standards closely resemble those of Japan. However, unlike the medical and dental fields, no validation studies have examined LLM performance on OT national examinations in English-speaking countries. Therefore, caution is warranted when considering the generalizability of these findings to English-based testing environments.

Visual-based performance and representative error analysis

Performance declined significantly when visual information processing was required; accuracy on visual-based questions decreased by 26–36% compared to text-based questions across all models. Consistent with this pattern, the UpSet plot analysis (Fig. 1) shows that all four models missed ten items, six of which were visually-based and four of which were text-only. This reinforces the idea that vision-dependent tasks concentrate common errors. Additionally, we observed very few exclusive correct responses by any one model on OT-specific visuals, which aligns with the non-significant inter-model differences for visually-based questions. This drop due to visually-based questions was greater than the 10% decline [27] in the Japanese National Medical Examination and similar to the results of the Japanese National Dental Examination (23–35% decline) [13]. A closer examination of a representative visual and practice-oriented OT item presented in the Results section, one of the ten questions that all four LLMs answered incorrectly, provides two key challenges. First, the Brunnstrom approach, a Japan-specific clinical framework, posed a major barrier. Although widely used in Japan, it is not internationally standardized; global stroke rehabilitation research predominantly relies on more detailed assessments such as the Fugl-Meyer Assessment. The course six-stage classification of the Brunnstrom method likely hindered LLMs, trained primarily on English-language data, from performing stage-based therapeutic reasoning. Second, LLMs struggled to integrate textual information with visual materials for recovery-stage-based decision-making. Correctly selecting the appropriate upper-limb task required understanding that stage III limits voluntary movement to synergy patterns and excluding tasks requiring finger extension or isolated elbow extension. For example, “catching a ball with both hands” could be executed within the patient’s current functional level, whereas “passing a ring behind the back” promotes partial dissociation from synergy and supports progression from stage III to IV. This suggested that LLMs may have prioritized immediate feasibility alone, failing to capture this therapeutic reasoning. These findings do not imply that LLMs cannot support OT education. Instead, they demonstrate the importance of using standardized assessments and enhancing visual materials with interpretive aids such as labels or arrows, and demonstrate how analyzing LLM errors can inform educators on teaching clinical reasoning, ultimately benefiting both AI performance and human learning. In addition, the limited amount of OT-specific literature in global databases may contribute to these knowledge gaps. Since academic publications can be one source of LLM training data, it has been reported that the accuracy of LLMs in a medical specialty is related to the number of publications in that specialty [14, 28]. Research productivity remains a persistent challenge in occupational therapy, requiring institutional support and mentoring to enhance faculty research outcomes [29]. This limited corpus of OT-specific literature may restrict LLMs’ training data, potentially explaining their poor performance on the JNEOT.

Structural constraints of AI and educational implications

Beyond gaps in training data, current AI systems face structural constraints in performing visual reasoning within the context of OT. Multimodal LLMs can classify static images but struggle to interpret three-dimensional joint alignments, scapular movements, trunk compensations, and subtle postural changes, which require multi-angle observation and hands-on assessment. Visual reasoning in OT involves integrating biomechanics, temporal changes, and nonverbal cues such as movement quality, effort, pain expressions, and fear-avoidance behaviors. These skills represent layered clinical judgment rather than simple pattern recognition, encompassing not only “what is happening” but also why a movement occurs, whether compensatory strategies are adaptive, and how task demands should be modified safely. Recognizing these distinctions clarifies the scope and boundaries of LLM applications in OT education. Consequently, educators should integrate AI tools thoughtfully, using them to supplement rather than replace experiential learning, and institutions should establish clear guidelines emphasizing transparency, accuracy verification, and patient privacy protection. From an ethical standpoint, using local AI models can reduce data-security risks in educational and clinical settings. However, their performance currently lags behind cloud-based systems, suggesting the trade-off between data protection and computational capability that educators must carefully manage.

Limitations, and future directions

This study has several limitations. As the analysis is limited to a single year of JNEOT, caution must be exercised when generalizing the results. However, this limitation was necessary to ensure the consistency of exam content and avoid bias from publicly available answer keys. This is because the criteria and patterns of national examinations change from year to year, and answers and explanations from previous examinations are likely to be publicly available. Another limitation of this study is that the language of evaluation is limited to Japanese; given that most LLMs are trained primarily on English-speaking data, examining changes in performance when problems are translated into English is essential to evaluating the true capabilities of these models. Future comparative evaluations in multiple languages are required. Furthermore, results for categories with small sample sizes (e.g., Rehabilitation Medicine, n = 1) should be interpreted cautiously. Future research should test multiple examination years, languages, and models trained with richer OT-specific data. Our findings also suggest that high-stakes assessments like the JNEOT may need to evolve by increasing the proportion of visual interpretation and clinical reasoning questions to align assessment with the skills most needed in an AI-integrated practice. Despite these limitations, this study provides practical guidance for OT educators navigating AI integration, demonstrating that while LLMs can enhance foundational learning, the development of visual-spatial interpretation and embodied clinical judgment must remain central to human-guided education. ,

Conclusion

LLM performance on the JNEOT shows that while these models excel at text-based questions and can support foundational knowledge acquisition, they perform poorly on visual-based items requiring clinical reasoning. Educators should therefore use LLMs as supplements for theoretical learning while preserving hands-on training for practical skills. Future research should evaluate multiple exam years, languages, and models trained with richer OT-specific data before LLMs are routinely used in education.

Abbreviations

AI

Artificial Intelligence

LLMs

Large Language Models

MCQs

Multiple-Choice Questions

OT

Occupational Therapy

WFOT

World Federation of Occupational Therapists

JNEOT

Japanese National Examination for Occupational Therapists

MHLW

Ministry of Health, Labour and Welfare of Japan

Authors’ contributions

Conceptualization: Kazuya Saita, Yuichi Mine; Methodology: Kazuya Saita, Yuichi Mine; Formal analysis and investigation: Kazuya Saita, Yuichi Mine, Satoru Amano; Writing - original draft preparation: Kazuya Saita, Yuichi Mine; Writing - review and editing: Kazuya Saita, Yuichi Mine, Satoru Amano; Resources: Yuichi Mine; Supervision: Yuichi Mine, Satoru Amano.

Funding

This work was supported by the Ministry of Education, Culture, Sports, Science and Technology of Japan under Grant 25K15961.

Data availability

Study data are available from the corresponding author upon reasonable request.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.