Comparing AI chatbot simulation and peer role-play for OSCE preparation: a pilot randomized controlled trial

Abstract

Background

Artificial intelligence (AI) is increasingly applied in medical education, but its role in fostering interactive clinical competencies remains underexplored. This pilot study aimed to compare the feasibility and educational impact of an AI chatbot–based simulation with traditional peer role-play (PRP) for Objective Structured Clinical Examination (OSCE) preparation, and to share practical lessons from implementing a novel AI tool in a trial setting.

Methods

Nineteen final-year Korean medicine students were randomly assigned to either an AI chatbot group (n = 9) or a PRP group (n = 10) after a baseline knowledge test. Both groups underwent a 30-minute physical examination practice session, followed by a one-hour clinical interview training session specific to their group. The AI chatbot group practiced with a GPT-4o/Claude 3.5-based chatbot providing scenario-driven responses and automated feedback, while the PRP group practiced in pairs under tutor supervision. All participants then completed two OSCE stations (dizziness and shoulder pain). Performance was assessed using a structured checklist covering four domains: history taking, physical examination, patient education, and physician-patient interaction. Post-study questionnaires evaluated the learning experience.

Results

Although the differences in OSCE scores between the groups did not reach statistical significance, several interesting and complementary trends were observed. For example, the PRP group tended to score higher in history taking (mean 74.4 vs. 66.2 in dizziness scenario; Hedges’ g = -0.68, mean 58.6 vs. 54.5 in shoulder pain scenario; Hedges’ g = -0.21), while the AI chatbot group showed a tendency towards higher scores in patient education (32.5 vs. 22.2 in dizziness scenario Hedges’ g = 0.44, 85.0 vs. 66.7 in shoulder pain scenario; Hedges’ g = 0.99). Survey results reflected these following trends. The PRP group valued the authenticity of the interaction and the exam-like environment. In contrast, the AI chatbot group reported higher satisfaction with the autonomy, opportunity for repetitive practice, and structured feedback.

Conclusion

In this pilot study, AI chatbot–based training and PRP demonstrated complementary strengths for OSCE preparation. While PRP appears effective for developing performance-based procedural and communication skills in a realistic setting, AI chatbots show potential for fostering clinical reasoning in a self-paced, reflective learning environment. These complementary strengths suggest a blended learning model, combining both methods, may be optimal for holistic clinical skills development. Further research is needed to validate these preliminary findings.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12909-025-08308-y.

Background

Demonstrating key clinical competencies such as information gathering, physical examination, clinical decision-making, and patient communication—including history taking and patient education—is essential before graduation. These skills require repeated and consecutive training with appropriate learning methods from the early stages of clinical education, and clear instructional objectives are therefore critical [1, 2]. Peer role-play (PRP) and standardized patient (SP) interview training have long been the primary approaches, but both are limited by heavy human-resource demands and restricted scheduling flexibility [3].

The potential for artificial intelligence (AI) in medical education is substantial, with recent studies demonstrating its effectiveness across a range of complex clinical tasks [4–9]. Advanced chatbots have successfully trained dental students in history-taking [10], improved intern physicians’ clinical reasoning [11], and enhanced self-regulated learning through automated feedback in studies like MedSimAI [12]. This trend is confirmed in the broader educational landscape, where AI has been shown to promote personalized learning and deliberate practice [13–16], and scoping reviews continue to highlight AI’s benefits for simulation and self-directed learning [17]. Despite this momentum, a critical gap persists. Most current AI applications in medicine are still confined to simpler functions like information delivery [18], automated scoring [19], or case presentation [20], rather than facilitating dynamic, conversation-based clinical reasoning. While prior studies highlighted the feasibility of AI in medical interviews and traditional Chinese medicine (TCM) inquiry, they focused on interaction quality rather than learning efficacy [21, 22]. Rigorous randomized controlled trials (RCT) that directly compare the efficacy of these novel AI tutors against traditional, resource-intensive methods like peer role-play are conspicuously scarce. This study was designed to address this specific gap.

We previously developed an AI chatbot as an internally designed educational platform [23, 24]. Scenarios created for Objective Structured Clinical Examination (OSCE) training were converted into chart‑style prompts optimized for SP role simulation, and GPT-4o was combined with Claude 3.5 Sonnet in a multi-layered architecture. The system not only generates natural responses to student inquiries but also provides both quantitative scoring and individualized text feedback based on the dialogue. To rigorously evaluate the educational impact of this newly developed system, a definitive trial would be the ultimate goal. However, before conducting a large-scale, definitive trial, a pilot study is an essential first step to evaluate feasibility, identify potential challenges, and generate preliminary data on educational effects. Therefore, this pilot RCT was designed with two primary objectives: First, to share the practical experiences and lessons learned from conducting an RCT with a novel AI educational tool, and second, to conduct a preliminary comparison of the educational effects of AI chatbot-based simulation and traditional PRP.

Methods

Study design

This study was conducted as an exploratory, randomized controlled pilot trial. Ethics approval and consent to participate The study was conducted in accordance with the ethical principles of the Declaration of Helsinki [25], and protocol was approved by Institutional Review Board (IRB) of Gil Korean Medicine Hospital, Gachon University (Approval No. GIRB-25–101). All participants were provided with a detailed explanation of the study objectives and procedures, and written informed consent to participate was obtained from all participants prior to their enrollment in the study.

Participants, recruitment and randomization

Participants were recruited through announcements posted on online communities for Korean Medicine doctors and students, and through official documents sent by Gachon University faculty to 11 other Schools of Korean Medicine nationwide, leading faculty at all 12 Korean medicine schools to promote the recruitment. The recruitment period was from January 24 to February 14, 2025.

A total of 26 volunteers, all of whom were students about to enter their fourth (final) clinical year, were initially recruited. All 26 volunteers were provided with pre-study materials via email and given approximately one week for self-directed learning, including schemata for dizziness and shoulder pain, differential diagnosis tables, structured guides for Korean medicine pattern identification and prescriptions, and physical examination guidelines. They were also informed in advance that final eligibility to participate would be determined based on a baseline knowledge test covering the pre-study materials.

Of the 26 initial volunteers, 19 attended the study session on the designated day. Upon arrival, these 19 participants were formally assessed for eligibility against the following criteria.

Inclusion criteria were: (1) students enrolled in (or on a leave of absence from) a School of Korean Medicine in 2025; (2) who had successfully completed the third year of the core curriculum, which covers the diagnosis of medical diseases and Korean Medicine pattern identification; (3) who passed a baseline knowledge test on the two clinical scenarios provided in the pre-study materials; and (4) who provided written informed consent after a clear explanation of the study.

Exclusion criteria were: (1) failure to pass the baseline knowledge test (a score below 60); (2) presence of physical or mental conditions that would impede the proper execution of the medical interview or diagnostic procedures; or (3) refusal to provide informed consent.

All 19 attending participants passed the baseline knowledge test, which was based on the content of the pre-study materials, and met all eligibility criteria. They were subsequently enrolled and, to ensure balanced baseline knowledge between groups, were stratified based on their test scores. Following stratification, participants were randomly assigned to either the AI chatbot group or the PRP group. The complete process of participant enrollment and allocation is summarized in the study flowchart (Fig. 1).

Fig. 1.

Overview of participant recruitment

Pre-study materials were distributed 1 week prior to the trial; scores from the baseline test were used for stratified block randomization. AI, artificial intelligence.

Sample Size Considerations

A formal priori sample size was not calculated for this pilot study. Considering practical constraints, such as the trial being held during a university vacation period, the geographical dispersion of the 13 Korean Medicine colleges nationwide, and the trial site’s limited accessibility, we set a pragmatic target of 30 participants, a number sufficient for securing descriptive statistics [26]. Initially, 26 students volunteered; however, due to last-minute schedule changes immediately before the new semester, only 19 were able to attend on the day of the trial. Despite this smaller-than-anticipated sample size, we decided to proceed with the study given its nature as an exploratory pilot trial focused on feasibility and generating preliminary data.

Training Methods for Each Group

Visualized information regarding the training methods and assessment for both the control and experimental groups can be found in the flowchart (Fig. 2).

Fig. 2.

Training and assessment process of the finally recruited participants

OSCE, objective and structured clinical examination.

Common education for physical examination of OSCE

Both groups first underwent identical 30-minute physical examination training. To practice physical examinations for dizziness and shoulder pain, a basic diagnostic tool set including a tuning fork, otoscope, alcohol swabs, and a tray, along with a bed, were prepared in advance. Participants, based on the pre-study materials, paired up one-on-one and alternately practiced physical examinations autonomously, without any educational intervention from the research team during this time. In the AI chatbot group, one participant without a partner was paired with a senior student assistant (final‑year Korean medicine student) who supported the session as a practice partner.

AI chatbot group training

Participants interacted with a proprietary AI chatbot featuring a multi-layered architecture that combines GPT-4o and Claude 3.5 Sonnet [23, 24]. For one hour, they performed simulated interviews autonomously under the supervision of a clinical expert. This chatbot acted as a patient, generating responses to learners’ inquiries based on pre-designed, chart-style prompts. After each interview, students could submit the session for automated scoring and receive large language model (LLM)-generated text feedback highlighting their strengths and areas for improvement. Students practiced two clinical presentations (dizziness and shoulder pain), with two cases per presentation, and were allowed to practice each case freely without restrictions on time or number of attempts.

PRP group training

Participants formed pairs and alternately performed physician and patient roles based on the same cases as the AI chatbot group. Unlike the chatbot group, practice time was strictly structured: each case was allotted 12 min for the interview and 2 min for feedback, with feedback provided as peer feedback between students. Each student undertook the role of physician and patient once per clinical presentation, completing a total of four practice sessions A clinical expert participated as a supervisor in this training; however, there was no active educational intervention, and the tutor’s role was limited to supervising the PRP training according to the time schedule.

Blinding

To minimize assessment bias, key personnel were blinded to group allocation. The professors and SPs who conducted the OSCE evaluations, as well as the data analyst who performed the data analysis, were unaware of the participants’ group assignments. Due to the nature of the educational intervention, blinding of the participants and the research staff responsible for guiding them through the training and assessment process was not feasible.

Assessment Methods

Learning: Simulated medical consulting (OSCE)

After the training, participants underwent an OSCE conducted face-to-face with SPs for the same two clinical presentations (dizziness and shoulder pain). Each case was evaluated based on a pre-developed checklist. With prior approval from the National Institute for Korean Medicine Development (NIKOM), the checklists for the dizziness and shoulder pain scenarios were adapted from an item pool developed by NIKOM in collaboration with a panel of clinical professors from Korean medicine colleges. These materials are intended to provide a standardized resource for clinical skills education.

The checklist consisted of four domains: history taking, physical examination, patient education, and physician-patient interaction (PPI). History taking, physical examination, and patient education were scored by professors from Korean medicine colleges, while PPI was scored by SPs. History taking was scored on a 1 − 0 point scale, physical examination on a 2-1-0 point scale, patient education on a 3-2-1-0 point scale, and PPI on a 1 − 0 point scale. This domain-specific scoring system was adopted to align with the established standards of the Korean National Licensing Examination for medical doctor [27, 28], ensuring contextual validity and rater familiarity with the assessment criteria. Each of these four domains had different full score criteria depending on the scenario, as detailed in Supplementary Table 3 of Supplementary file 1.

Reaction: Student survey

Two questionnaires were used in this study. First, the ‘OSCE Learning-Process Survey’, completed by both the AI chatbot group and the PRP group, was designed to understand the overall experience and perceptions of the learning methods. It included questions categorized into two main domains: ‘satisfaction with the learning method’ (Q1-Q22, Likert scale) and ‘perceived usability of the learning method’ (multiple-choice and free-text, Q23-Q24 Likert scale for recommendation intention) (Supplementary Table 1 of Supplementary file 1, Supplementary file 2).

Second, the ‘Chatbot Usability and Improvement Survey’, administered only to the AI chatbot group, was designed to precisely evaluate the chatbot’s usability. It comprised six subdomains: ‘chatbot response quality’ (Q1-Q6, Likert), ‘auto-scoring system’ (Q7-Q9, Likert), ‘text feedback quality’ (Q10-Q23, Likert), ‘system stability’ (Q24-Q30, Likert), ‘error reporting & improvement requests’ (free-text), and ‘additional comments’ (free-text) (Supplementary Table 2 of Supplementary file 1, Supplementary file 3).

Both questionnaires were developed based on a prior literature review (Calgary–Cambridge Observation Guide) [29] and meetings involving three Korean medicine specialists, one medical education expert, and two AI expert. And both questionnaires were refined based on feedback from preliminary pilot testing (IRB No. GIRB-25–102).

Statistical Analysis Methods

Baseline characteristics analysis

To compare baseline characteristics between the two groups (AI chatbot vs. PRP), continuous or ordinal variables such as age, OSCE experience, and baseline knowledge test scores were described as mean ± standard deviation (mean ± SD). Given the small sample size, between-group comparisons for these variables were performed using the Mann–Whitney U test. Categorical variables, such as gender, were presented as frequencies and percentages (n, %), and Fisher’s exact test was used to assess significant differences in distribution between the groups. All statistical tests were two-sided, and a p-value < 0.05 was considered statistically significant.

An exploratory sensitivity analysis was additionally performed to examine whether the results were influenced by the observed gender imbalance between groups. In this analysis, gender was included as a covariate to obtain adjusted comparisons. To ensure the robustness of these exploratory findings, we used several distinct statistical methods, including rank-based ANCOVA, robust regression, and permutation tests. Given the small sample size and the post-hoc nature of this analysis, the detailed findings are presented as supplementary material and should be interpreted as exploratory only.

Learning: OSCE scores

Raw scores collected for the OSCE evaluation were converted to percentages (0–100) relative to the maximum score for each domain (supplementary Table 3). Converted percentage scores were described as mean ± SD.

Considering the small sample size of each group in this study (n = 9 vs. 10), a non-parametric approach was consistently applied to all domains, regardless of the normality test results. First, the distribution characteristics of each variable were confirmed at a descriptive statistical level using the Shapiro–Wilk test, and all between-group comparisons were performed using the Mann–Whitney U test. Effect sizes were calculated using Hedges’ g, which is suitable for non-parametric tests with sample size correction. For the four assessment domains performed in each clinical presentation (dizziness, shoulder pain), raw p-values were subjected to multiple comparison correction within the scenario using the Benjamini–Hochberg method, and the adjusted p-values (p-adj) were reported.

Reaction: Student survey

For the survey results, we performed quantitative analyses for the 5-point Likert scale items, complemented by a descriptive summary of the free-text comments. The ‘OSCE Learning-Process Survey,’ answered by both the AI chatbot group and the PRP group, was analyzed with a focus on between-group comparisons. For each item, the mean difference between the two groups was calculated, and a 95% bootstrap confidence interval (CI) was computed, considering the small sample size. Effect sizes were calculated using Hedges’ g which includes a correction for small sample sizes and is therefore preferred over Cohen’s d in this context, and significant differences were identified mainly for items with an absolute value of , which corresponds to a conventional threshold for a medium-to-large effect size [30], and these were visualized with a forest plot. For between-group comparisons, raw p-values were subjected to multiple comparison correction using the Benjamini–Hochberg method, and statistical significance was assessed using the resulting adjusted p-values (p-adj). For multiple-choice or binary-choice items, we focused on visual comparison of response distributions using bar graphs.

Meanwhile, the ‘Chatbot Usability and Improvement Survey’ was a single-group survey answered only by the AI chatbot group. The analysis focused on the difference between the mean level within the single group and a benchmark value. To assess participant satisfaction, a predefined benchmark value of (on a 5‑point Likert scale) was used, corresponding to ‘Agree’ on the 5-point Likert scale and representing a common threshold for positive satisfaction [31]. Based on this benchmark, Hedges’ g was calculated as a single-group effect size. Similarly, considering the small sample size, a 95% bootstrap CI for the mean was calculated, and items with a large effect, specifically those with an absolute value of (a conventional threshold for a medium-to-large effect [30]), were visualized with a forest plot. All tests applied two-sided testing, and the significance level was set at p < 0.05.

Results

Baseline characteristics

A total of 19 students participated in the study and were randomly assigned to either the AI chatbot group (n = 9) or the PRP group (n = 10). Due to stratified randomization based on baseline knowledge test scores, the initial baseline levels between the two groups were similar. In terms of demographic information, no significant difference was observed in age, but a statistically significant difference was found in sex (p = 0.0325, Fisher’s exact test). As this study is an exploratory pilot study, the potential impact of sex imbalance cannot be entirely excluded, but the primary objective is to identify the educational effectiveness of AI-based training methods (Table 1).

Table 1.

Baseline demographic characteristics of participants in the chatbot group and control group

	Chatbot	Control	p-value	Test
Baseline knowledge test: Dizziness	46.7 ± 5.0	47.0 ± 4.8	0.9193	Mann–Whitney
Baseline knowledge test: Shoulder Pain	37.8 ± 10.9	41.0 ± 7.4	0.5475	Mann–Whitney
Age	25.9 ± 2.8	26.7 ± 6.6	0.7718	Mann–Whitney
Sex	Female(5), Male(4)	Female(10), Male(0)	0.0325*	Fisher’s exact
OSCE_experience	0.2 ± 0.4	0.3 ± 0.7	1.0000	Mann–Whitney

p-values < 0.05 were considered statistically significant. Statistically significant values are marked with an asterisk*.

To assess the potential impact of the observed gender imbalance, an exploratory sensitivity analysis was conducted (Supplementary Table 4 of Supplementary file 1). The results, adjusted for gender, were generally consistent with the primary unadjusted findings. Given the small sample size and the post-hoc nature of this analysis, the detailed findings are presented as supplementary material and should be interpreted as exploratory only.

Learning – Simulated medical consultation

Following the training interventions, the primary objective outcome—OSCE performance—was assessed to compare the educational effects of the two methods. Comparison of OSCE scores across the four domains for each clinical presentation, converted to a 0–100 point scale, revealed differing trends between the two groups depending on the scenario type. In the dizziness scenario (Fig. 3), the PRP group generally recorded higher scores. Particularly in history taking (mean 74.4 vs. 66.2) and physical examination (31.5 vs. 18.8), the PRP group’s advantage corresponded to moderate-to-large effect sizes (Hedges’ g = −0.68, −1.11, adjusted p-value = 0.2698, 0.1792). In the PPI domain, the PRP group was slightly superior, but the difference was negligible (Hedges’ g = −0.06, adjusted p = 1.0000). Conversely, a notable trend emerged in the patient education domain, where the AI chatbot group showed a higher mean score (32.5 vs. 22.0, Hedges’ g = 0.44, adjusted p = 0.4967), indicating a small-to-medium effect size that did not reach statistical significance after adjustment.

Fig. 3.

Comparison of OSCE Performance Scores between the AI Chatbot and PRP Groups: Dizziness Scenario A History Taking, B Physical Examination, C Patient Education, and D Physician–Patient Interaction (PPI).

In the shoulder pain scenario (Fig. 4), the complementary strengths of the AI chatbot group were evident. Particularly in patient education (mean 85.0 vs. 66.7, Hedges’ g = 0.99, adjusted p = 0.1535) and PPI (84.7 vs. 78.4, Hedges’ g = 0.69, adjusted p = 0.1535), the AI chatbot group recorded higher scores, corresponding to large and moderate effect sizes, respectively. On the other hand, no significant difference was observed between the two groups in history taking and physical examination, and the physical examination score was even slightly higher for the AI chatbot group (Hedges’ g = −0.21, 0.15, adjusted p = 7283). This overall pattern of complementary strengths, while not reaching statistical significance, was consistent across both scenarios and supported by notable effect sizes.

Fig. 4.

Comparison of OSCE Performance Scores between the AI Chatbot and PRP Groups: Shoulder Pain Scenario A History Taking, B Physical Examination, C Patient Education, and D Physician–Patient Interaction (PPI).

Reaction – Student Survey

The complementary trends observed in the objective OSCE scores were closely mirrored by the students’ subjective learning experiences. To understand the perceptions underlying these performance trends, we analyzed the results from the post-training surveys, as detailed below.

OSCE learning-process survey

OSCE Learning-Process Survey was completed by both the AI chatbot group and the PRP group, and the analysis focused on between-group comparisons.

Likert scale items

No item reached statistical significance after multiple-comparisons adjustment (adjusted p ≥ 0.05); accordingly, Fig. 5 omits p-values, orders items by Hedges’ g in descending order, and marks a |g| = 0.5 cutoff to help readers identify items exceeding that threshold.

Fig. 5.

Forest Plot of Between-Group Mean Differences in Survey Responses (AI Chatbot vs. PRP Groups)

The AI chatbot group tended to rate higher on ‘Q5. The practice method felt free from time and environmental restrictions (Mean Diff = 0.73, Hedges’ g = 0.64) and ‘Q6. It was easy to self-regulate the learning pace’ (Mean Diff = 0.80, Hedges’ g = 0.60).

Conversely, items where the PRP group scored higher than the AI chatbot group were ‘Q3. It helped improve communication skills with patients’ (Mean Diff=−1.13, Hedges’ g=−1.06), ‘Q17. My ability to handle various unexpected situations improved’ (Mean Diff=−1.18, Hedges’ g=−0.80), ‘Q19. I felt appropriate tension and responsibility during the medical interview practice’ (Mean Diff=−0.79, Hedges’ g=−0.78), ‘Q11. I could systematically practice the OSCE examination process’ (Mean Diff=−0.48, Hedges’ g=−0.68), ‘Q12. I gained a clear understanding of the OSCE examination process’ (Mean Diff=−0.34, Hedges’ g=−0.60), ‘Q24. I would recommend this learning method to other students’ (Mean Diff=−0.27, Hedges’ g=−0.50), and ‘Q23. I would like to continue using this method for future OSCE preparation’ (Mean Diff=−0.36, Hedges’ g=−0.58) (Fig. 5, Table 2).

Table 2.

Survey items with notable differences between chatbot and Role-play groups (|Hedges’ g| ≥ 0.5)

item	Mean Chatbot	Mean Control	CI Low	CI High	Hedge’s g	Adjusted p
Q5_No_Restrictions	4.33	3.60	−0.18	1.64	0.64	0.53
Q6_Self_Paced	4.00	3.20	−0.31	1.84	0.60	0.53
Q23_Future_Use	4.33	4.60	−0.69	0.16	−0.50	0.53
Q24_Recommendation	4.44	4.80	−0.90	0.09	−0.58	0.53
Q12_Clear_Understanding	4.56	4.90	−0.89	0.09	−0.60	0.53
Q11_ Systematic_Practicing	4.22	4.70	−1.03	0.08	−0.68	0.53
Q19_Felt_Responsible	4.11	4.90	−1.78	−0.02	−0.78	0.53
Q17_Handling_Situations	3.22	4.40	−2.44	0.04	−0.80	0.45
Q3_ Improved_Communication	3.67	4.80	−2.02	−0.29	−1.06	0.21

CI_low: Lower bound of 95% confidence interval for mean difference. CI_high: Upper bound of 95% confidence interval for mean difference. p_adj: p-value adjusted using Benjamini-Hochberg method. Rows above the double horizontal rule list items with Hedges’ g ≥ + 0.5 (moderate positive effect), whereas rows below the rule list items with Hedges’ g ≤ − 0.5 (moderate negative effect).

Taken together, these exploratory patterns suggest that chatbot-based training may be advantageous for reducing spatio-temporal constraints and enabling self-paced learning, whereas role-play may better support communication, handling unexpected situations, and procedural understanding, possibly by inducing appropriate exam-like tension. However, given the pilot sample and non-significant p-values, these differences should be interpreted cautiously.

Items are sorted in descending order by effect size (Hedges’ g). Blue bars represent items where the Chatbot group scored higher than the PRP group. Horizontal lines represent 95% confidence intervals for the mean differences. The upper dashed line marks + 0.5 and the lower dashed line − 0.5, the conventional cut-offs for a moderate effect size (|Hedges’ g| ≥ 0.5). AI, artificial intelligence, PRP, peer role-play. OSCE, objective and structured clinical examination.

Multiple-choice items

These quantitative patterns from the Likert scales were further illuminated by the students’ selections in the multiple-choice items. Both groups showed similar patterns in areas where they experienced difficulty during training (Fig. 6A). Both the AI chatbot group and the PRP group reported a high proportion of difficulty in ‘Physical examination’ and ‘Explain & plan’, indicating these are areas that commonly require challenges in both training methods.

Fig. 6.

Distribution of survey responses on perceived usability of the learning method by group A Difficult Areas: Distribution of responses regarding which clinical areas participants found most challenging during their training. B Improved Skills: Distribution of responses regarding which clinical competencies participants felt were most improved through their respective training methods (chatbot vs. roleplay). Results are presented as cumulative percentages with actual response counts in parentheses. Multiple-choice responses allowed participants to select multiple competencies/areas.

Meanwhile, a notable descriptive difference appeared in the competencies perceived as improved (Fig. 6B). The AI chatbot group showed a relatively higher perception of improvement in ‘Structure the interview,’ while the PRP group primarily reported improvements in areas such as ‘Build collaborative rapport,’ ‘Stress management,’ and ‘Handle unexpected situations’. This indicates that the role-play learning method, based on actual human interaction, is more effective in enhancing interpersonal skills and the ability to respond to unforeseen situations, suggesting that chatbot-based training may have limitations in these aspects.

A distinct pattern was also observed between the two groups in the clinical competencies perceived as most helpful (Fig. 7A). For the AI chatbot group, ‘Patient education: plan’ was identified as the most helpful item, whereas for the PRP group, ‘Physician-patient relationship’ had a high response rate. The chatbot’s specific text feedback and quantitative scoring may have contributed to perceived helpfulness in the patient-education domain, potentially touching elements of clinical reasoning. Consistent with this, in OSCE score comparisons the AI chatbot group showed relatively higher performance in patient education than in other domains; nevertheless, between-group differences were not statistically significant and should be viewed as hypothesis-generating.

Fig. 7.

Distribution of survey responses on perceived usability of the learning method by group A Helpful Clinical Competencies: Distribution of responses regarding which clinical competencies participants perceived as most beneficial for their learning through their respective training methods. B Helpful Symptoms: Distribution of responses regarding which clinical scenarios (symptom presentations) participants found most useful for their OSCE preparation. Results are presented as cumulative percentages with actual response counts in parentheses.

Clear differences were also observed in the clinical scenarios (symptom presentations) perceived as most helpful (Fig. 7B). All participants in the AI chatbot group selected the ‘Dizziness’ scenario, while most participants in the PRP group selected the ‘Shoulder pain’ scenario.

Free-text Items

Finally, participants’ own words in the free-text responses provided rich, qualitative explanations for the patterns observed in the quantitative survey data. Analysis of the free-text responses from the surveys revealed clear advantages and disadvantages for both learning methods. For the PRP group, realism was cited as a major advantage, with comments such as “practice in a real-life situation,” “experience real exam room atmosphere,” and “appropriate tension formation.” Additionally, the benefits of direct interaction were emphasized, including “ability to perform physical examination directly” and “real-time feedback.” Conversely, major disadvantages included “time constraints,” “limitations in the quality of peer feedback,” and “uncertainty about accurate diagnosis.”

For the AI chatbot group, key advantages mentioned were “no spatio-temporal constraints”, “ability to practice alone”, “repeatable learning”, “structuring of the consultation flow”, “offer appropriate feedback with nice quality”. However, limitations identified included “inability to implement physical examination”, “lack of understanding of Korean medicine terminology”, “absence of real interaction with patients”, and “scoring speed delay.” Furthermore, AI chatbot group students suggested specific technical improvements for future usability, such as “improving Korean medicine terminology recognition”, “adding time measurement function” and “implementing realistic pulse and tongue diagnosis simulations”.

In summary, descriptive patterns suggest that role-play may better approximate a realistic clinical setting and elicit exam-like tension, whereas the chatbot may offer greater accessibility, repeatability, and higher-quality, targeted feedback. Taken together, the free-text findings outline perceived strengths and limitations and point to potential improvements, including the option of a hybrid format to be examined in future work.

Chatbot usability and improvement survey: within-group analysis

Analysis of the chatbot usability survey results showed that the chatbot received excellent ratings compared to the benchmark value () for the following items: ‘Q1. The chatbot accurately reflected the patient information given in the scenario’ (Mean=4.78, Hedges’ g=1.61, adjusted p = 0.02), ‘Q18. I could apply the content of the feedback’ (Mean=4.67, Hedges’ g = 1.22, adjusted p = 0.06), ‘Q10. The provided feedback accurately quoted my conversation’ (Mean = 4.44, Hedges’ g=0.77), ‘Q16. The length of the feedback was appropriate’ (Mean=4.56, Hedges’ g = 0.70), and ‘Q17. The content of the feedback was easy to understand’ (Mean = 4.56, Hedges’ g = 0.70).

Conversely, items identified as needing improvement were ‘Q25. Scoring speed: The chatbot scored at an appropriate speed’ (Mean = 3.11, Hedges’ g=−0.77) and ‘Q6. The chatbot appropriately understood traditional Korean medicine (TKM) terminology’ (Mean = 3.00, Hedges’ g=−0.82) (Fig. 8, Table 3).

Fig. 8.

Usability Survey Item Means for AI Chatbot Group

Table 3.

Chatbot usability survey items showing notable differences from benchmark value (µ₀ = 4, |Hedges’ g| ≥ 0.5)

item	Mean	CI Low	CI High	Hedge’s g	Adjusted p
Q1_Scenario_Accuracy	4.78	4.44	5.00	1.61	0.02
Q18_Feedback_Applicability	4.67	4.33	5.00	1.22	0.06
Q10_Feedback_Accuracy	4.44	4.11	4.78	0.77	0.22
Q16_Feedback_Length	4.56	4.11	5.00	0.70	0.23
Q17_Feedback_Clarity	4.56	4.11	5.00	0.70	0.23
Q25_Scoring_Speed	3.11	2.44	3.78	−0.77	0.22
Q6_TKM_Terminology_ Understanding	3.00	2.33	3.67	−0.82	0.22

CI_low Lower bound of 95% confidence interval for mean difference, CI_high Upper bound of 95% confidence interval for mean difference. Adjusted p: p-value adjusted using Benjamini-Hochberg method. Rows above the double horizontal rule list items with Hedges’ g ≥ + 0.5 (moderate positive effect), whereas rows below the rule list items with Hedges’ g ≤ − 0.5 (moderate negative effect). TKM, traditional Korean medicine.

These results suggest that while the AI chatbot is excellent at appropriately implementing clinical scenarios and providing useful feedback, improvements are needed in its utilization of specialized TKM terminology and the speed of the automated scoring system.

Items are sorted in descending order by mean scores. Blue dots represent the mean scores for each item on a 1–5 Likert scale, with horizontal lines indicating 95% confidence intervals. The vertical dotted line marks the benchmark (µ₀ = 4). The upper dashed line corresponds to + 0.5 and the lower dashed line to − 0.5 (Hedges’ g), the conventional thresholds for a moderate effect size relative to the benchmark. OSCE, objective and structured clinical examination. TKM, traditional Korean medicine.

Discussion

This pilot study was designed to compare the feasibility and educational impact of AI chatbot-based training and traditional PRP in OSCE preparation, as a preliminary step to inform future research directions. Our findings indicate that while both training methods resulted in comparable overall OSCE performance with no statistically significant differences, noteworthy complementary trends emerged. Objectively, the PRP group showed a tendency towards proficiency in procedural skills such as history taking, whereas the AI chatbot group showed potential in domains requiring clinical reasoning, such as patient education. This pattern was consistently reflected in the students’ subjective experiences. The PRP group reported higher satisfaction with the perceived authenticity of the interaction and the exam-like environment, while the AI chatbot group valued the autonomy, the opportunity for repetitive practice, and the structured feedback.

Prior to comparative analysis of the educational effects, it is valuable to share several lessons learned from the practical execution of this pilot trial. Firstly, we reaffirmed the critical importance of rigorous randomization. Despite stratification by baseline knowledge, score differences in the physical examination, the only domain where both groups received identical training, suggested that our method did not fully mitigate underlying baseline variations. This underscores the need for more stringent randomization protocols in future educational trials. Secondly, the study highlighted the difficulty of standardizing human-dependent variables. For example, even with a structured checklist, ensuring consistent performances and scoring from SPs and raters is a significant challenge, as subtle variations in interaction and judgment are inevitable. This operational difficulty, compounded by the psychometric limitations of the instruments themselves as noted earlier, reinforces the need for robust rater training and validated checklists in future studies. Finally, implementing the AI chatbot revealed that technical stability is as crucial as the model’s response-generation capabilities. Ensuring reliable server performance and having technical support on standby were critical operational challenges, highlighting that the successful integration of AI in education depends on a robust technical infrastructure.

While statistical significance was unfortunately not achieved in most domains, examining the effect sizes and qualitative feedback reveals the complementary strengths of the two approaches. Our pilot data suggest that a key advantage of PRP appears to lie in the development of ‘embodied skills’ through realistic interaction. The tendency for the PRP group to score higher in history taking and physical examination appears consistent with their higher satisfaction ratings for experiencing exam-like tension (Q19 of OSCE Learning-Process Survey), improving communication skills (Q3 of OSCE Learning-Process Survey), and handling unexpected situations (Q17 of OSCE Learning-Process Survey). This suggests that PRP facilitates the physical embodiment of complex clinical tasks, including time management, non-verbal communication, and immediate responses to unpredictable patient reactions. It may be particularly effective in training ‘performance’ itself—translating what is known into what is actually done in a real-time, interactive context [32]. Conversely, the trends observed indicate a potential for the AI chatbot to enhance cognitive skills, particularly ‘clinical reasoning’, by providing a safe, reflective space. The chatbot group’s trend towards higher scores in patient education may be explained by their satisfaction with the autonomy afforded by the absence of time and space constraints (Q5, Q6 of OSCE Learning-Process Survey) and the provision of structured feedback(Q18, Q10, Q16, Q17 of Chatbot Usability Survey). Clinical reasoning is a core higher-order cognitive skill that involves synthesizing scattered patient information to formulate a diagnosis and a logical management plan [33]. The chatbot can be conceptualized as a ‘cognitive playground,’ a low-stakes environment where students can freely engage in deliberate practice [34] by repeatedly formulating hypotheses and learning from mistakes without fear of judgment. Our findings extend the work of recent studies, such as MedSimAI [12], which highlighted AI’s capacity for simulation and generating formative feedback, and research demonstrating ChatGPT-4o’s effectiveness in improving intern physicians’ clinical case management [11]. While these studies showcased the potential of AI, our study contributes preliminary evidence from a direct comparative trial against a traditional method (PRP), suggesting that conversational AI can be a valuable tool for training the complex cognitive processes inherent in clinical reasoning and patient-centered communication [35–37].

The outcomes of this pilot study also establish a clear foundation for future research and development. User feedback provided specific directives for improving the AI chatbot. Technically, priorities include fine-tuning the language model to enhance its understanding of specialized TKM terminology and resolving scoring speed latency. Educationally, realism could be enhanced by incorporating features like a timer or by implementing more immersive technologies such as 3D avatars or voice-based conversations using pre-recorded instructor audio [38, 39]. Based on the distinct advantages of each method identified in this study, we propose two enhanced educational models for future implementation. First, a stepwise, blended learning model could be an ideal strategy, where students initially train their clinical reasoning with the chatbot and subsequently practice procedural and communication skills through PRP or with SPs [32]. Second, to maximize learning outcomes, it is essential to integrate a structured debriefing session with peers or tutors following the chatbot training [40]. Such a session would bridge the gap between the AI’s strength in developing clinical reasoning and the human interaction needed to refine communication and professionalism.

In line with the objectives of a pilot study, we have detailed the following limitations to ensure a transparent interpretation of our findings and to provide a clear foundation for future definitive trials. First, the study’s primary limitation is its small sample size, which has several significant consequences. It limited the statistical power to detect significant differences, meaning our findings should be considered preliminary and hypothesis-generating. It also contributed to a gender imbalance between groups despite randomization. Although an exploratory sensitivity analysis adjusting for this imbalance suggested our directional trends were robust, this issue strongly reinforces the need for a definitive trial with a larger, more balanced sample using appropriate stratification. Finally, the small sample size precluded the formal psychometric validation of our outcome measures. For instance, the satisfaction survey was developed by an expert panel but could not be formally validated. Similarly, while the OSCE checklists were based on established development strategies [41], their credibility and reliability could not be formally analyzed. Rigorous validation of both instruments is therefore a critical step for a future, adequately powered study. Second, the study’s generalizability is limited. Recruitment was geographically concentrated, and only two clinical scenarios were used, though we intentionally selected them from distinct domains (musculoskeletal and internal medicine) to broaden the scope. While the scenarios were specific to Korean medicine, they were built on a standard medical schema, suggesting our core findings on learning modalities could be widely applicable. These constraints are consistent with the exploratory goals of a pilot study designed to test feasibility and guide a future definitive trial. Also, the short training duration is another key limitation, as a single one-hour session may be insufficient to produce lasting educational changes. This minimal duration was intentionally set for this pilot study, partly because participants were senior students presumed to have foundational knowledge, and partly to control for confounding variables in a short-term intervention. Nevertheless, our conclusions are accordingly limited to the immediate effects of the training. Accordingly, such a definitive trial should be conducted over a more sustained training period and evaluate the long-term retention of skills and their transfer to authentic clinical settings.

Despite these limitations, this study provides a valuable contribution as one of the first randomized controlled trials to directly compare a sophisticated LLM-based chatbot with traditional PRP for OSCE preparation in Korean medicine. Our findings offer preliminary evidence for the complementary strengths of the two modalities and highlight the potential of a blended learning model. Based on the lessons learned from this pilot trial, we propose that a future definitive RCT should be conducted with the following enhancements: (1) a larger, multi-center sample to ensure statistical power and generalizability; (2) a longitudinal design to measure long-term skill retention and transfer; and (3) the use of rigorously validated instruments for both performance assessment and satisfaction surveys. Such a study would be well-positioned to definitively test the hypothesis that a blended model, leveraging AI for cognitive skills and PRP for interactive performance, is the optimal strategy for holistic clinical skills development.

Conclusions

This pilot randomized controlled trial suggests that AI chatbot–based training and PRP offer complementary benefits for OSCE preparation. The AI chatbot appeared effective for developing clinical reasoning in a self-paced, repeatable environment, while PRP was superior for practicing human-centered communication and procedural skills. These findings suggest a blended learning model that leverages the unique strengths of both AI and human interaction. This study provides a foundational framework and preliminary evidence for the strategic integration of conversational AI into medical education curricula to create more efficient, scalable, and comprehensive training programs.

Abbreviations

AI

Artificial Intelligence

CI

Confidence Interval

IRB

Institutional Review Board

LLM

Large Language Model

NIKOM

National Institute for Korean Medicine Development

OSCE

Objective Structured Clinical Examination

PPI

Physician–Patient Interaction

PRP

Peer Role–Play

RCT

Randomized Controlled Trial

SP

Standardized Patient

SD

Standard Deviation

TKM

Traditional Korean Medicine

TCM

Traditional Chinese Medicine

Authors’ contributions

HY Lee contributed to the study design, drafted the initial manuscript, and participated in overall writing. J Kim contributed to the study design, co-designed and developed the AI chatbot, conducted data visualization and participated in overall writing. A Jeong and H Choi acted as on-site investigators and conducted the clinical trial involving human participants. A Jeong and S Choi supervised the scientific and ethical conduct of all human-participant research process. H Bae performed the statistical analyses. CE Kim and JH Kim conceived the research question, contributed to the study design, and oversaw the overall execution of the study.

Funding

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (RS-2024-00339889). And this work was supported by National Research Foundation of Korea (NRF) grants funded by the Korean government (MSIT) (no. 2022R1I1A2066653).

Data availability

The datasets generated and analysed during the current study are not publicly available due to concerns related to participant confidentiality, but are available from the corresponding author on reasonable request.

Declarations

Ethics approval and consent to participate

The study was conducted in accordance with the ethical principles of the Declaration of Helsinki, and the protocol was approved by Institutional Review Board (IRB) of Gil Korean Medicine Hospital, Gachon University (Approval No. GIRB-25-101). All participants were provided with a detailed explanation of the study objectives and procedures, and written informed consent to participate was obtained from all participants prior to their enrollment in the study.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.