Evaluating the role of ChatGPT in patient questions regarding drug allergies

Abstract

Background

The use of artificial intelligence (AI) technology is increasing. It takes patients with drug allergies longer to get an appointment at allergy clinics. It remains unclear whether AI technology will be able to help patients with drug allergies. In this study, we aimed to evaluate ChatGPT-5's responses to frequently asked questions about drug allergies in terms of accuracy, currency, comprehensiveness, usefulness, and clarity of language.

Methods

Frequently asked real patient questions about drug allergies were reviewed by experienced allergy specialists, and the 10 most common and clinically relevant questions were submitted to ChatGPT-5. Forty independent allergy-immunology specialists reviewed these answers using a standardized rubric that evaluates accuracy, currency, comprehensiveness, usefulness, and clarity of language, each rated on a 1–5 Likert scale (1 = very poor, 5 = excellent).

Results

No questions received a score of 1 on any item across all domains (accuracy, currency, comprehensiveness, usefulness, and understandability). The lowest overall index score was for accuracy (median 3.9, Q1–Q3: 3.6–4.3), while the highest was for understandability (median 4.4, Q1–Q3: 3.7–4.9). Overall index scores were compared across domains, and statistically significant differences were found between accuracy and the other domains (p < 0.05), with accuracy showing relatively lower median values compared to currency, comprehensiveness, usefulness, and understandability. The overall index percentages of responses scoring ≥4 were 71.5% for accuracy, 74.8% for currency, 77.8% for comprehensiveness, 76.5% for usefulness, and 79.0% for understandability.

Conclusion

ChatGPT-5 demonstrated high accuracy, currency, comprehensiveness, usefulness, and understandability. Given the higher prevalence of drug allergies and the growing patient demand for allergy immunology specialists, the support of AI models for providing information to patients about drug allergies in this regard is essential.

Introduction

Drug allergy is a spectrum of immunologically mediated hypersensitivity reactions with different clinical presentations. These types of adverse drug reactions not only negatively affect the patient's quality of life but can also lead to delayed treatment, unnecessary tests, and even death.¹

Adverse drug reactions affect 10–20% of hospitalized patients and more than 7% of the general population, and hypersensitivity reactions account for approximately one-third of all adverse drug reactions.² In a recent review, 8.3% of 126,306 total participants reported self-reported drug allergy, and 68.2% of them reported skin symptoms, and 10.8% reported anaphylactic or systemic reactions.³ Chiang et al found that the absolute prevalence and incidence of documented drug allergy were 5.61% and 277/100,000 population, respectively.⁴ Drug allergies also play an essential role in emergency room visits. In a study, it was determined that 14.4% of 94,073 patients who applied to the emergency room were admitted due to drug allergy and 0.6% due to anaphylaxis.⁵

In drug allergy, different endotypes cause similar phenotypes. Unfortunately, most biomarkers have not advanced beyond analytic or clinical validity, so evaluating a drug allergy patient in a clinic takes more time.⁶ Due to the scarcity of allergy specialists and the increasing prevalence of drug allergies, it often takes a long time for patients to get appointments with allergy specialists so that drug allergy poses a significant burden to patients and healthcare systems. Providing patients with accurate information can help them to decrease their anxiety, reduce misuse of drugs before appointments, understand more easily during patient-doctor consultations, and increase the effectiveness of the consultation. In this context, large language models (LLMs) such as ChatGPT are increasingly being explored as potential tools for providing accessible health-related information to patients. The use of artificial intelligence (AI) technology in the field of allergy is becoming increasingly widespread.

Although the use of AI technology in allergic diseases has become widespread in recent years, studies specifically evaluating LLM-based tools in drug allergy remain limited.7, 8, 9, 10, 11, 12

In this study, we sought to evaluate the potential of ChatGPT-5 as a tool for managing patients with drug allergy. To this end, we evaluated ChatGPT-5's responses to frequently asked patient questions about drug allergies in terms of accuracy, currency, comprehensiveness, usefulness, and understandability.

Method

Questions directed to ChatGPT-5

Frequently asked questions by patients applying to the immunology and allergy clinic for drug allergies were recorded over 1 month. Two experts with at least 5 years of experience in immunology and allergy identified the 10 most frequently asked and clinically relevant questions based on these patient inquiries and the literature. Based on previous LLM evaluation studies, the number of representative patient questions has been limited to 10.13, 14, 15 During this process, the experts tailored the questions to realistically reflect the language and expressions commonly used by patients. Questions were simulated as if asked by patients and delivered to ChatGPT-5 in a manner typical of patient speech. All questions were asked in Turkish, reflecting the language spoken by the patient population in the clinical context of this study.

ChatGPT-5 (version GPT-5, OpenAI), was accessed through its official platform (https://chat.openai.com) on October 14, 2025. All questions were submitted to ChatGPT-5 using a standardized zero-shot prompt in independent chat sessions to avoid context contamination. We did not give any prior examples or sample answers. We used this model because we wanted the assessment to be consistent and fair across for all questions. We did not provide any examples in advance, so we wanted ChatGPT's answers to reflect general reasoning skills rather than learning from previous cues.

Additionally, to prevent ChatGPT from adjusting its answers based on previous questions in the same session, we asked each question in a new, separate chat session. To ensure this, we cleared the chat history before each question, preventing any potential interactions from previous questions. We asked each question only once and did not impose word limits or other restrictions on responses. Thus, this ensured that each answer was presented freshly and independently from ChatGPT.

We focused on single-turn responses to evaluate ChatGPT-5's initial response quality under standardized conditions. Allowing follow-up dialogue could have improved the answers through user-driven feedback. However, the primary aim of this study was to evaluate the model's initial response quality under standardized conditions. Although real-world patient interactions with LLMs may involve follow-up questions and iterative dialogue, the present study focused on evaluating the model's initial response quality under standardized conditions.

Questions directed to ChatGPT-5 and responses of ChatGPT-5 were in Supplementary Table 1.

Characteristics of evaluators and data collection process

In this study, the responses generated via ChatGPT-5 were evaluated by immunology and allergy experts. The primary objective was to assess the potential of ChatGPT-5, an AI-based platform, to provide accurate and helpful guidance to patients with drug allergies regarding symptoms, diagnosis, diagnostic tests, treatment, and disease management. Responses generated by ChatGPT-5 were evaluated for accuracy, currency, comprehensiveness, usefulness, and understandability. An evaluation form was prepared using questions submitted to ChatGPT-5 and the responses it generated. The rubric method was used for the evaluations.¹⁶

The number of evaluators required for a rubric evaluation generally varies depending on the study's specificity and scope, and there are no strict rules regarding the number of evaluators. However, a minimum of 5 evaluators is recommended for content validity assessment in such studies. Additionally, if a more comprehensive analysis is planned, the number of evaluators on the evaluation panels can be increased to ensure more objective results, while taking into account their qualifications and expertise.^17,18 In this study, 40 immunology and allergy specialists from various institutions were recruited using snowball sampling, and online evaluation forms (Google Forms) were distributed to the evaluators by using email and WhatsApp.

Forty immunology and allergy specialists independently rated each response on a five-point Likert scale (1 = very poor, 5 = excellent) for accuracy, currency, comprehensiveness, usefulness, and understandability.

Evaluation of ChatGPT responses by evaluators

Immunology and allergy specialists evaluated each category using a rubric method, scoring each question on a scale of 1–5. A rubric is a rating scale that systematically evaluates a performance, assignment, presentation, article, or other work based on specific criteria. In other words, it provides a clear guide for the evaluator, indicating "what to grade and at what level."¹⁹ It can also be used in the healthcare field.^20,21 The evaluation domains for this study were determined after a comprehensive review of previous healthcare-related LLM evaluation studies20, 21, 22, 23 and recent methodological literature on human evaluation of LLMs in healthcare, particularly the QUEST framework proposed by Tam et al.¹⁶

Scoring method

To evaluate the quality of ChatGPT-5's responses, a five-point rubric designed explicitly for specialist physicians was employed. The rubric consisted of 5 criteria: accuracy, currency, comprehensiveness, usefulness, and understandability. In this study, all evaluators were informed about each evaluation domain and the predefined scoring system prior to the evaluation process.

Each criterion was rated on a scale from 1 to 5 points.

• •Accuracy assessed the correctness of information, ranging from completely incorrect or missing (1 point) to completely correct (5 points).
• •Currency evaluated whether the response was up-to-date, from not up-to-date (1 point) to completely up-to-date (5 points).
• •Comprehensiveness measured the extent to which the response covered essential aspects, from covering only 1 aspect (1 point) to covering all critical elements (5 points).
• •Usefulness reflected the practical applicability and benefit of the response, from not functional at all (1 point) to beneficial and applicable (5 points).
• •Understandability assessed the clarity of the response, ranging from not understandable (1 point) to completely understandable (5 points).

The four- and five-point Likert responses were interpreted as indicating that evaluators found the responses for each criterion highly satisfactory. An overall index score for each domain was calculated by summing all ratings in that category and averaging them, providing a single composite measure per domain.^20,21

Statistical analysis

All statistical analyses were conducted using IBM SPSS Statistics version 23.0 (IBM Corp., Armonk, NY, USA). Descriptive statistics were summarized as frequency (n) and percentage (%), mean ± standard deviation (SD) for continuous variables, and median (minimum–maximum) and interquartile range (Q1–Q3) for ordinal variables.

Because the Likert-scale ratings used in this study represent ordinal data, nonparametric tests were preferred. To evaluate potential differences in paired overall index scores across evaluation domains, the Wilcoxon Signed-Rank Test was applied. Overall index scores were compared according to age, academic title, duration of ChatGPT use, and self-reported ChatGPT knowledge level using the Mann–Whitney U test. A p-value of <0.05 was considered statistically significant.

For each evaluation domain (accuracy, currency, comprehensiveness, usefulness, and understandability), domain-specific index scores were calculated by averaging all individual Likert ratings given by physicians across the 10 questions. For example, if a physician rated the "accuracy" of 10 ChatGPT responses as x₁, x₂, …, x₁₀ on a 5-point Likert scale, the Accuracy Index Score for that physician was computed as:

$$\text{Accuracy}\text{Index}\text{Score}=\left({\mathrm{x}}_1+{\mathrm{x}}_2+\dots +{\mathrm{x}}_{10}\right)/10\text{.}$$

This calculation was repeated for all 40 physicians, and the median (Q1-Q3) and minimum-maximum of these individual index scores were reported in the tables.

Each response was independently rated by specialist physicians on a 5-point Likert scale, with a score of ≥4 indicating that the evaluator deemed the response highly satisfactory for the given criterion. The “Overall Evaluation Percentage” was calculated for each domains as the average proportion of evaluators assigning scores of 4 or higher across all questions.^20,21

Results

Characteristics of allergy immunology specialists

A total of 40 allergy and immunology specialists participated in the study. Of these, 10 (25.0%) were male and 30 (75.0%) were female. The mean age of participants was 41.0 ± 7.9 years, and the median age was 39.0 years (range: 32.0–74.0). Regarding professional status, 28 (70.0%) were specialists, and 12 (30.0%) held academic titles, including assistant, associate, or full professorships. The vast majority of participants (36, 90.0%) reported prior experience with LLMs such as ChatGPT, Gemini, DeepSeek, or Copilot, whereas 4 (10.0%) had never used such models. Similarly, 36 (90.0%) of the physicians reported using ChatGPT before the study, while 4 (10.0%) had not. Of the 36 evaluators who reported using ChatGPT previously, 31 (86.1%) stated that they used ChatGPT for academic purposes. In terms of self-reported knowledge about ChatGPT, most participants rated their understanding as moderate (47.5%) or a little (35.0%), while 4 (10.0%) described their knowledge as substantial and 1 (2.5%) as excellent; only 2 (5.0%) participants indicated that they did not know the platform. Regarding duration of ChatGPT use, 13 (32.5%) had used the tool for less than 6 months, 11 (27.5%) between 6 months and 1 year, and 12 (30.0%) between 1 and 3 years; only 4 (10.0%) participants had never used ChatGPT before the study (Table 1).

Table 1.

Characteristics of the allergy and immunology specialists who evaluated ChatGPT's responses

Characteristics	N = 40
Gender, n (%)
Male	10 (25.0)
Female	30 (75.0)
Age, year
Mean ± SD	41.0 ± 7.9
Median (min-max)	39.0 (32.0–74.0)
Title, n (%)
Specialist	28 (70.0)
Associate Professor/Professor	12 (30.0)
Use of LLMs (ChatGPT, Gemini, DeepSeek, Copilot vs), n (%)
No	4 (10.0)
Yes	36 (90.0)
Have used ChatGPT, n (%)
No	4 (10.0)
Yes	36 (90.0)
Knowledge about ChatGPT, n (%)
No knowledge	2 (5.0)
A little knowledge	14 (35.0)
Moderate knowledge	19 (47.5)
Substantial knowledge	4 (10.0)
Excellent knowledge	1 (2.5)
Duration of ChatGPT use, n (%)
No use	4 (10.0)
<6 months	13 (32.5)
6 months-1 year	11 (27.5)
1–3 years	12 (30.0)

Evaluation of ChatGPT's answers

Findings related to the detailed expert evaluation of ChatGPT-5's responses to the 10 drug allergy questions (Q) are summarized in Supplementary Table 2. None of the participating physicians assigned a score of 1 to any item across all domains (accuracy, currency, comprehensiveness, usefulness, and understandability). Ratings of 2 were infrequent throughout the dataset. It was observed that 2 points were given to all domains of answers of Q4, 5, 7, and 10, with a range between 2.5% and 22.5%. The questions Q4 (“My family doctor said I might be allergic to paracetamol. Can I take another painkiller?”) and Q5 (“When a different medication is prescribed than the one I am allergic to, how can I be sure that it will not cause allergies?”) received a higher frequency of scores of 2 and 3 across multiple domains compared to the other questions. For Q4, 2-point ratings were observed in 5 domains, ranging between 7.5% and 22.5%, while 3-point ratings ranged from 15% to 35.0%. For Q5, domains included 2-point scores ranging from 2.5% to 7.5%, and 3-point ratings ranged from 20.0% to 40.0%.

For almost all questions, except Q4 and 5, the median rating across all 5 evaluation domains was 4.0. The highest median ratings were observed in Q8 and 9, which addressed documentation of drug allergy and accidental drug exposure. Q8 received median scores of 5.0 for usefulness and understandability, and Q9 received median scores of 5.0 for accuracy and understandability (Table 2). The overall index percentage of responses scoring ≥4 ranged from 71.5% for accuracy to 79.0% for understandability (Fig. 1). The lowest overall index score was for accuracy (median 3.9, Q1–Q3: 3.6–4.3), while the highest was for understandability (median 4.4, Q1–Q3: 3.7–4.9). When overall index scores were compared across domains (Fig. 2), statistically significant differences were found between accuracy and the other domains (p < 0.05), with accuracy showing relatively lower median values compared to currency, comprehensiveness, usefulness, and understandability. The boxplot further illustrates these distributions and shows the significant pairwise differences between accuracy and the other domains (p = 0.043, p = 0.011, p = 0.024, p = 0.005, and p = 0.042), (Fig. 2).

Table 2.

Expert evaluation of ChatGPT responses based on accuracy, currency, comprehensiveness, usefulness, and understandability.

Experts, N = 40	Experts with Likert response of ≥4, n (%)	Median (Q1-Q3) score	Min-max score
Question 1
Accuracy	30 (75.0)	4.0 (3.25–4.75)	3.0–5.0
Currency	33 (82.5)	4.0 (4.0–4.0)	3.0–5.0
Comprehensiveness	30 (75.0)	4.0 (3.25–5.0)	3.0–5.0
Usefulness	29 (72.5)	4.0 (3.0–5.0)	3.0–5.0
Understandability	29 (72.5)	4.0 (3.0–5.0)	2.0–5.0
Question 2
Accuracy	30 (75.0)	4.0 (3.25–4.0)	3.0–5.0
Currency	29 (72.5)	4.0 (3.0–4.0)	3.0–5.0
Comprehensiveness	28 (70.0)	4.0 (3.0–5.0)	3.0–5.0
Usefulness	30 (75.0)	4.0 (3.25–5.0)	3.0–5.0
Understandability	28 (70.0)	4.0 (3.0–5.0)	2.0–5.0
Question 3
Accuracy	32 (80.0)	4.0 (4.0–5.0)	3.0–5.0
Currency	33 (82.5)	4.0 (4.0–5.0)	3.0–5.0
Comprehensiveness	34 (85.0)	4.0 (4.0–5.0)	2.0–5.0
Usefulness	31 (77.5)	4.0 (4.0–5.0)	2.0–5.0
Understandability	30 (75.0)	4.5 (3.25–5.0)	2.0–5.0
Question 4
Accuracy	18 (45.0)	3.0 (3.0–4.0)	2.0–5.0
Currency	20 (50.0)	3.5 (3.0–4.0)	2.0–5.0
Comprehensiveness	21 (52.5)	4.0 (3.0–4.0)	2.0–5.0
Usefulness	21 (52.5)	4.0 (3.0–4.0)	2.0–5.0
Understandability	31 (77.5)	4.0 (4.0–5.0)	2.0–5.0
Question 5
Accuracy	22 (55.0)	3.0 (3.0–4.0)	2.0–5.0
Currency	23 (57.5)	3.0 (3.0–4.75)	2.0–5.0
Comprehensiveness	29 (72.5)	3.0 (3.0–4.75)	2.0–5.0
Usefulness	25 (62.5)	4.0 (3.0–5.0)	2.0–5.0
Understandability	30 (75.0)	4.0 (3.25–5.0)	2.0–5.0
Question 6
Accuracy	32 (80.0)	4.0 (4.0–5.0)	3.0–5.0
Currency	30 (75.0)	4.0 (3.25–5.0)	3.0–5.0
Comprehensiveness	35 (87.5)	4.0 (4.0–5.0)	3.0–5.0
Usefulness	33 (82.5)	4.0 (4.0–5.0)	2.0–5.0
Understandability	34 (85.0)	4.0 (4.0–5.0)	2.0–5.0
Question 7
Accuracy	30 (75.0)	4.0 (3.25–5.0)	2.0–5.0
Currency	33 (82.5)	4.0 (4.0–5.0)	2.0–5.0
Comprehensiveness	35 (87.5)	4.0 (4.0–5.0)	2.0–5.0
Usefulness	34 (85.0)	4.0 (4.0–5.0)	2.0–5.0
Understandability	33 (82.5)	4.0 (4.0–5.0)	2.0–5.0
Question 8
Accuracy	33 (82.5)	4.0 (4.0–5.0)	3.0–5.0
Currency	34 (85.0)	4.0 (4.0–5.0)	2.0–5.0
Comprehensiveness	33 (82.5)	4.0 (4.0–5.0)	2.0–5.0
Usefulness	35 (87.5)	5.0 (4.0–5.0)	2.0–5.0
Understandability	35 (87.5)	5.0 (4.0–5.0)	3.0–5.0
Question 9
Accuracy	32 (80.0)	5.0 (4.0–5.0)	3.0–5.0
Currency	33 (82.5)	4.0 (4.0–5.0)	2.0–5.0
Comprehensiveness	34 (85.0)	4.0 (4.0–5.0)	3.0–5.0
Usefulness	35 (87.5)	4.0 (4.0–5.0)	3.0–5.0
Understandability	34 (85.0)	5.0 (4.0–5.0)	3.0–5.0
Question 10
Accuracy	27 (67.5)	4.0 (3.0–5.0)	2.0–5.0
Currency	31 (77.5)	4.0 (4.0–5.0)	2.0–5.0
Comprehensiveness	32 (80.0)	4.0 (4.0–5.0)	2.0–5.0
Usefulness	33 (82.5)	4.0 (4.0–5.0)	2.0–5.0
Understandability	32 (80.0)	4.0 (4.0–5.0)	2.0–5.0
All questions	Overall index percentage of responses scoring ≥4, %a	Overall index Scoreb	Overall index Scoreb
Accuracy	71.5	3.9 (3.6–4.3)	2.6–5.0
Currency	74.8	4.0 (3.6–4.4)	2.7–5.0
Comprehensiveness	77.8	4.2 (3.6–4.5)	2.8–5.0
Usefulness	76.5	4.1 (3.7–4.5)	2.8–5.0
Understandability	79.0	4.4 (3.7–4.9)	2.3–5.0

^aThe percentage of responses scoring ≥4 across all 10 questions are provided for each of the 5 evaluation areas.

^bThe overall index scores for all questions are derived by aggregating and averaging the scores from all 5 areas across each question

Fig. 1.

Overall index percentage of experts' ratings (≥4) across the 5 evaluation domains

Fig. 2.

Boxplot of overall index scores across 5 evaluation domains.

Foodnote: Pairwise comparisons of the overall index scores were performed using the Wilcoxon Signed Ranks Test. Only p-values <0.05 are shown in the figure. Non-significant comparisons: Currency vs. Comprehensiveness (p = 0.071), Currency vs. Usefulness (p = 0.155), Comprehensiveness vs. Usefulness (p = 0.941), and Usefulness vs. Understandability (p = 0.060)

Overall index scores were compared according to age, academic title, duration of ChatGPT use, and self-reported ChatGPT knowledge level. Evaluators aged ≥40 years demonstrated significantly higher understandability scores compared with evaluators aged <40 years (median: 4.8 vs. 4.0, p = 0.043). No significant differences were observed in the other subgroup comparison analyses (all p values > 0.05), (Table 3).

Table 3.

Comparison of overall index scores according to evaluator characteristics.

Groups	Overall index score		p valuea
	Median (Min-Max)	Median (Min-Max)
	Age <40 years old, n = 22	Age ≥40 years old, n = 18
Accuracy	3.9 (3.2–5.0)	4.0 (2.6–4.8)	0.300
Currency	4.0 (3.1–5.0)	4.2 (2.7–4.9)	0.567
Comprehensiveness	4.1 (3.3–5.0)	4.3 (2.8–4.9)	0.540
Usefulness	3.9 (3.2–4.9)	4.2 (2.8–5.0)	0.436
Understandability	4.0 (2.3–5.0)	4.8 (2.9–5.0)	0.043
Groups	Specialist doctor, n = 28	Associate Professor/Professor, n = 12
Accuracy	3.9 (3.2–5.0)	4.2 (2.6–4.8)	0.197
Currency	4.0 (3.1–5.0)	4.4 (2.7–4.9)	0.193
Comprehensiveness	4.1 (3.3–5.0)	4.5 (2.8–4.9)	0.069
Usefulness	4.0 (3.2–5.0)	4.2 (2.8–5.0)	0.646
Understandability	4.2 (2.3–5.0)	4.7 (2.9–5.0)	0.234
Groups	ChatGPT Usage duration <6 months, n = 17	<6 months, n = 17 ≥ 6 months, n = 23
Accuracy	3.8 (3.5–5.0)	4.0 (2.6–4.8)	0.206
Currency	3.8 (3.5–5.0)	4.2 (2.7–4.9)	0.202
Comprehensiveness	3.9 (3.3–5.0)	4.3 (2.8–4.9)	0.373
Usefulness	3.9 (3.2–4.9)	4.1 (2.8–5.0)	0.426
Understandability	4.0 (2.7–5.0)	4.7 (2.3–5.0)	0.247
Groups	Limited/No ChatGPT knowledge, n = 16	Moderate/Advanced ChatGPT knowledge, n = 24
Accuracy	4.0 (3.5–5.0)	3.9 (2.6–4.8)	0.244
Currency	4.1 (3.5–5.0)	3.9 (2.7–4.9)	0.203
Comprehensiveness	4.3 (3.8–5.0)	3.9 (2.8–4.9)	0.115
Usefulness	4.4 (3.7–5.0)	3.9 (2.8–5.0)	0.056
Understandability	4.7 (2.3–5.0)	4.0 (2.7–5.0)	0.272

^aMann-Whitney U test

Discussion

In this study, we evaluated the potential of ChatGPT-5 to provide information to patients about drug allergies. Firstly, we identified the 10 most frequently asked questions by patients admitted to hospitals with drug allergies. We posed these questions to ChatGPT-5 and sent the responses to 40 allergy and immunology physicians. 40 specialists independently rated each response on a five-point Likert scale (1 = very poor, 5 = excellent) for accuracy, currency, comprehensiveness, usefulness, and understandability.

None of the participating physicians assigned a score of 1 to any item across all domains, indicating that no response was perceived as completely inaccurate or misleading. The majority of responses were rated 4 or 5, indicating strong expert agreement on the overall adequacy of ChatGPT's medical explanations. In total, scores of 4 or higher accounted for more than 70% of all evaluations, reflecting high satisfaction with the accuracy, clarity, and clinical usefulness of the responses.

Questions 4 and 5 are directly related to medication use and clinical decision-making, which inherently involve greater uncertainty and potential patient risk. Experts may have been more critical in assessing ChatGPT's answers for these cases, as they require nuanced interpretation, safety considerations, and context-specific recommendations. For Q4, ratings of 2 were observed ≥3, in all domains, including accuracy, currency, usefulness, comprehensiveness, and understandability, suggesting that some experts found the provided information to be incomplete or lacking up-to-date clinical detail. Similarly, Q5 received 2 points in all domains, indicating that ChatGPT's responses might not have adequately addressed practical concerns about preventing allergic reactions when switching medications. In contrast, the other questions, which focused on definitions, diagnosis, and general knowledge of drug allergies (eg, Q1, Q2, Q3, Q6, and Q7), received predominantly high ratings (4–5 points). This pattern reflects greater confidence in the reliability and clarity of ChatGPT's information in those contexts. Overall, these findings suggest that ChatGPT performs more consistently in providing general educational information than in complex, risk-sensitive clinical scenarios that require individualized medical judgment.

For almost all questions, except questions 4 and 5, the median rating across all 5 evaluation domains (accuracy, currency, comprehensiveness, usefulness, and understandability) was 4.0, reflecting a generally favorable expert assessment of ChatGPT's performance. This consistent pattern across domains indicates that the model can generate responses that specialists perceive as clinically acceptable and understandable, supporting its potential role as an adjunct tool in patient education and clinical decision support. In the context of drug allergies, this performance suggests that ChatGPT may serve as a helpful communication aid for patients seeking reliable information on allergic reactions, diagnosis, testing procedures, and documentation, thereby enhancing patient understanding and making the specialists' jobs easier. Pairwise comparisons between evaluation domains demonstrated that understandability scores were generally higher than several other domains, including accuracy, currency, and comprehensiveness. This finding may suggest that ChatGPT-5 is particularly effective in generating clear and patient-friendly explanations. However, despite these statistical differences, the scores in all domains were generally high and close to each other.

The highest expert consensus was observed for questions 8 and 9, which focused on drug allergy documentation and accidental exposure. Q8 received median scores of 5.0 for usefulness and understandability, and Q9 received median scores of 5.0 for accuracy and understandability. These findings suggest that the model performs remarkably well when addressing practical, clearly defined topics related to patient safety and allergy management, but not when selecting alternative medicines. The experts’ strong agreement indicates that ChatGPT provided concise, accurate, and easily understandable information in situations where standardized clinical recommendations already exist, such as documentation practices and the management of unintended drug exposure.

A study evaluating ChatGPT-3.5's performance on allergy questions in clinical practice, providing a systematic approach for rating errors, and analyzing the potential patient consequences of these errors identified 6 critical errors across 120 multidisciplinary allergy questions: 1 in dermatology, 2 in pediatrics, and 3 in pulmonology. This study also identified 1 critical pediatric food allergen error that could be life-threatening. Also, in this study, it was demonstrated that ChatGPT had good accuracy (mean accuracy 4.1/5, standard deviation: 0.78, range: 1–5) and completeness and perceived humanness were sufficient, but lowest for pediatric questions; in addition it was found that readability was at an academic level for most responses.²⁴

In a study evaluating the quality of ChatGPT-generated information on allergic rhinitis, a Likert scale from 1 to 5 was used to assess response accuracy, and only 4 authors independently rated the responses. Of the 20 questions asked about allergic rhinitis, 8 received 5 points (no inaccuracies), 5 received 4 points (minor, harmless inaccuracies), 6 received 3 points (potentially misinterpretable inaccuracies), and 1 response received 2 points (minor, potentially harmful inaccuracies).²⁵ In our study, 10 frequently asked questions about drug allergy were evaluated by 40 immunology and allergy specialists, and not only accuracy but also currency, comprehensiveness, usefulness, and understandability were evaluated. And in most questions, satisfactory answers of ChatGPT-5 had been detected.

In a publication on allergic rhinitis (AR) and chronic rhinosinusitis (CRS), 189 AR and 242 CRS-related questions were asked to GPT-3.5 and GPT-4, and 2 senior rhinology professors independently evaluated the answers and, in cases of disagreement, by a third rater. In this study, ChatGPT performed satisfactorily overall, answering more than 80% of the questions correctly in all categories. Specifically, GPT-4.0 significantly outperformed GPT-3.5 in accuracy on AR-related questions, but the difference was not clear for CRS-related questions. Response accuracy was not affected when questions were asked in Chinese.⁹ In our study, as in this publication, for almost all questions, except questions 4 and 5, the median rating across all 5 evaluation domains was 4.0, reflecting a generally favorable expert assessment of ChatGPT's performance and in the subgroup analyses based on evaluator characteristics, only evaluators aged ≥40 years demonstrated higher understandability scores, whereas no significant differences were observed according to academic title, duration of ChatGPT use, or self-reported ChatGPT knowledge level. The higher understandability scores among older evaluators may reflect differences in clinical communication perspectives and expectations regarding patient-oriented explanations, as more experienced clinicians may place greater value on simplified and easily understandable responses for patient education purposes.

Twenty-four questions derived from the EAACI/GA2LEN/EuroGuiDerm/APAAACI recommendations were prompted to ChatGPT-4 to obtain responses. The questions were categorized into 3 groups: A) Classification and Diagnosis, B) Assessment and Monitoring, and C) Treatment and Management Recommendations. The DISCERN instrument for quality assessment, the Journal of the American Medical Association (JAMA) benchmark criteria for reliability evaluation, and Flesch scores for readability analysis were used to evaluate the responses, which were separately evaluated by allergy specialists. In the results, categories A and C demonstrated insufficient reliability according to JAMA, with median scores of 1 and 0, respectively. Category B demonstrated a low reliability score (median 2, interquartile range 2), while the information quality of category C questions was satisfactory (median 51.5, interquartile range 12.5). All 3 groups demonstrated poor readability according to Flesch assessment scores.⁷

In a comparative study of ChatGPT's performance in dentistry and allergy-immunology assessments, questions from the Swiss Federal Licensing Examination in Dental Medicine (SFLEDM) and the European Examination in Allergy and Clinical Immunology (EEAACI) were presented to ChatGPT 3 and ChatGPT 4, with and without priming. As a result, the average accuracy rates in the SFLEDM and EEAACI assessments were 63.3% and 79.3%, respectively, and both ChatGPT versions performed better on EEAACI than SFLEDM, with ChatGPT 4 outperforming ChatGPT 3 across all tests.²⁶

In a study on atopic dermatitis, ChatGPT information appeared to align with the currently accepted allergy and immunology guidelines and was supported by reliable references.¹¹

Gerek et al aimed to evaluate the adherence of large language models to established guidelines in the management of hereditary angioedema. Their findings indicated that ChatGPT and Gemini demonstrated relatively higher adherence to these guidelines; however, inconsistencies in citation practices underscore the need for further validation and optimization to improve their reliability in medical applications.⁸

In addition, there are numerous studies in the literature regarding ChatGPT use in different areas.^20,21,27 Yapar et al evaluated the potential of ChatGPT to support early home care for patients following orthopedic procedures and to improve public health outcomes. Their findings suggested that ChatGPT can provide accurate and practical information for post-orthopedic care.²⁰ Yaş et al compared the performance of ChatGPT and Google Gemini in the management of adolescent idiopathic scoliosis. The results demonstrated that both models exhibited moderate accuracy and communication quality in scoliosis-related scenarios, with ChatGPT showing a modest advantage. Nevertheless, the authors emphasized that, despite their potential in patient education and informational support, these models should be used with caution due to current limitations in accuracy and comprehensiveness.²¹

The main strength of the study is that the questions were prepared based on questions asked by real patients. Before starting this study, we carefully examined the questions asked by patients admitted to hospitals with drug allergies, and identified the 10 most frequently asked questions by expert and experienced physicians. Another advantage of this study is that we had 40 immunology and allergy specialists who evaluated ChatGPT-5's responses. Compared with studies in the literature that used LLMs in allergy, our study included a larger number of raters.

However, several limitations should also be considered. First, the study was conducted in a single country and evaluated only 1 chatbot model, ChatGPT-5. Second, the responses were assessed only within the predefined domains of accuracy, currency, comprehensiveness, usefulness, and understandability. A structured qualitative content analysis evaluating additional response characteristics such as core informational content, safety-related warnings, and recommendations for physician consultation was not prospectively performed during the original evaluation process. Third, some evaluation domains, particularly comprehensiveness and usefulness, may have partially overlapped in interpretation despite the predefined rubric descriptions. Although a structured rubric was used to standardize evaluations, no formal psychometric validation, pilot standardization process, or formal calibration meetings were performed before the expert assessments. Fourth, inter-rater reliability analyses such as ICC or kappa were also not performed, because the primary aim of this study was not to assess agreement among experts but rather to reflect individual expert judgments regarding the quality of ChatGPT-generated responses. Fifth, the use of snowball sampling may have introduced selection bias and possible overrepresentation of certain professional peer groups, particularly in subjective evaluation domains such as understandability. Sixth, no evaluator assigned the lowest possible score (1 point) to any response across the evaluated domains. This finding may reflect the generally acceptable quality of the responses, although the possibility of a tendency to avoid extreme ratings in subjective Likert-based evaluations cannot be completely excluded. Seventh, only single-turn interactions were evaluated, and repeated prompting or response variability analyses were not performed. In real-world settings, patients may ask follow-up questions and engage in iterative conversations with LLMs, which could potentially influence the quality and completeness of responses. Finally, only the 10 most frequently asked patient questions were evaluated. Although this number is comparable to previous scenario-based LLM evaluation studies, these questions may not fully represent the diversity and complexity of real-world drug allergy inquiries.

Conclusion

While literature existed on the use of ChatGPT in various medical fields, there was a lack of data in the field of drug allergy.7, 8, 9, 10, 11, 12^,20,21,24, 25, 26, 27 Patients' evaluations in allergy units for drug allergies take a long time due to the crucial history and the lack of highly specific and sensitive diagnostic tests for most drugs. Considering the scarcity of allergists and the increasing prevalence of drug allergies, the time required for patient appointments with allergy specialists is often long worldwide. Therefore, it is crucial for AI LLMs to provide patients with easily accessible, accurate, up-to-date, comprehensive, and understandable information in this field. We aim to make a significant contribution to the literature with this study. We believe that AI technology will become even more effective in drug allergy over time.

Abbreviations

AI, Artificial intelligence; LLMs, Large language models; SD, Standard deviation;Q1–Q3, Interquartile range; Q, Questions; AR, Allergic rhinitis; CRS, Chronic rhinosinusitis; JAMA, Journal of the American Medical Association; SFLEDM, Swiss Federal Licensing Examination in Dental Medicine; EEAACI, European Examination in Allergy and Clinical Immunology

Disclosure statement regarding use of generative artificial intelligence

No artificial intelligence was used at any stage of the study until its submission for publication. Only, as required by the study's design, ChatGpt was asked questions about drug allergies and its responses were recorded. These questions and ChatGpt responses are also uploaded as attachments.

Data availability statement

The data supporting the findings of this study are available from the corresponding author upon request.

Author contributions

TST made the design, obtaining ethics committee approval, data collection, and writing of the study. SA made the design, data processing and writing the study.

Statement of ethics

Ethics approval was obtained from the Scientific Research Ethics Committee of Antalya Training and Research Hospital (decision number: 14/31, date: August 28, 2025). All experts were informed of the study's aim, and informed consent was obtained.

Consent for publication

All authors approve consent for publication.

Funding sources

There was not any financial source for this study.

Declaration of competing interest

The authors declare no conflict of interest in relation to this research, authorship, and/or publication of this article. However, unrelated to the submitted work, it is reported that TST was a speaker for Novartis and AstraZeneca.

Appendix A. Supplementary data

The following are the Supplementary data to this article:

Supp Table 1

Questions directed to ChatGPT-5 and responses of ChatGPT-5.

Supp Table 2

Evaluation of ChatGPT's Answers to the Questions by the Experts: Detailed Distribution of Physician Ratings.