A comparative study of ChatGPT 4o and DeepSeek in addressing CIED infection-related questions: Accuracy and readability assessment

Abstract

This study aimed to compare the effectiveness of 2 artificial intelligence (AI) models, ChatGPT 4o and DeepSeek, in responding to questions about infections associated with cardiovascular implantable electronic devices (CIED). The focus was on evaluating their accuracy and readability, which are critical for their use in clinical settings. A comparative analysis was conducted using 30 questions based on American Heart Association’s guidelines for CIED-related infections. Each question was asked to both AI models under 2 conditions: once without additional context and once with guideline-based prompts. Accuracy was assessed using a 4-level grading scale by 2 cardiovascular experts. Readability was measured using the Flesch-Kincaid Grade score and word-count metrics. Without guideline prompts, ChatGPT 4o provided comprehensive answers for 24 out of 30 questions (80.00%), with 5 correct but incomplete answers (16.67%) and one partially correct answer (3.33%). DeepSeek also provided comprehensive answers for 24 questions (80.00%) but had 6 correct but incomplete answers (20.00%). With guideline prompts, ChatGPT 4o’s comprehensive answer rate increased to 93.33% (28/30), while DeepSeek’s rate rose to 90.00% (27/30). No significant difference in overall accuracy was found (P = .34). In terms of readability, ChatGPT 4o had a higher word count (859.10 ± 235.90) compared to DeepSeek (526.27 ± 100.45), with a statistically significant difference (P <.01). The Flesch-Kincaid Grade Score for ChatGPT 4o (15.40 ± 1.18) was higher than that of DeepSeek’s (13.91 ± 1.42), indicating more complex responses (P <.01). With guidelines, both models showed reduced verbosity, with ChatGPT 4o’s word-count dropping to (624.00 ± 249.01) and DeepSeek’s to (549.43 ± 117.40); however, this change was not statistically significant (P = .13). Similarly, slight improvements in readability with guidelines were observed for both models, but these were not statistically significant (P = .11). Both AI models demonstrated the ability to provide accurate and clinically relevant information for managing CIED infections. The use of guideline-based prompts significantly improved the completeness of their responses. ChatGPT 4o provided more detailed answers, while DeepSeek produced more concise, potentially easier-to-understand outputs.

1. Introduction

Cardiovascular implantable electronic devices (CIED), including pacemakers, implantable cardioverter-defibrillators, and cardiac resynchronization therapy devices, have significantly transformed the management of various cardiovascular conditions.^[1] However, the rising incidence of infections associated with these devices presents a considerable challenge in patient care. Such infections can lead to severe health complications, higher mortality rates, and increased healthcare costs.^[2] The reported prevalence of CIED-related infections ranges from 1.48% to 5.40%, with even higher rates observed in high-risk patient populations.^[3] Such infections can lead to severe health complications, including life-threatening infections such as endocarditis and sepsis, as well as higher mortality rates and increased healthcare costs.

The management of CIED infections requires a comprehensive strategy involving antibiotic treatment, device removal, and surgical debridement of the infected area.^[4] Early and accurate diagnosis plays a key role in improving patient outcomes. Clinical practice guidelines, such as those from the American Heart Association (AHA), offer evidence-based recommendations for the prevention, diagnosis, and management of these infections.^[5] However, the complexity of these guidelines, combined with the rapidly advancing clinical landscape, can pose challenges for healthcare professionals in making well-informed decisions.

The advancement of Industry 4.0 technologies, particularly artificial intelligence (AI), has significantly transformed healthcare delivery by enabling the processing of vast medical datasets, identifying complex patterns, and supporting innovations in diagnosis, risk stratification, and personalized treatment planning.^[6,7] Furthermore, innovations such as the clustered regularly interspaced short palindromic repeats-Cas system have been harnessed to combat pathogens and revolutionize medicine, offering new approaches to managing infections.^[8] Specifically for CIED-related infections, AI can assist in early detection through the analysis of remote monitoring data (e.g., temperature, heart rate variability), prediction of infection risk based on comorbid conditions (such as diabetes or chronic kidney disease), and optimization of antibiotic therapy by integrating patient-specific microbiological and clinical features. These applications demonstrate AI’s potential to support timely and precise clinical decision-making in managing CIED infections. ChatGPT, developed by OpenAI, is a chatbot powered by an advanced language model designed for natural language tasks,^[9] while DeepSeek is another AI model known for its strength in language processing.^[10] Although ChatGPT and DeepSeek are among the most advanced AI models with strong language processing capabilities, their comparative performance in clinically specialized domains – particularly in managing device-related infections – has not been well explored. Most existing studies focus on general medical knowledge or patient education content, without addressing how these models perform when guided by clinical practice guidelines in highly specialized scenarios such as CIED infections. To our knowledge, this is the first study to conduct a structured evaluation of ChatGPT 4o and DeepSeek in this context, providing a comparative analysis of their accuracy and readability in delivering infection-specific, guideline-aligned responses.

This study aims to evaluate the effectiveness of ChatGPT 4o and DeepSeek in responding to clinical questions related to CIED infections, with a particular focus on accuracy and readability. Accuracy was assessed using a 4-level expert scoring rubric, while readability was measured by the Flesch-Kincaid Grade Level and word count. By examining how well these models deliver reliable, guideline-aligned, and clinically relevant information, we aim to clarify their potential role in supporting CIED infection management. Furthermore, we investigate whether incorporating guideline-based prompts enhances the quality of AI-generated responses, providing insight into strategies for optimizing the clinical utility of AI tools.

2. Methods

2.1. Study design

This study compared the performance of 2 AI models, ChatGPT 4o and DeepSeek, in responding to questions about infections associated with CIED. The primary focus was to assess the accuracy and readability of their responses, both with and without prompts derived from clinical guidelines. ChatGPT 4o and DeepSeek were selected for comparison because they represent state-of-the-art large language models (LLM) with advanced capabilities in natural language understanding and generation. ChatGPT 4o has been widely used in clinical and educational settings for its ability to produce context-rich responses, while DeepSeek has demonstrated effectiveness in multilingual medical information processing and structured content generation. Their inclusion in this study enables a meaningful evaluation of high-performance AI models in specialized medical domains such as CIED infection management.

2.2. Question source

A total of 30 questions were generated by Doubao, an AI-driven question-answering platform, based on the clinical practice guideline “Update on Cardiovascular Implantable Electronic Device Infections and Their Prevention, Diagnosis, and Management: A Scientific Statement From the AHA: Endorsed by the International Society for Cardiovascular Infectious Diseases.”^[5] Doubao employs advanced natural language processing algorithms to generate questions that are contextually relevant and aligned with current clinical guidelines. The choice of Doubao was made to ensure the generated questions reflect key aspects of CIED infection management, including prevention, diagnosis, and treatment. To ensure clinical relevance and accuracy, the generated questions were reviewed and validated by 2 experienced cardiovascular specialists, each with over a decade of experience in the field. The specialists assessed the questions for both clinical applicability and accuracy, confirming that the questions effectively addressed the most important elements of CIED infection management. The questions also covered a range of difficulty levels, from basic to advanced, to thoroughly evaluate the performance of both AI models in answering questions of varying complexity.

The algorithmic quality of Doubao question generation process is based on its ability to align questions with established clinical guidelines, ensuring that they cover a comprehensive range of topics relevant to the management of CIED infections. Additionally, the clinical relevance of these questions was confirmed through expert review, ensuring that the generated questions were both accurate and applicable to real-world clinical settings. This validation process helped to ensure that the questions were appropriately designed to assess the AI models’ capabilities in addressing both basic and complex aspects of CIED infection management.

2.3. Question posing

On March 11, 2025, both ChatGPT 4o and DeepSeek were given all 30 inquiries. Each question was asked twice: once without additional context or prompts, and once with prompts based on established clinical guidelines to provide context and improve response relevance. Responses from both models were recorded verbatim for further analysis.

To assess the AI models’ performance in answering clinical questions with and without guideline-based context, we used the following examples:

Without guideline prompt:

Example question: “What are the general preventive measures for CIED infections?”

AI input (without guideline context): “What are the general preventive measures for CIED infections?”

This prompt was presented to the AI models without any context, allowing us to evaluate the models based on their general knowledge of CIED infection diagnosis.

With guideline prompt:

Example question: “What are the general preventive measures for CIED infections?”

AI input (with guideline context): “According to the AHA’s scientific statement, ‘Update on Cardiovascular Implantable Electronic Device Infections and Their Prevention, Diagnosis, and Management,’ What are the general preventive measures for CIED infections?”

In this case, the AI models were prompted with the specific context of the AHA guidelines, ensuring that the responses were informed by the latest evidence-based recommendations for CIED infection management.

We have included a complete list of all 30 questions and their corresponding input prompts, both with and without guideline context, in Appendix A, Supplemental Digital Content, https://links.lww.com/MD/R280 to facilitate reproducibility.

2.4. Assessment criteria

The assessment of the AI-generated responses focused on 2 primary dimensions: accuracy and readability.

2.4.1. Accuracy

• Grading scale: responses were evaluated using a 4-level grading scale^[11]:

• Comprehensive: the response is accurate and complete, requiring no additional information.

• Correct but incomplete: tresponse contains accurate information but lacks some details.

• Some correct and some incorrect: the response includes both accurate and inaccurate elements.

• Completely incorrect: the response is entirely inaccurate or misleading.

2.4.2. Readability

The Flesch-Kincaid grade score assesses the readability of text by considering sentence length and word complexity. A lower score indicates that the text is easier to understand. Additionally, the word count of each response was recorded to evaluate the conciseness of the AI-generated answers. Both the Flesch-Kincaid grade score and the word count were determined using the ‘Tests Document Readability’ webpage, which can be found at https://www.online-utility.org/english/readability_test_and_improve.jsp.

2.5. Evaluation process

The responses were evaluated independently by 2 cardiovascular specialists, both possessing over ten years of clinical experience, without any knowledge of the AI model that produced each response. In instances where the specialists disagreed, a third expert was brought in to help achieve consensus.

2.6. Statistical analysis

Statistical analysis was performed using SPSS Statistics version 26.0 (Chicago). Continuous variables are presented as means with standard deviations, while categorical variables are expressed as counts (n) and percentages (%). For continuous variables with a normal distribution, a one-way analysis of variance (ANOVA) with the least significant difference post hoc test was used for pairwise comparisons. For non-normally distributed continuous variables, the Kruskal-Wallis H test was applied. Fisher Exact Test was used to analyze categorical variables. Cohen kappa statistic was calculated to assess the agreement between the 2 AI models. All tests were 2-tailed, and a P-value of <.05 was considered statistically significant.

2.7. Ethical approval

Ethical approval was not required because the study involved the analysis of AI-generated text and did not involve human subjects.

3. Results

3.1. Accuracy of responses

In a comparative analysis of 2 AI models, ChatGPT 4o and DeepSeek, the performance metrics were assessed both with and without guideline prompts. Initially, without guidelines, ChatGPT 4o produced 24 comprehensive answers (80.00%), 5 correct but incomplete answers (16.67%), and 1 answer that was partially correct and partially incorrect (3.33%). In contrast, DeepSeek also delivered 24 comprehensive answers (80.00%) but had 6 correct but incomplete answers (20.00%), with no responses that were partially correct or completely incorrect. Upon introducing guideline prompts, both models demonstrated improved performance. ChatGPT 4o with guidelines achieved 28 comprehensive answers (93.33%) and only 2 correct but incomplete answers (6.67%). Similarly, DeepSeek with guidelines provided 27 comprehensive answers (90.00%) and 3 correct but incomplete answers (10.00%), as illustrated in Table 1. Statistical analysis indicated no significant difference in overall accuracy between the 2 models (Fisher Exact Test, P = .34). The introduction of guidelines seemed to have a favorable impact on both platforms, with a tendency toward higher comprehensive responses, although these differences were not statistically significant (Fig. 1).

Table 1.

Accuracy assessment of ChatGPT 4o and DeepSeek responses to CIED infection questions.

Grading scale	ChatGPT 4o (n [%])	DeepSeek (n [%])	ChatGPT 4o + guideline (n [%])	DeepSeek + guideline (n [%])
Total (n = 30)
Comprehensive	24 (80.00)	24 (80.00)	28 (93.33)	27 (90.00)
Correct but incomplete	5 (16.67)	6 (20.00)	2 (6.67)	3 (10.00)
Some correct and some incorrect	1 (3.33)	0 (0)	0 (0)	0 (0)
Completely incorrect	0 (0)	0 (0)	0 (0)	0 (0)
Prevention (n = 10)
Comprehensive	9 (90.00)	10 (100.00)	8 (80.00)	9 (90.00)
Correct but incomplete	1 (10.00)	0 (0)	2 (20.00)	1 (10)
Some correct and some incorrect	0 (0)	0 (0)	0 (0)	0 (0)
Completely incorrect	0 (0)	0 (0)	0 (0)	0 (0)
Diagnosis (n = 10)
Comprehensive	9 (90.00)	8 (80.00)	10 (100.00)	9 (90.00)
Correct but incomplete	1 (10.00)	2 (20.00)	0 (0)	1 (10.00)
Some correct and some incorrect	0 (0)	0 (0)	0 (0)	0 (0)
Completely incorrect	0 (0)	0 (0)	0 (0)	0 (0)
Treatment (n = 10)
Comprehensive	6 (60.00)	6 (60.00)	10 (100.00)	9 (90.00)
Correct but incomplete	3 (30.00)	4 (40.00)	0 (0)	1 (10.00)
Some correct and some incorrect	1 (10.00)	0 (0)	0 (0)	0 (0)
Completely incorrect	0 (0)	0 (0)	0 (0)	0 (0)

Figure 1.

Comparison of comprehensive answer rates for ChatGPT 4o and DeepSeek with and without guideline prompts.

The Kappa values of the AI models are detailed in Table 2. The comparison of ChatGPT 4o and DeepSeek yielded a kappa value of 0.27, which, according to the Landis and Koch classification, represents a “fair” level of agreement (0.21–0.40 = fair). The P-value of .09 suggests that the performance difference was not statistically significant, although a trend toward agreement was observed. In the comparison of ChatGPT 4o with the guidelines versus DeepSeek with guidelines, the kappa value of 0.35 indicated a fair level of agreement, and the P-value of .05 did not meet the conventional threshold for statistical significance (P <.05), but it suggests a trend toward improved agreement that may warrant further investigation.

Table 2.

Inter-rater reliability kappa values for ChatGPT 4o and DeepSeek comparisons.

Comparison groups	Kappa	95% CI	P-value
ChatGPT 4o versus DeepSeek	0.286	(−0.08 to 0.65)	.09
ChatGPT 4o + guideline versus DeepSeek + guideline	0.348	(−0.22 to 0.92)	.05

In the treatment domain, both ChatGPT 4o and DeepSeek started with a moderate performance, each achieving a comprehensive response rate of 60.00%. However, when provided with guidelines, ChatGPT 4o excelled, reaching a perfect comprehensive response rate of 100%, while DeepSeek followed closely with a 90% rate. In the area of prevention, ChatGPT 4o initially recorded a 90.00% comprehensive response rate, which decreased to 80.00% upon receiving the guidelines, whereas DeepSeek maintained an impressive 100.00% rate. Regarding diagnosis-related questions, both platforms began with a 90.00% comprehensive response rate; ChatGPT 4o sustained this rate even with guidelines, while DeepSeek experienced a slight decline to 80.00% when guidelines were applied (Table 1).

3.2. Readability of responses

ChatGPT 4o produced responses with a notably higher average word count of (859.10 ± 235.90) compared to DeepSeek, which had an average of (526.27 ± 100.45). This indicates that ChatGPT 4o tends to generate more elaborate responses, and the difference was statistically significant (P <.01). After implementing guidelines, there was a decrease in verbosity for both platforms; the word count for ChatGPT 4o with guidelines dropped to (624.00 ± 249.01), while DeepSeek with guidelines decreased to (549.43 ± 117.40). However, this change was not statistically significant (P = .13) (Tables 3 and 4).

Table 3.

Flesch–Kincaid grade score and word-count metrics for ChatGPT 4o and DeepSeek responses.

Grading scale	ChatGPT 4o	DeepSeek	ChatGPT 4o + guideline	DeepSeek + guideline	F	P
Total (n = 30)
number of words (mean ± SD)	859.10 ± 235.90	526.27 ± 100.45	624.00 ± 249.01	549.43 ± 117.40	19.61	<.01
Flesch–Kincaid grade Score (mean ± SD)	15.40 ± 1.18	13.91 ± 1.42	16.58 ± 1.63	15.96 ± 1.65	17.77	<.01
Prevention (n = 10)
Number of words (mean ± SD)	615.80 ± 244.96	464.80 ± 13.51	425.60 ± 180.56	433.50 ± 80.34	–	–
Flesch–Kincaid grade score (mean ± SD)	15.72 ± 1.64	13.51 ± 1.08	16.75 ± 1.64	16.23 ± 1.44	–	–
Diagnosis (n = 10)
Number of words (mean ± SD)	928.20 ± 104.99	484.50 ± 68.57	669.60 ± 204.30	556.90 ± 57.82	–	–
Flesch–Kincaid grade score (mean ± SD)	15.19 ± 1.12	13.66 ± 2.07	16.86 ± 1.98	16.92 ± 1.80	–	–
Treatment (n = 10)
Number of words (mean ± SD)	1033.30 ± 59.09	629.50 ± 73.02	776.80 ± 231.24	657.90 ± 80.93	–	–
Flesch–Kincaid grade score (mean ± SD)	15.29 ± 0.63	14.56 ± 0.59	16.13 ± 1.28	14.74 ± 0.86	–	–

Table 4.

Detailed mean differences in Flesch–Kincaid grade score and word count between ChatGPT 4o and DeepSeek comparisons.

Variable	Comparison groups	Mean difference (I-J)	Standard error	P	95% Confidence interval
Flesch–Kincaid grade score	ChatGPT 4o versus DeepSeek	1.49	0.38	<.01	(0.73–2.25)
Flesch–Kincaid grade score	ChatGPT 4o versus ChatGPT 4o + guideline	-1.18	0.38	<.01	(-1.94 to −0.42)
Flesch–Kincaid grade score	ChatGPT 4o versus DeepSeek + guideline	-0.56	0.38	.15	(-1.32 to 0.20)
Flesch–Kincaid grade score	DeepSeek versus ChatGPT 4o + guideline	-2.67	0.38	<.01	(−3.43 to −1.91)
Flesch–Kincaid grade score	DeepSeek versus DeepSeek + guideline	-2.05	0.38	<.01	(−2.81 to −1.29)
Flesch–Kincaid grade score	ChatGPT 4o + guideline versus DeepSeek + guideline	0.62	0.38	.11	(−0.14 to 1.38)
Number of words	ChatGPT 4o versus DeepSeek	332.83	48.57	<.01	(236.64–429.03)
Number of words	ChatGPT 4o versus ChatGPT 4o + guideline	235.1	48.57	<.01	(138.90–331.30)
Number of words	ChatGPT 4o versus DeepSeek + guideline	309.67	48.57	<.01	(213.47–405.86)
Number of words	DeepSeek versus ChatGPT 4o + guideline	-97.73	48.57	.05	(−193.93 to −1.54)
Number of words	DeepSeek versus DeepSeek + guideline	-23.17	48.57	.63	(−119.36 to 73.03)
Number of words	ChatGPT 4o + guideline versus DeepSeek + guideline	74.57	48.57	.13	(−21.63 to 170.76)

In terms of readability, measured by the Flesch-Kincaid Grade Score, ChatGPT 4o had an average score of (15.40 ± 1.18), which corresponds to a reading level typical of undergraduate college students, indicating relatively complex text. This score was significantly higher than DeepSeek’s score of (13.91 ± 1.42) (P <.01). With the introduction of guideline prompts, the readability score for ChatGPT 4o with guidelines increased to (16.58 ± 1.63), and for DeepSeek with guidelines, it increased to (15.96 ± 1.65). A higher Flesch-Kincaid Grade Score indicates that the text became more difficult to read. However, these improvements were not statistically significant (P = .11) (Tables 3 and 4).

Statistical analysis confirmed significant variations in both readability and word count across different platforms and conditions. The F-test results for the Flesch–Kincaid Grade Scores showed a highly significant difference (F = 17.77, P <.01), as did the word count (F = 19.61, P <.01), indicating that both the choice of platform and the use of guidelines significantly affected these aspects of response quality (Table 3). Detailed comparisons between the 2 groups can be found in Table 4.

4. Dicussion

This study provides insights into how ChatGPT 4o and DeepSeek perform when addressing questions about CIED-related infections. Both AI models demonstrated strengths and weaknesses in providing accurate, clinically relevant responses, especially in the realm of clinical decision support for infections related to CIED.

4.1. Accuracy of responses

Both models achieved a notable 80% comprehensive response rate without guideline prompts, demonstrating their ability to provide clinically pertinent information. When guideline-based prompts were introduced, both models showed improvement: ChatGPT 4o increased to 93.33%, and DeepSeek to 90%. While these improvements highlight the role of structured guidelines in enhancing AI accuracy, it is important to note that the P-value for the overall comparison of accuracy between the models did not reach statistical significance (P = .34). This suggests that, although guideline prompts improved performance, the differences in accuracy between the 2 models were not statistically significant.

The models’ performance varied across the different aspects of CIED infections. In prevention, DeepSeek initially achieved 100% accuracy, which dropped to 90% with guidelines. In contrast, ChatGPT 4o’s accuracy decreased from 90% to 80% with guidelines. This decrease may be due to several factors, including the complexity of the guideline prompts, conflicting interpretations of the guidelines, and limitations in how the models parse long structured inputs. In diagnosis, DeepSeek’s accuracy rose from 90% to 100%, while ChatGPT 4o improved from 80% to 90%. In treatment, both models began with 60% accuracy, but ChatGPT 4o reached 100%, and DeepSeek improved to 90%. These differences suggest that diagnosis and treatment are generally more straightforward and benefit more from structured guidelines, while prevention requires more complex decision-making, which may explain the initially lower performance. The guidelines likely provided clearer frameworks, particularly benefiting ChatGPT 4o in the treatment area.

The inter-rater reliability between the 2 models was mild to moderate, with a Kappa coefficient of 0.29 (P = .09), improving slightly to 0.35 (P = .05) with guideline prompts. Although these P-values did not achieve statistical significance, the trend suggests that guideline prompts can enhance consistency and alignment with clinical standards, but further studies with larger sample sizes may be necessary to confirm these findings and their applicability in real-world clinical settings.^[12,13]

4.2. Readability of responses

The readability evaluation revealed a notable difference in verbosity between ChatGPT 4o and DeepSeek. ChatGPT 4o generated longer responses, with an average word count of (859.10 ± 235.90), compared to DeepSeek’s (526.27 ± 100.45), a statistically significant difference (P <.01). After applying the guidelines, both models became less verbose, although the change wasn’t statistically significant. This suggests that guideline-based prompts may lead to more concise responses, which is beneficial in clinical settings where time efficiency is crucial.^[14]

The Flesch-Kincaid grade score, a measure of text complexity, showed that ChatGPT 4o’s responses were more complex, scoring (15.40 ± 1.18) compared to DeepSeek’s (13.91 ± 1.42) (P <.01). A score of 15.40 corresponds to a college-level (undergraduate) reading level, indicating a more complex text, while a score of 13.9 aligns with high school-level reading, indicating a less complex structure. Even after the guidelines were applied, ChatGPT 4o’s complexity remained high, scoring (16.58 ± 1.63), with a significant difference between scores with and without guidelines (P = .03). DeepSeek’s score improved slightly to (15.96 ± 1.65) after guidelines were applied (P <.01). However, no significant difference was found when comparing DeepSeek with guidelines to ChatGPT 4o with guidelines (P = .11). Given the higher complexity of ChatGPT 4o’s responses, clinicians or nonspecialists under time pressure may find its responses more difficult to process compared to DeepSeek’s responses, which were simpler and potentially more accessible.

The F-test results confirmed significant differences in both Flesch-Kincaid scores and word count (F = 17.77, P <.01 for Flesch–Kincaid grade scores; F = 19.61, P <.01 for word count). This indicates that both the AI platform and the use of guidelines significantly influenced response quality. While ChatGPT 4o offers more detailed answers, its complexity can make it harder to understand, especially for those seeking succinct responses. In contrast, DeepSeek’s simpler responses may be easier to comprehend, although they lack details. This difference is crucial in clinical practice, where balancing clarity and thoroughness is crucial.

4.3. Implications for clinical practice

The findings highlight the potential of AI models, including ChatGPT 4o and DeepSeek, in managing infections related to CIED by providing accurate and relevant information. Both models performed well, with noticeable improvements in accuracy when guided by structured clinical prompts. This suggests that ChatGPT 4o can be a valuable tool for clinicians needing comprehensive guidance, offering evidence-based recommendations aligned with clinical protocols.^[15] However, DeepSeek, with its simpler and more concise outputs, may be more suitable for patient-facing communication or quick summaries, particularly when conveying complex information to patients or nonspecialists. While both models have shown utility, their effectiveness varied across different areas, particularly in treatment, which is more complex and requires tailored approaches. This indicates the need for further development of AI models that can better handle individualized treatment plans.

The differences in readability between the 2 models also have important implications for clinical practice. ChatGPT 4o tends to produce more detailed responses, which could be beneficial for clinicians requiring in-depth information. DeepSeek, on the other hand, produces clearer, more concise outputs, which may be better suited for patient education or when time constraints are a factor, ensuring that nonspecialists and patients can understand the key points. Clinicians can use these models to create understandable explanations of CIED infections, treatment options, and preventive strategies, helping patients engage in their care and follow treatment plans.

Additionally, AI’s ability to generate precise responses can streamline clinical workflows, allowing healthcare providers to quickly access relevant information and focus more on patient care. Future research should examine integrating AI models with electronic health records for real-time, patient-centered recommendations based on the latest clinical guidelines. Furthermore, developing specialized AI models trained specifically on CIED infection data could improve performance and address challenges such as personalized treatment and risk assessment.

4.4. Limitations of the study

While the results of this study are promising, several limitations must be acknowledged. First, the AI efficacy assessment was based on a limited set of queries, which does not reflect the models’ performance over time or in varying clinical situations. Second, the study focused on only 2 AI models; incorporating a broader range of models would offer a more comprehensive understanding of AI’s capabilities in managing CIED infections. Additionally, the models were not evaluated in real clinical environments, where factors like patient demographics and data variability could lead to different outcomes. Finally, human specialists evaluated the models’ outputs, which could introduce subjectivity based on individual biases or interpretations of clinical guidelines.

Moreover, while we used the Flesch–Kincaid grade level to assess readability, we acknowledge that this method alone may not fully capture the complexity of medical content, especially in specialized clinical domains like CIED infections. The Flesch-Kincaid Grade Level is based on general readability principles and may not account for the density of medical terminology or the clinical nuances vital for accurate understanding in healthcare settings. Although it provides useful information about general readability, future studies could consider incorporating additional measures, such as medical terminology density or user comprehension proxies, to better assess the models’ ability to communicate complex medical information.

Furthermore, we did not apply more advanced statistical tests, such as paired t-tests or repeated measures ANOVA, to assess the within-model differences (e.g., word-count changes pre- and post-guideline). While we reported aggregate differences in word count, more robust statistical methods could have provided deeper insights into within-subject variability and model performance. This limitation highlights the potential for future research, where these additional statistical methods could be employed to better understand the impact of guideline-based prompts on the models’ responses.

In addition to these statistical and readability limitations, there are several critical real-world challenges of LLMs in clinical settings that must be emphasized. For instance, LLMs may generate hallucinations, producing plausible but factually incorrect content, especially when addressing complex or underrepresented clinical scenarios. In the case of CIED infections, hallucinations – such as erroneous antibiotic regimens or misclassification of infection stages – could misguide clinical decisions or patient education, posing potential risks to patient safety. The ability of these models to maintain accuracy across diverse and evolving clinical scenarios remains a key area for future refinement.

Another important limitation is the medicolegal responsibility of AI models in clinical practice. Current legal frameworks lack clear guidelines on liability attribution for AI-generated recommendations. If adverse events occur following reliance on AI responses, it remains ambiguous whether responsibility lies with clinicians, healthcare institutions, or AI developers. This ambiguity could hinder widespread adoption of LLMs in clinical practice, as stakeholders may be reluctant to assume unregulated risks. As AI models become more integrated into clinical workflows, regulatory clarity will be essential to mitigate these concerns and ensure safe AI usage.

Finally, the lack of transparency in LLMs poses challenges to clinical integration. The “black-box” nature of LLMs obscures the reasoning process behind their outputs, making it difficult for clinicians to trace how specific conclusions are derived from input data. This lack of transparency undermines trust in AI responses and complicates the validation of their clinical appropriateness. Greater transparency in the decision-making processes of AI models, including the identification of contributing factors and potential biases, is critical to ensuring their safe integration into clinical settings.

5. Conclusion

In conclusion, this study demonstrates the potential of AI models, including ChatGPT 4o and DeepSeek, in providing accurate and clinically relevant information for managing CIED infections. Both models showed noticeable improvements when guided by structured clinical prompts, suggesting that AI can be a valuable tool for clinicians, offering evidence-based recommendations aligned with clinical protocols. However, the models’ performance varied across different aspects, particularly in treatment, which is more complex and requires tailored approaches, highlighting the need for further refinement.

The differences in readability between the 2 models also have important implications for clinical practice. ChatGPT 4o tends to provide more detailed responses, which can be beneficial for clinicians needing comprehensive guidance, while DeepSeek offers clearer, more concise outputs, making it better suited for patient education or situations requiring quick summaries.

Looking forward, future research should focus on integrating AI models with electronic health records for real-time, patient-specific recommendations, which could further enhance the clinical utility of AI. Additionally, developing specialized AI models trained on CIED infection data could improve model performance in personalized treatment plans and risk assessments. Moreover, further studies should explore refining AI’s ability to parse complex clinical guidelines and improve its transparency to ensure safe integration into clinical workflows.

Author contributions

Conceptualization: Xiaoli Chen.

Data curation: Yu Chen, Weihua Shen, Yini Zhang.

Methodology: Jianhua Fan, Jiasheng Wen.

Project administration: Xiaoli Chen.

Supervision: Ling Li.

Validation: Chang Yu, Jiasheng Wen.

Writing – original draft: Chang Yu, Jianhua Fan.

Writing – review & editing: Xiaoli Chen.