Feasibility of LLM-assisted post-discharge tuberculosis care: A comparative study of medication counseling, patient education, and follow-up planning

Abstract

Background

Tuberculosis (TB) remains a major global health concern, with poor post-discharge adherence driving relapse and drug resistance. While China’s 2024 guidelines advocate for “intelligent post-discharge management,” the role of Large Language Models (LLMs) in this specialized care remains unexplored.

Objective

To explore the feasibility of LLMs as auxiliary tools to complement TB specialists in post-discharge medication counseling, patient education, and follow-up planning.

Method

This exploratory study compared two LLMs (ChatGPT-4o, DeepSeek-R1) with TB physicians using 17 standardized clinical cases. Responses were assessed by three blinded specialists using objective metrics (precision, recall, F1) and subjective ratings across seven domains. Linear mixed models (LMMs) were employed to analyze the fixed effects of participant type while controlling for the random effects of clinical cases.

Result

In patient education, LLM-generated responses exhibited higher objective completeness and accuracy than those of physicians (precision: 0.86 vs 0.50, P < .001; recall: 0.53 vs 0.14, P= .026; F1: 0.65 vs 0.22, P= .018). Conversely, no statistically significant differences were observed in objective metrics for medication counseling and follow-up planning (all P > .050), although LLMs demonstrated a tendency toward broader but more discordant coverage. Subjectively, LLMs were rated significantly higher in suitability, understandability, and empathy across all tasks (all P < .050). No significant differences were found regarding perceived safety or text readability (P > .050).

Conclusion

This study suggests the potential of LLMs as auxiliary tools in TB post-discharge management. While quantitative performance in high-stakes tasks was comparable to physicians, the complementary qualitative patterns support a hybrid human-AI approach. These preliminary findings provide a basis for integrating LLMs into clinical workflows under professional supervision to enhance efficiency in resource-limited settings.

Introduction

Tuberculosis (TB) continues to be one of the most serious infectious diseases worldwide and continues to pose a major challenge to global health systems. ¹ In 2023, 10.8 million new cases were reported globally, with a persistent burden of multidrug-resistant (MDR-TB) and rifampicin-resistant TB (RR-TB). ² The complexity of TB treatment regimens often leads to poor adherence, contributing to treatment failure, relapse, and drug resistance.^2–4 After completing treatment and discharge, patients are required to self-manage their care, including medication adherence and follow-up, which can be challenging due to limited healthcare resources, low health literacy, and socio-economic barriers.

Following treatment completion and hospital discharge, TB management shifts from healthcare institutions to home or community settings, entering a stage that relies heavily on patient self-management. During this stage, patients must adhere to regular medication schedules, monitor for adverse effects, and attend follow-up to ensure treatment continuity and effectiveness.^4–7 However, limited healthcare resources, uneven distribution of medical staff, and differences in patients’ literacy and socioeconomic conditions often affect follow-up participation and adherence, disrupting continuity of care and treatment success.^8–10 Importantly, these barriers are not evenly distributed: they disproportionately affect populations facing social and structural vulnerabilities (e.g., poverty, limited access to services, low health literacy), thereby amplifying within- and between-country inequities in post-discharge TB outcomes.^9,11,12 As a high TB burden country, China faces similar challenges, with a TB incidence ranking fourth globally (6.5%) and significant multidrug-resistant TB issues. ¹³ Traditional follow-up methods such as phone calls or home visits are labor-intensive and unsustainable at scale. Consequently, inadequate post-discharge care contributes to low adherence, increased relapse risk, and reduced overall treatment success, underscoring the importance of innovative and sustainable management models.^14,15

Against this backdrop, health systems are increasingly exploring digital health and AI-enabled approaches to extend post-discharge support beyond facilities, including personalized follow-up, health monitoring, medication support, and on-demand consultation. As one example, the National Health Commission of China released a 2024 guidance document describing“intelligent post-discharge management,” defined as the use of artificial intelligence (AI) and digital technologies to provide personalized follow-up, health monitoring, medication management, and consultation support for discharged patients. ¹⁶ Notably, health systems across high-burden contexts globally face the common challenge of implementing scalable, person-centered follow-up and education after discharge to ensure continued engagement in care. ¹⁷

Large language models (LLMs), a new class of AI systems capable of understanding and generating natural language, have shown promising applications in healthcare. Their strengths in contextual reasoning and patient-centered communication make them particularly suitable for health education, counseling, and follow-up support.^18–20 Existing research has highlighted their effectiveness in medical question-answering, clinical explanation generation, and patient education initiatives.^21–23 However, empirical evaluations of their accuracy, readability, empathy, and practical safety in TB-specific post-discharge settings remain scarce.

ChatGPT-4o and DeepSeek-R1 are representative large language models in medicine, representing closed-source and open-source models widely applied in the field. Both perform well in medical reasoning, diagnosis, and patient communication. Research shows ChatGPT-4o has strong multimodal reasoning and medical knowledge integration, while DeepSeek-R1 matches GPT-4 in clinical decision support and offers advantages in privacy and local deployment.^24–26

Therefore, this exploratory study used 17 standardized cases (10 real and 7 simulated) to compare responses independently generated by two LLMs (ChatGPT-4o and DeepSeek-R1) and TB specialists against expert-developed reference standards. The evaluation covered eight dimensions: accuracy, completeness, conciseness, safety, operability, readability, understandability, and empathy. By systematically comparing LLM- and clinician-generated outputs, this study seeks to provide empirical evidence for the safe, effective, and patient-centered integration of LLMs into post-discharge TB management, with potential implications for clinical efficiency, equity of access to counseling/education, and patient experience.

Method

Study design

This exploratory study was conducted in accordance with the CHART guidelines ²⁷ (a completed checklist is provided in Appendix 1 Table 1). We compared the performance of two large language models (LLMs)—ChatGPT-4o (a closed-source model developed by OpenAI, released in May 2024) and DeepSeek-R1 (an open-source model developed by DeepSeek, released in January 2025) with that of tuberculosis (TB) specialists in three post-discharge management tasks: medication counseling, patient education, and follow-up care planning. We evaluated responses from the LLMs and physicians against expert-developed reference standards, focusing on accuracy, completeness, and readability. Seventeen cases were included, consisting of seven standardized cases and ten real clinical cases from the hospital. Each answer generator, including two LLMs and two TB specialists, completed three post-discharge management tasks for each case, yielding 51 responses per answer generator and 204 responses in total.

A total of 17 cases were included, with seven standardized cases developed by the research team based on the 10th edition of the medical textbook Diagnostics and Classification of Tuberculosis (WHO 196-2017), incorporating expert clinical input to reflect common types of TB and symptoms.^28,29 Ten additional real-world cases were selected from the hospital, representing varying disease severity and comorbidities. All cases included structured information on symptoms, discharge medications, and follow-up needs. Two TB specialists independently reviewed each case for clinical representativeness and completeness; disagreements were resolved through consultation with a third expert. All cases were standardized to include demographic details, chief complaints, medical history, test results, medication information, and patient concerns.

Previous studies have consistently demonstrated that Regular medication intake, patient education, and continuous management are critical determinants of successful tuberculosis (TB) treatment. Irregular medication intake and insufficient TB-related knowledge have been associated with poor adherence and higher risks of relapse, whereas patient education, medication monitoring, and regular follow-up have been shown to improve treatment outcomes.^30,31 However, workforce shortages and unequal healthcare resources limit the continuity and personalization of traditional management models.^32–34 In parallel, scalable digital approaches are increasingly explored to strengthen continuity of care after discharge, particularly in high-burden and resource-constrained settings. ³⁵

Based on these findings and policy directions, this study identified three core post-discharge management scenarios: medication counselling, patient education and follow-up plan, to explore the feasibility of applying LLM to enhance adherence management and optimize recovery among TB patients. The study flow is shown in Appendix 1 Figure 1.

Case development and selection

To cover common and clinically representative TB management scenarios in primary care, cases were selected using purposive sampling based on expert consensus. Three TB experts (each with a senior professional title or clinical experience >15 years) selected 7 common TB case types based on the 10th edition of the Diagnostics and Classification of Tuberculosis (WS 196-2017) and developed corresponding standardized cases.

To reflect real-world clinical complexity, the same experts selected ten real-world cases from the case repository of Shenzhen Third People’s Hospital using maximum variation sampling. These cases were selected to cover common comorbidities, diverse demographic characteristics, and varying disease severity.

Before model input, all real-world cases were structured using the same presentation template as the standardized cases to reduce potential bias from differences in case information presentation. The template included patient age, sex, residence, education, occupation, chief complaint, medical history, comorbidities, examination findings, medications, patient concerns, and psychological status.

Cases were eligible for inclusion if they met all of the following criteria: (1) a confirmed diagnosis of TB according to national or international diagnostic standards; (2)complete clinical and discharge information, including medical history, examination findings, medication regimens, and follow-up information; (3) clinical features or post-discharge management needs representative of common post-discharge TB management scenarios. Cases were excluded if they involved: (1) rare or highly atypical scenarios beyond routine post-discharge TB management; (2) overly complex conditions requiring highly specialized individualized management and not representative of common post-discharge TB management scenarios. Details of patient characteristics, inclusion and exclusion criteria, and case information are provided in Appendix 1 (Tables 2–4).

Participants and grouping

The physician group included two TB physicians. Physician inclusion criteria were as follows: (1) a national practicing certificate; (2) being an attending physician or above with at least 3 years of independent clinical experience in TB diagnosis and treatment. Physician exclusion criteria were as follows: (1) involvement in case development, reference answer formulation, or evaluation criteria development; (2) failure to complete all case-based tasks or submission of incomplete responses. The detailed basic characteristics of the physician group are provided in Appendix 1 (Table 5). ChatGPT-4o and DeepSeek-R1 were selected as representative mainstream LLMs because of their strong reasoning abilities, conversational performance, and structured text generation capabilities. Post-discharge tuberculosis management involves multidimensional tasks, including health education, medication guidance, and follow-up planning, which require not only medical knowledge but also language understanding and interactive communication abilities. In addition, these models are widely accessible and increasingly applied in healthcare-related scenarios.^36,37 This study selected these models to evaluate the capability of general-purpose large language models in post-discharge tuberculosis management.

To prevent memory effects, each case prompt was entered in a new session, clearing prior conversation history to ensure independent responses. All data were collected by June 27, 2025.

Standardized prompts

To ensure consistent and comparable responses, the research team developed standardized prompts including core case information and patient concerns. Prompts were presented concisely and input separately to the LLMs and physicians for independent responses. All cases used the same standardized prompt: “Assume you are an experienced TB physician.” Participants answered three fixed questions on medication counseling, patient education, and follow-up planning. Prompts were reviewed by two TB clinicians for clarity and appropriateness. Full prompt content is provided in Table 4 (Appendix 1).

Response Collection and standardization

All selected cases were standardized and de-identified into the same anonymized vignettes for both LLMs and physicians. Direct or indirect identifiers, such as patient names and hospital registration numbers, were removed, and only task-relevant clinical information was retained. The LLMs were accessed via web-based interfaces, with only anonymized vignettes entered; no original records or patient identifiers were uploaded to the LLMs or provided to physicians. Both groups answered the same standardized case questions. Physicians completed written responses independently without communication. Before expert scoring, two researchers not involved in scoring standardized all responses to reduce source-recognition bias, removing source-revealing stylistic features without altering substantive clinical content.

Outcome measures

To comprehensively evaluate the performance of LLMs and physicians in three post-discharge TB management scenarios: medication counseling, patient education, and follow-up plan, we assessed both objective performance and subjective perception. Objective outcomes included accuracy and completeness, measured by precision and recall. Precision reflects the proportion of correct answers among those provided, while recall reflects the proportion of the gold-standard answers captured among all that should have been given. The gold-standard answers were developed through expert consensus. Three senior specialists (Associate Chief Physician or above) independently drafted answers based on the Chinese Tuberculosis Prevention and Control Technical Guidelines and their professional judgments. ³⁸ These drafts were then consolidated into a final version through joint review and discussion, with a fourth senior expert adjudicating any unresolved disagreements. To summarize overall performance, we also calculated the F1 score, which provides a single measure balancing precision and recall. Subjective perception was evaluated by three senior TB specialists, blinded to response sources across seven dimensions: applicability, information quality, conciseness, safety, understandability, operability, and empathy. In addition, text readability was assessed using a validated Chinese readability formula. Evaluation metrics and their scoring criteria are summarized in Appendix 1 (Table 6).

Expert rating

Three senior TB specialists (≥10 years’ experience, chief or associate chief physicians), independent of the physician group, formed the expert panel. They first discussed and agreed upon reference answers, which were used solely to guide scoring. Each expert independently rated all LLM and physician responses across seven predefined dimensions—applicability, information quality, conciseness, safety, understandability, operability, and empathy—using a 5-point Likert scale (1 = very poor, 5 = excellent). To minimize potential bias, evaluations were conducted in a double-blind manner with all responses standardized in formatting to conceal the source identities from evaluators. The final score for each response was calculated as the mean of the three experts’ ratings. Inter-rater consistency was assessed using the intraclass correlation coefficient (ICC).

Content analysis

Three clinical scenarios were prespecified: medication counseling, patient education, and follow-up planning. To identify the key information required in each context, three experts with over a decade of clinical experience extracted essential points from relevant materials in the Technical Guidelines for Tuberculosis Prevention and Control in China, the Guidelines for the Rational Use of Anti-Tuberculosis Drugs in Primary Care Settings, and the Health Education Prescription (2020 edition).^38–41 The expert panel then convened to reconcile and finalise the comprehensive set of information points for each scenario. Any discrepancies were adjudicated by a fourth expert. The finalised essential information points were categorised thematically (e.g., ‘rifampicin administered once daily’ was grouped under the theme ‘medication frequency’), resulting in a coding scheme. Each coding item included definitions for both error and omission types (Appendix 1 Table 7). Errors referred to content inconsistent with the scenario-specific reference answer, and omissions referred to required items not mentioned in the response.

Sentences were used as the unit of analysis. Multiple items in a sentence were coded separately. Repeated mentions of the same item were counted once. Items from different subcategories could co-occur. Two coders conducted the analysis independently, with discrepancies resolved by a third expert. Coding examples are provided in and summary counts of errors and omissions by respondent and scenario are provided in Appendix 1 Table 8.

Statistical analysis

SPSS v27.0, Python v3.13, and Origin Pro 2025 were used for analysis and visualization. Scores from three raters were averaged for each response. Because rating scores were non-normally distributed, quantitative outcomes are presented as medians with interquartile ranges (IQRs).

Between-group differences between the LLM and physician groups were assessed using linear mixed-effects models for medication counseling, patient education, and follow-up planning, with group as a fixed effect and answer generators and cases as random intercepts. Given the ordinal nature of Likert-type scores and the limited sample size, case-level Wilcoxon signed-rank tests comparing paired LLM and physician group scores were performed as robustness analyses. Sensitivity analyses restricted to the 10 real-world cases were conducted using the same mixed-effects modeling framework.

Inter-rater reliability was assessed using ICC based on a two-way random-effects model with absolute agreement. Error and omission counts were summarized descriptively. A two-sided p < 0.05 denoted statistical significance.

Ethical Considerations

This study was approved by the Medical Ethics Committee of Shenzhen Third People’s Hospital (Approval No. 2024-235). All case data were de-identified to protect patient privacy. Participating physicians provided written informed consent, and data were used solely for research purposes. As the LLMs are publicly available tools and do not involve human subjects, no additional ethical approval was required.

Result

Analysis of information accuracy and completeness

Table 1 presents the precision, recall, and F1 scores of the LLM and tuberculosis specialist groups across three post-discharge management tasks: medication counseling, patient education, and follow-up planning.

Table 1.

Comparison of response accuracy and completeness between LLM and physician groups across medication counseling, patient education, and follow-up care plan.

​	Group		β	95%CI	P value
	LLMs	Physicians
Medication counseling	​	​	​	​	​
Precision	0.63 (0.50-0.78)	0.50 (0.42-0.50)	0.129	-0.013-0.273	P= .060
Recall	0.34 (0.23-0.50)	0.05 (0.04-0.07)	0.289	-0.134-0.712	P= .099
F1 score	0.44 (0.33-0.53)	0.09 (0.08-0.12)	0.332	-0.041-0.705	P= .062
Patient Education
Precision	0.86 (0.71-1.00)	0.50 (0.46-0.67)	0.293	0.215-0.372	P<.001
Recall	0.53 (0.44-0.62)	0.14 (0.09-0.18)	0.427	0.127-0.728	P= .026
F1 score	0.65 (0.56-0.71)	0.22 (0.15-0.29)	0.434	0.181-0.688	P= .018
Follow-up plan
Precision	0.33 (0.25-0.50)	0.73 (0.60-0.75)	-0.308	-0.741-0.125	P= .092
Recall	0.23 (0.20-0.30)	0.18 (0.12-0.28)	0.089	-0.114-0.293	P= .200
F1 score	0.28 (0.22-0.35)	0.27 (0.14-0.30)	.005	-0.161-0.172	P=0.897

Note. Precision was used to evaluate the accuracy of responses, recall reflected their completeness, and the F1 score, defined as the harmonic mean of precision and recall, summarized overall performance. Values are presented as median (interquartile range). Between-group differences were estimated using linear mixed-effects models, with effect estimates (β), 95% confidence intervals, and P values reported.

For Patient Education, the LLM group had higher median precision, recall, and F1 scores than the physician group, and the between-group differences were statistically significant. (precision: 0.86 vs 0.50, β=0.293, 95%CI,0.215-0.372, P<.001; recall: 0.53 vs 0.14, β=0.427, 95%CI,0.127-0.728, P= .026; F1:0.65 vs 0.22, β=0.434, 95%CI,0.181-0.688, P= .018).

In the medication counseling and follow-up planning tasks, no significant between-group differences were observed in precision, recall, or F1 score between the LLM and physician groups (all P > 0.05), with detailed results shown in Table 1.

To further examine potential differences between the two models in the accuracy and completeness of their responses to post-discharge management tasks, we compared the precision, recall, and F1 score of DeepSeek-R1 and ChatGPT-4o. DeepSeek-R1 appeared to show greater content completeness, with both recall and F1 score higher than those of ChatGPT-4o, and the differences were statistically significant (recall: 0.53 vs 0.14, β= 0.427, 95% CI, 0.127-0.728, P = .026; F1 score: 0.65 vs 0.22, β = 0.434, 95% CI, 0.181-0.688, P = .018). ChatGPT-4o showed numerically higher precision than DeepSeek-R1, although the difference was not statistically significant (precision: 0.86 vs 0.50, β= 0.293, 95% CI, 0.215–0.372, P = .441). Detailed results are presented in Appendix 1 (Table 9).

Analysis of errors and omissions

LLMs and physicians displayed distinct patterns in errors and omissions. LLMs provided broader content coverage but with more factual and logical errors, whereas physicians’ responses were more accurate but less comprehensive. Specifically, LLMs produced 137 misdiagnosed or inaccurate items versus 36 for physicians, concentrated in medication counseling and follow-up plans, including dosage, frequency, and follow-up timing errors. Physicians’ errors were mainly unrelated to information rather than clinical inaccuracies ( Figures 1 and 2).

Figure 1.

Stacked bar Charts of error counts and type proportions for LLM and physicians across different tasks. The figure shows the total number of error message entries and the proportions of different error types for LLM and physicians across three tasks: (a) Medication Counseling, (b) Patient Education, and (c) Follow-up Care Plan. Each stacked bar represents the total number of error entries, with different colors representing different error types.

Figure 2.

Stacked bar Charts of omission message counts and type proportions for LLM and physicians across different tasks. The figure shows the total number of omission messages and the proportions of different omission message types for LLM and physicians across three tasks: (a) Medication Counseling, (b) Patient Education, and (c) Follow-up Care Plan. Each stacked bar represents the total number of omission messages, with different colors representing different omission message types.

Regarding omissions, physicians missed more key information overall (699 items) compared to LLMs (462 items). LLMs’ omissions were mostly in dosage and frequency, while physicians additionally omitted content in adverse reaction management. In the patient education and follow-up plan, LLMs missed 175 and 252 items, respectively, versus physicians’ 314 and 403 items, indicating physicians’ coverage was less complete. LLMs thus offered a comprehensive framework with lower detail accuracy, while physicians’ responses were precise but limited in scope.

Subjective information perception

Three senior TB experts independently rated all responses on applicability, safety, understandability, operability, conciseness, empathy, and overall information quality. Inter-rater consistency was good (ICC = 0.62; 95%CI, 0.58-0.65; P < .001).

For medication counseling, the LLM group had higher median ratings than the physician group in Suitability, Conciseness, Understandability, Empathy, Subjective information quality, and Operability, and the between-group differences were statistically significant (Suitability: 4.00 vs 2.67, β=1.520, 95%CI, 0.802-2.237, P=.012; Conciseness: 4.33 vs 3.00, β=1.059, 95%CI, 0.582-1.536, P=.011; Understandability: 3.67 vs 2.67, β=0.872, 95%CI, 0.080-1.665, P=.042; Empathy: 3.00 vs1.33, β=1.676, 95%CI, 0.612-2.741, P=.021; Subjective information quality: 3.33 vs1.67, β=1.676, 95%CI, 0.262-3.091, P=.036; Operability: 4.00 vs 3.67, β=1.098, 95%CI,0.418-1.778, P=.020). No significant between-group difference was observed for Safety (P > 0.05).

For patient education, the LLM group had higher median ratings than the physician group in Suitability, Conciseness, Understandability, Empathy, and Operability, with statistically significant between-group differences (Suitability: 4.33 vs 2.67, β=1.569, 95%CI, 0.804-2.333, P=.013; Conciseness:4.33 vs 3.00, β=1.088, 95%CI, 0.118-2.058, P=.040; Understandability: 4.00 vs 2.67, β=1.098, 95%CI, 0.155-2.041, P=.038; Empathy: 2.67 vs 1.33, β=1.264, 95%CI, 0.038-2.491, P=.047; Operability: 4.00 vs 2.83, β=1.254, 95%CI, 0.527-1.983, P=.018). No significant between-group differences were observed for Safety or Subjective information quality (all P > 0.05).

For follow-up planning, the LLM group had higher median ratings than the physician group in Suitability, Conciseness, Understandability, and Empathy, and the between-group differences were statistically significant (Suitability: 4.00 vs 2.67, β=1.461, 95%CI, 0.591-2.330, P=.019; Conciseness: 4.33 vs 3.00, β=1.020, 95%CI, 0.076-1.963, P=.043; Understandability: 3.67 vs 2.67, β=1.000, 95%CI, 0.156-1.844, P=.036; Empathy: 3.00 vs 1.67, β=1.578, 95%CI, 0.709-2.448, P=.016). No significant between-group differences were observed for Safety, Subjective information quality, or Operability (all P > 0.05). Detailed results for each subjective dimension are provided in Supplementary Table 10 and Figure 3.

Figure 3.

Seven-Dimensional Comparison of subjective perception of information between LLM and physicians. The figure shows median scores of subjective perception of information across seven dimensions for the LLM and physician groups in three tasks (a: Medication Counseling; b: Patient Education; c: Follow-up Care Plan). Error bars represent the interquartile range. Asterisks (*) indicate statistically significant between-group differences at the two-sided 0.05 level (P < 0.05).

Readability analysis

No statistically significant between-group differences were observed in Readability scores between the LLM and physician groups across the medication counseling, patient education, and follow-up planning tasks (all P > 0.05). Detailed results are provided in Supplementary Table 10 and Figure 4.

Figure 4.

Comparison of readability scores between LLM and physicians across tasks. The figure shows median readability scores of LLM and physicians across three tasks (Medication Counseling, Patient Education, Follow-up Care Plan), with two-sided 0.05 significance indicated (P < 0.05).

Robustness and sensitivity analysis

Case-level Wilcoxon signed-rank tests yielded results consistent with the linear mixed-effects model analysis, supporting the robustness of the main findings (Appendix 1, Table 11). Sensitivity analysis restricted to the 10 real-world cases showed higher LLM scores than physician scores in accuracy, completeness, readability, and subjective information perception, with effect directions consistent with the main analysis, although statistical significance changed for several indicators. Specifically, in medication counseling, precision and readability scores were higher in the LLM group, with statistically significant between-group differences (P< .001 and P=.047); in patient education, recall scores were higher in the LLM group, but the between-group difference was not statistically significant (P=.060); and in follow-up planning, safety scores were higher in the LLM group, with a statistically significant between-group difference (P=.025). Details are provided in Appendix 1 (Table 12).

Discussion

The main findings of this exploratory study are as follows. (1) Using recall, precision, and F1, we observed a between-group difference that reached the statistical significance threshold only for the health education task, but not for the other two tasks. (2) Expert ratings were higher for LLMs on appropriateness, conciseness, understandability, and empathy, while no differences were observed for safety or formula-based readability. (3) LLMs showed broader coverage but more inaccurate items than physicians (137 vs 36), mainly involving dose/frequency, adverse-effect management, and follow-up timing. These findings suggest the need for professional oversight and larger validation studies.

Several factors may explain these task-dependent differences. First, LLMs are optimized to produce fluent, structured, and often exhaustive text; when the reference standard is itemized, this tendency can improve recall but also increases the risk of false positives and internally inconsistent recommendations. ⁴² Second, LLMs do not inherently have access to patient-specific longitudinal data, local protocols, or validated pharmacologic calculators, which can increase hallucinations and reduce precision in regimen- or timing-sensitive decisions. ⁴³ Third, clinicians may prioritize brevity and clinically salient points (rather than a complete counseling script), especially in vignette-based tasks, which can reduce recall under checklist-based scoring.

In medication counseling, no statistically significant differences were observed between LLMs and physician-generated responses in objective completeness or accuracy based on recall, precision, and F1. Despite comparable aggregate metrics, qualitative review and error profiling suggested distinct patterns. LLM outputs tended to be more expansive and itemized but more often included content misaligned with the question focus or reference standard, with inaccuracies clustering in regimen-critical details such as dosage and administration. Physician responses contained fewer inaccuracies overall but more frequent omissions, particularly in adverse-reaction management. This pattern is consistent with prior studies describing broad drug-information coverage alongside limited pharmacologic logic validation and hallucination risk.^44–46 Given these complementary tendencies, a workflow may be explored under strict professional verification and risk control, in which LLMs draft counseling scripts and topic checklists, key regimen-specific details such as drug, dose, and frequency are verified before use, and structured inputs, guideline retrieval, and automated dose-range checks are applied. Given persistent risks in safety-critical details and the small exploratory design, unsupervised use for patient counseling is not recommended, and larger prospective validation is warranted.

In patient education, LLMs surpassed physicians in precision, recall, F1 scores, and expert ratings for understandability, conciseness, and empathy. This aligns with recent studies reporting that LLMs enhance health literacy materials and patient communication, improving engagement and accessibility. ⁴⁷ LLM-generated content more closely matched patients’ language habits, making it easier for non-specialist audiences to understand, as supported by readability scores.^48,49 Although physicians’ explanations were generally accurate, they tended to be more technical and less patient-centered. In contrast, LLMs may help reduce variability in educational quality arising from differences in physicians’ communication styles and offer more consistent and comprehensible support, particularly in resource-limited settings.^50,51 Looking forward, optimizing LLMs through personalization to patient backgrounds may further improve adherence and equity in education delivery. For example, LLMs draft personalized patient materials, reviewed by physicians to ensure accuracy and relevance, enhancing communication and adherence.

In follow-up planning, LLMs and physicians exhibited no statistically significant differences in accuracy or completeness. Qualitative analysis revealed that LLM plans deviated more in timing and task selection—such as inappropriate intervals or omissions—whereas physician plans were narrower yet generally accurate. These observations mirror recent evidence highlighting the limited reliability of LLMs in clinical reasoning and individualized decision-making. ⁵² Such deviations could delay necessary monitoring or degrade complication management, underscoring that LLMs should not function as independent decision-makers. ⁵³ Instead, they should serve as a reference framework to minimize omissions, particularly in resource-limited settings. Future research should explore human–AI collaborative models that integrate the comprehensive coverage of LLMs with physician precision. For instance, LLMs can draft initial plans for physician refinement to ensure clinical accuracy and patient-specific relevance.

After restricting the analysis to real-world cases, the LLM group still showed higher scores than the physician group in accuracy, completeness, readability, and subjective information perception, suggesting that the effect directions of the main analysis were relatively robust. Although statistical significance changed for some indicators, the effect directions remained unchanged. This variation may be related to the reduced sample size and greater heterogeneity of real-world cases. Therefore, the overall trend of higher LLM scores appeared relatively stable, whereas task-specific differences should be interpreted cautiously and validated in larger real-world datasets.

Finally, the global health value proposition of LLM-enabled support is greatest where TB burden is high, and health workforce capacity is limited, but responsible deployment requires governance, privacy protection, and equity-by-design. International guidance emphasizes that AI for health should protect human rights, ensure transparency and accountability, and prevent harm, and the WHO digital health strategy highlights the need for inclusive and sustainable digital health systems that do not widen existing inequities.^54,55 Recent work has underscored that LLM deployment can inadvertently exacerbate disparities through differential access, digital literacy barriers, and sociodemographic biases in model outputs.^56,57 At the same time, LLMs may help improve equitable access to understandable health information when embedded in supervised workflows and evaluated across languages and health-literacy levels. ⁵⁸ Therefore, we recommend a cautious, tiered human–AI collaborative approach and explicitly discourage unsupervised patient-only use of general-purpose chatbots for medication dosing or follow-up scheduling.

Limitations

Several limitations should be acknowledged. First, this single-center study included a limited number of 17 cases (7 simulated and 10 real), which may limit generalizability to more heterogeneous clinical scenarios. Although robustness and sensitivity analyses generally supported the direction of the main findings, changes in statistical significance across some indicators suggest that the results should be interpreted as exploratory. Future multi-center studies with larger real-world case sets are needed to validate these findings. Second, although expert scoring showed good consistency, subjectivity and evaluator bias cannot be excluded, which may affect assessment robustness. Future studies should incorporate more objective or multi-rater evaluation strategies. Third, this study did not include domain-specific medical or biomedical large language models, which may perform better in clinically specialized tasks and may lead to a conservative estimate of model capability. Future work should systematically compare general-purpose and medical-domain models within the same task framework. Finally, the use of a standardized single-prompt strategy may not fully capture model output variability or prompt sensitivity, potentially underestimating optimal performance. Future studies should explore structured and advanced prompting strategies to improve robustness and accuracy. In addition, the evaluation was text-based, and future research should incorporate multimodal models to improve real-world clinical applicability.

Conclusion

This study suggests the potential of LLMs as auxiliary tools in TB post-discharge management, particularly in patient education. While LLMs and physicians exhibited comparable quantitative performance in medication counseling and follow-up planning, their complementary qualitative patterns support the feasibility of a hybrid human–AI approach. These findings provide preliminary evidence for integrating LLMs into clinical workflows to enhance efficiency in resource-limited settings. Future large-scale, multi-center real-world studies are essential to validate these observations and ensure sustained clinical reliability.

Appendix.

Abbreviations

LLM

Large Language Model

TB

Tuberculosis

ORCID iDs

Jin Liao https://orcid.org/0009-0009-3964-9919

Lanping Zhang https://orcid.org/0000-0003-2718-1748

Ethical considerations

This study was approved by the Medical Ethics Committee of Shenzhen Third People’ s Hospital (Approval No. 2024-235). As this study used fully de-identified clinical data and did not involve direct participation of patients, the ethics committee approved a waiver of informed consent. All participating physicians provided written informed consent prior to participation in the study. This study did not involve a clinical trial; therefore, clinical trial registration was not applicable. Clinical trial number: not applicable.

Data Availability Statement

The datasets generated or analyzed during this study are available from the corresponding author on reasonable request.*