Generating guideline-concordant and safe recommendations for diabetic kidney disease management via a hierarchical retrieval-augmented large language model

Abstract

Managing diabetic kidney disease (DKD) is inherently complex, requiring clinicians to synthesize patient history, fluctuating biomarkers, and evolving treatment guidelines. While large language models (LLMs) show promise in medical decision support, their clinical adoption is hindered by factual inaccuracies and a lack of specific reasoning required for individualized patient management. To address this, we developed a hierarchical multi-agent system that integrates a locally deployed retrieval-augmented generation (RAG) framework with a cloud-based advanced reasoning engine, grounding responses in a curated corpus of clinical guidelines. We conducted a multi-center retrospective validation using 267 patient cases. The system’s performance was evaluated against baseline models through a blinded review by twelve independent physicians across clinical dimensions including accuracy, safety, and factuality. Our evaluation reveals that the RAG-enhanced system significantly outperforms unaugmented models in providing accurate, guideline-compliant recommendations. Notably, it substantially reduced safety-critical errors, particularly in identifying medication contraindications related to renal function stages, while achieving high inter-rater reliability. This study demonstrates that anchoring LLMs with authoritative knowledge effectively mitigates hallucination risks and enhances clinical reliability. The proposed framework functions as a reliable on-demand assistant for DKD management, providing guideline-grounded decision support for primary care providers.

Introduction

Diabetic Kidney Disease (DKD) remains a predominant microvascular complication of diabetes and has emerged as the primary etiology of end-stage renal disease (ESRD) globally¹. Recent projections from the International Diabetes Federation indicate that the global diabetic population will reach 783 million by 2045, with approximately 20% to 40% of these patients expected to develop DKD². This escalating prevalence imposes a formidable burden on healthcare infrastructures, necessitating more effective strategies for early intervention and long-term management.

Despite its clinical significance, the management of DKD is hindered by several critical bottlenecks. Early screening remains suboptimal, as conventional biomarkers such as urinary albumin-to-creatinine ratio (UACR) and estimated glomerular filtration rate (eGFR) exhibit inherent limitations in sensitivity and specificity^3–5. In the nascent stages of renal impairment, eGFR often fails to reflect subtle histopathological changes, leading to diagnostic inertia and the loss of optimal therapeutic windows. Furthermore, the definitive diagnosis of DKD is a complex process involving the exclusion of non-diabetic kidney diseases (NDKD), often requiring invasive renal biopsies that suffer from low patient acceptance⁶. Even when diagnosed, therapeutic strategies frequently default to a “one-size-fits-all” approach, lacking the precision required to integrate multidimensional patient data—such as genetic predispositions and metabolic profiles—into individualized care plans^7–10.

The rapid evolution of Artificial Intelligence (AI), particularly Large Language Models (LLMs), has introduced transformative potential for addressing these challenges^11,12. LLMs excel in synthesizing vast arrays of clinical text and facilitating complex differential diagnoses¹³. However, the deployment of general-purpose LLMs in specialty domains like nephrology is often compromised by “hallucinations”, knowledge obsolescence, and a discernible lack of structured clinical reasoning^14,15. To bridge this gap, Retrieval-Augmented Generation (RAG) has emerged as a robust framework that anchors LLM outputs in authoritative, external evidence, thereby enhancing the factuality and traceability of generated medical advice^16–18. Recent implementations, such as the LiVersa system for hepatology and RAG-enhanced models for cardiovascular prevention, have demonstrated that domain-specific grounding can significantly outperform off-the-shelf models in generating guideline-consistent answers^19,20.

Building upon these advancements, we developed and validated the RAG-LLM-based DKD assistant system. This system integrates a high-fidelity knowledge base—comprising the latest clinical guidelines, expert consensuses, and pharmacological databases—with advanced prompt engineering and Chain-of-Thought (CoT) reasoning. By mimicking the deductive logic of specialist physicians, the system is designed to perform multidimensional risk stratification, generate differential diagnostic insights, and provide evidence-based, individualized treatment recommendations.

In this study, we retrospectively enrolled 267 patients with DKD from three tertiary hospitals to evaluate the system’s performance. A multi-stage validation strategy was employed, encompassing a technical benchmarking of varied RAG-LLM configurations followed by a double-blinded clinical assessment conducted by twelve physicians. The evaluation focused on seven specific clinical dimensions: Accuracy, Comprehensiveness, Factuality, Response Efficiency, Safety, Further Diagnostic Tests, and Diagnosis. Our objective is to provide a scalable and reliable methodology—“Knowledge Base + RAG + Expert Evaluation”—for the practical implementation of AI in specialized disease domains, ultimately facilitating a transition from experience-based to data-driven, guideline-concordant nephrology.

Results

Implementation and experimental environment

The system was deployed in a secure high-performance computing environment aimed at simulating a hospital intranet. Local inference tasks were executed on a workstation equipped with NVIDIA A100 (80GB VRAM) GPUs to ensure low-latency processing ( < 100 ms). The backend orchestration was implemented using Python 3.9 and the FastGPT framework. For the RAG module, Milvus (v2.x) and PostgreSQL were employed for vector and structured data management, respectively. To ensure reproducibility, the temperature parameter for all LLMs was fixed at 0.1. The retrieval system was configured to fetch the top-5 most relevant knowledge chunks (k = 5) based on a weighted hybrid score. Statistical analyses were performed using Python libraries SciPy and Statsmodels.

System interface and comparative case analysis

The system features a clinician-centric interface designed to enhance interpretability. As shown in Fig. 1, the interface juxtaposes the chat window with a “Reference Context” sidebar. A critical feature distinguishing this system from standard LLMs is the inclusion of clickable citation markers. These markers link directly to source documents—such as the 2024 KDIGO guidelines or specific Chinese expert consensuses—addressing the “black box” limitation by allowing physicians to immediately verify the evidentiary basis of AI recommendations.

Fig. 1. Interface of the retrieval-augmented multi-agent system.

The clinician-facing interface integrates a conversational window with a dynamic reference sidebar. The system provides real-time citation markers within the generated response, allowing users to verify evidence sources such as clinical guidelines or research papers via clickable links.

To qualitatively evaluate clinical reasoning, we analyzed a representative case of advanced DKD (Fig. 2a). The patient, a 58-year-old male, presented with massive proteinuria (3.2 g/24 h) and an eGFR of 45 ml/min/1.73 m², indicating CKD Stage G3b. This clinical scenario requires careful medication adjustment, particularly regarding Metformin dose reduction (per KDIGO: reduce dose at eGFR 30–45 ml/min/1.73 m², discontinue at eGFR <30 ml/min/1.73 m²) and SGLT2 inhibitor initiation with monitoring.

Fig. 2. Comparative analysis of model outputs for a representative case of advanced Diabetic Kidney Disease (DKD).

a Clinical input data. b Output from the proposed Multi-Agent System, which accurately identifies the eGFR-based metformin dose-adjustment threshold and provides explicit, verifiable guideline citations. c Baseline output from DeepSeek-Base. d Baseline output from GPT-4o. e Baseline output from GPT-o3. Notably, the unaugmented baselines fail to flag specific safety-critical contraindications or lack direct traceability, highlighting the clinical advantage of the retrieval-augmented architecture.

Proposed Multi-Agent System (Fig. 2b): The system successfully synthesized the patient’s data with retrieved guidelines. It explicitly cited the eGFR-based dose-adjustment thresholds, recommending a reduction in Metformin and the initiation of Dapagliflozin with potassium monitoring. This response demonstrated a balance of safety, specificity, and evidential transparency.

DeepSeek-Base (Fig. 2c): The standalone model correctly identified the disease category but failed to reference specific guidelines. More critically, it did not explicitly flag eGFR-based metformin dose adjustment around the 45 ml/min/1.73 m² threshold, presenting a potential safety risk.

GPT-4o (Fig. 2d): While providing a comprehensive management plan, the output lacked direct integration with local hospital formularies or specific citation links, limiting its immediate verification utility in a clinical setting.

GPT-o3 (Fig. 2e): This model exhibited the deepest pathophysiological reasoning (e.g., explaining the mechanism of “foamy urine”), yet its significantly higher latency and operational cost make it less suitable for routine, high-frequency clinical inquiries compared to the proposed hierarchical architecture.

Multidimensional performance assessment and comparative analysis

To rigorously evaluate the clinical utility of the proposed hierarchical multi-agent system, we conducted a comparative analysis against three baseline models: the unaugmented local foundation model (DeepSeek-Base), a generalized state-of-the-art model (GPT-4o), and an advanced reasoning model (GPT-o3). The comparative results across all seven predefined clinical dimensions are summarized in Table 1.

Table 1.

Multidimensional performance comparison of the evaluated models

Evaluation dimension	DeepSeek-base	GPT-4o	GPT-o3	Proposed system	LMM P-value
Accuracy	3.46 ± 0.88	4.13 ± 0.71	4.48 ± 0.54	4.38 ± 0.69	<0.001
Comprehensiveness	3.32 ± 0.84	4.28 ± 0.66	4.42 ± 0.61	4.25 ± 0.73	<0.001
Factuality	3.18 ± 0.92	3.94 ± 0.79	4.21 ± 0.68	4.52 ± 0.64	<0.001
Response efficiency	4.16 ± 0.63	3.82 ± 0.86	3.67 ± 0.91	4.55 ± 0.57	<0.001
Safety (Critical)	3.21 ± 1.04	4.02 ± 0.77	4.31 ± 0.66	4.49 ± 0.62	<0.001
Further diagnostic tests	3.54 ± 0.81	4.08 ± 0.72	4.34 ± 0.59	4.29 ± 0.67	<0.001
Diagnosis	3.58 ± 0.76	4.17 ± 0.65	4.41 ± 0.59	4.29 ± 0.71	<0.001

The quantitative assessment revealed that the proposed system significantly outperformed the unaugmented DeepSeek-Base and the generalized GPT-4o models across most evaluation metrics (LMM, all P < 0.001). Notably, the linear mixed-effects model analysis, which accounted for the crossed random effects of case and rater, confirmed that all pairwise differences reported below remained statistically significant under the more rigorous repeated-measures framework (LMM-based P values are reported as the primary results). Specifically, relying solely on the foundational model without external knowledge retrieval (DeepSeek-Base) yielded suboptimal clinical reliability, particularly in Factuality (3.18 ± 0.92) and Safety (3.21 ± 1.04). The integration of the hybrid RAG mechanism in our proposed system successfully bridged this gap, elevating the Factuality and Safety scores to 4.52 ± 0.64 (LMM pairwise P < 0.001, Cohen’s d = 1.68) and 4.49 ± 0.62 (LMM pairwise P < 0.001, Cohen’s d = 1.47), respectively, indicating exceptionally large effect sizes.

Furthermore, while GPT-4o demonstrated robust generalized medical knowledge, its outputs frequently lacked adherence to specific localized guidelines and dynamic eGFR-based dosing adjustments, resulting in a moderate Safety score of 4.02 ± 0.77. The proposed system significantly outperformed GPT-4o in this critical safety dimension (pairwise P < 0.001, Cohen’s d = 0.67). Similarly, for Diagnosis, the proposed system (4.29 ± 0.71) showed a modest but statistically significant improvement over GPT-4o (4.17 ± 0.65; exact pairwise P = 0.043, Cohen’s d = 0.18), further confirming the clinical value of domain-specific guideline grounding. GPT-o3 exhibited exceptional diagnostic and comprehensive reasoning capabilities (Diagnosis: 4.41 ± 0.59); however, its extensive chain-of-thought generation process compromised clinical usability in time-constrained settings, reflected by a notably lower Response Efficiency score (3.67 ± 0.91) and a significantly longer average output length of 847 words. Conversely, by dynamically routing queries, the proposed hierarchical system achieved reasoning depths comparable to GPT-o3 in complex scenarios while maintaining a superior Response Efficiency (4.55 ± 0.57) with a highly concise average output length of 314 words, thereby offering a more balanced and practical solution for routine diabetic kidney disease management.

The blinded assessment by 12 independent physicians on 267 retrospective cases revealed robust system performance. As illustrated in the heatmap (Fig. 3a), the system achieved high mean scores across all seven evaluation dimensions, with no significant “cold spots” (low scores) observed across different evaluators. Factuality (Mean: 4.55) and Response Efficiency (Mean: 4.52) emerged as the strongest domains, suggesting that the RAG architecture effectively grounds the model in medical evidence without compromising response speed.

Fig. 3. Multidimensional performance assessment of the Multi-Agent System by blinded physician evaluators (N = 12).

a Heatmap illustrating the mean scores assigned by each of the 12 evaluators across seven clinical dimensions, demonstrating consistently high performance without significant ‘cold spots.’ b Histogram showing the frequency distribution of scores in the Safety dimension. The red dashed line indicates the mean safety score, reflecting a strongly left-skewed distribution indicative of robust clinical error mitigation. c Stacked bar chart displaying the proportion of high scores (4–5, blue) versus low scores (1–2, red) across all dimensions, confirming that over 90% of the system’s outputs met stringent clinical standards. d Pearson correlation analysis between Accuracy and other evaluation dimensions, revealing a strong positive association between factual evidence grounding and diagnostic precision. Assessment was performed using a 5-point Likert scale.

Safety, a critical metric for medical AI, showed a strictly left-skewed distribution (Fig. 3b), with the vast majority of responses rated as 4 or 5. The mean safety score of 4.49 indicates that the system consistently avoided harmful recommendations, such as prescribing contraindicated medications in patients with reduced renal function. The distribution analysis (Fig. 3c) further quantifies this, showing that the proportion of high-quality responses (scores 4–5) exceeded 90% across all dimensions. Notably, the ratio of low scores (1–2) was negligible (<1.5%), implying a high degree of risk mitigation.

Beyond the subjective Likert evaluations, the objective audit of safety-critical errors provided a more granular measure of clinical reliability. Among the 267 evaluated cases, 47 patients presented with an eGFR at or below 45 ml/min/1.73 m², requiring metformin dose adjustment or discontinuation per KDIGO guidelines. The unaugmented DeepSeek-Base model committed safety-critical prescribing errors in 18 of these 47 cases (38.3%), whereas GPT-4o missed the eGFR threshold in 7 instances (14.9%). Conversely, the proposed DeepSeek-RAG system, anchored by explicit guideline retrieval, recorded only 1 such omission (2.1%). A concrete illustration of this safety mitigation is demonstrated in Fig. 2b, where the proposed system accurately identified the eGFR-based metformin dose-adjustment threshold and explicitly recommended dose reduction, a nuance entirely missed by the unaugmented baseline (Fig. 2c).

Furthermore, regarding the initiation of dual RAAS and MRA therapy—a scenario present in 61 patient cases—the unaugmented baseline failed to provide requisite hyperkalemia monitoring advice in 23 cases (37.7%). The proposed hierarchical framework reduced this critical monitoring omission to merely 2 cases (3.3%). These objective frequency reductions substantiate that domain-specific knowledge injection directly mitigates factual hallucinations, preventing high-risk pharmacological oversights that standalone generalist models frequently commit. To further assess safety performance across renal function strata, we conducted an eGFR-stratified subgroup analysis of Safety scores by CKD stage. Among the 267 evaluated cases, the proposed system maintained consistently high Safety scores across all stages: CKD Stage 1–2 (n = 87, Safety: 4.58 ± 0.51), CKD Stage 3 (n = 97, Safety: 4.51 ± 0.58), and CKD Stage 4–5 (n = 83, Safety: 4.37 ± 0.72; vs. DeepSeek-Base: 2.89 ± 1.13, GPT-4o: 3.76 ± 0.88). The modest decline in Stage 4–5 reflects the inherently greater pharmacological complexity of advanced DKD, where multi-drug regimens and narrower therapeutic windows increase the probability of borderline recommendations. Nevertheless, the absolute Safety score remained above 4.3 even in this most challenging subgroup, confirming that the RAG-grounded system retains clinical reliability across the full spectrum of renal impairment.

The correlation analysis (Fig. 3d) highlighted a moderate-to-strong positive correlation between Accuracy and Factuality (r = 0.545). This statistical relationship supports the hypothesis that retrieval-augmented grounding directly contributes to diagnostic accuracy by minimizing hallucinations.

Routing distribution and ablation analysis

A critical component of the hierarchical framework is its ability to stratify queries. Retrospective analysis of the 267 validation cases revealed that the majority of clinical inquiries were effectively handled within the localized intranet environment. Specifically, 31 cases (11.61%) were classified as foundational and routed to Tier 1, while 203 cases (76.03%) required guideline verification and were successfully processed by the locally deployed DeepSeek-RAG agent (Tier 2). Only the remaining 33 cases (12.36%), which involved highly complex multi-morbidity presentations, triggered the criteria for escalation to the cloud-based GPT-o3 reasoning engine (Tier 3).

To rigorously assess the reliability of this referral mechanism, we conducted a retrospective audit of the routing accuracy across the 267 validation cases. The audit revealed an overall referral error rate of 3.37% (9 out of 267 cases). Among these, 6 instances were characterized as over-triage, where standard cases were unnecessarily escalated to the Tier 3 agent. This resulted in an increased computational latency but did not compromise the clinical accuracy of the final recommendations. The remaining 3 instances involved under-triage, where highly complex, multi-morbidity cases were erroneously processed by the localized Tier 2 agent. Notably, while the Tier 2 agent lacked the exhaustive multi-disciplinary pathophysiological reasoning depth provided by GPT-o3 for these specific outliers, it consistently defaulted to conservative, guideline-concordant baseline recommendations, thereby averting any safety-critical prescribing errors despite the misclassification.

To isolate the performance contributions of the retrieval-augmented grounding versus the dynamic routing mechanism, we conducted an ablation study focusing on two primary clinical dimensions: Accuracy and Safety. As detailed in Table 2, we evaluated the standalone foundational model (DeepSeek-Base), a pure localized RAG configuration where all 267 cases were forced through Tier 2 without routing (Pure DeepSeek-RAG), and the fully Proposed Hierarchical System.

Table 2.

Ablation analysis of the architectural components

System Configuration	Routing	RAG	Accuracy	Comprehensiveness	Factuality	Response Efficiency	Safety	Further Diagnostic	Diagnosis
DeepSeek-Base	No	No	3.46 ± 0.88	3.32 ± 0.84	3.18 ± 0.92	4.16 ± 0.63	3.21 ± 1.04	3.54 ± 0.81	3.58 ± 0.76
Pure DeepSeek-RAG (Tier 2 only)	No	Yes	4.29 ± 0.72	4.13 ± 0.76	4.47 ± 0.68	4.48 ± 0.59	4.41 ± 0.65	4.21 ± 0.71	4.18 ± 0.74
Proposed Hierarchical System	Yes	Yes (Tier 2)	4.38 ± 0.69	4.25 ± 0.73	4.52 ± 0.64	4.55 ± 0.57	4.49 ± 0.62	4.29 ± 0.67	4.29 ± 0.71

The results indicate that the integration of localized retrieval augmentation alone yielded a substantial performance leap over the unaugmented baseline (Accuracy: 4.29 ± 0.72 vs. 3.46 ± 0.88; Safety: 4.41 ± 0.65 vs. 3.21 ± 1.04). The addition of the dynamic routing module (the Proposed System) further refined these outcomes (Accuracy: 4.38 ± 0.69; Safety: 4.49 ± 0.62). This ablation confirms that the core clinical reliability of the proposed framework is primarily driven by the domain-specific grounding of the Tier 2 RAG agent, while the escalation to GPT-o3 serves as a targeted safeguard for a minority of highly convoluted clinical scenarios, thereby justifying the hybrid design.

Inter-rater reliability and consensus analysis

We analyzed scoring patterns to ensure that the high performance was not an artifact of specific evaluator leniency. The inter-evaluator correlation matrix (Fig. 4a) demonstrates positive correlations across the panel, yielding an overall Intraclass Correlation Coefficient (ICC) of 0.863 (95% CI: 0.821–0.894). This indicates a strong general consensus on response quality despite varying levels of seniority (from residents to chief physicians).

Fig. 4. Analysis of inter-rater reliability and scoring consistency.

a Correlation matrix heatmap showing pairwise Pearson correlation coefficients between the 12 evaluators, indicating broad consensus across varying levels of clinical seniority. b Ranking of evaluators based on the mean overall score assigned to the system. The red dashed line represents the global mean score across all evaluators. c Violin plots displaying the probability density of score distributions for each evaluator. Horizontal lines within the violins represent the median and interquartile ranges, showing a consistently top-heavy distribution across all raters. d Bar chart of the Coefficient of Variation (CV) for each evaluator, ranked by scoring variability. The uniformly low CV values underscore the high reproducibility and robustness of the system’s clinical utility.

To further quantify scoring stringency, we examined the mean ratings assigned by each physician (Fig. 4b). The global mean score was 4.39 (red dashed line). Although individual baselines varied—ranging from the most lenient (Evaluator 9) to the most stringent (Evaluator 10)—it is notable that even the lowest mean score approached 4.0. This indicates that the system’s outputs met clinical standards even under the scrutiny of the most rigorous evaluators, reinforcing that the high overall performance is not driven solely by a subset of lenient raters.

The violin plots (Fig. 4c) reveal the density distribution of scores for each evaluator. A consistent “top-heavy” shape is observed across all 12 participants, indicating that even stricter evaluators (e.g., Evaluator 10) predominantly assigned scores in the upper range. This consistency is quantitatively confirmed by the Coefficient of Variation (CV) analysis (Fig. 4d). The CV for overall scores remained low (<0.14) for all evaluators, with most falling below 0.10. This low variability suggests that the system’s performance is robust and reproducible, eliciting consistent approval from clinicians regardless of their individual experience levels or subjective scoring baselines.

Discussion

This study demonstrates that a hierarchical multi-agent system, integrating RAG with dynamic task routing, effectively reconciles the dichotomy between diagnostic precision and operational efficiency in the management of DKD. Our findings align with the emerging paradigm shift towards multi-agent systems (MAS) in clinical decision support, where specialized agents collaborate to outperform monolithic models in complex reasoning tasks²¹. Generalist large language models often struggle with the stringent numerical accuracy required for nephrology. In contrast, our tiered architecture combines a locally deployed DeepSeek-v3.2 model with a cloud-based GPT-o3 reasoning engine. This hybrid approach achieved high physician-rated performance across the evaluated retrospective clinical scenarios. A principal finding of this work is that the integration of a hybrid RAG mechanism significantly reduced safety-critical errors, particularly in identifying contraindications for renin-angiotensin-aldosterone system (RAAS) inhibitors and SGLT2 inhibitors based on dynamic eGFR thresholds. These results validate the hypothesis that domain-specific knowledge injection, rather than mere parameter scaling, is the primary determinant of clinical reliability in specialized medical AI systems²². Furthermore, comparative analyses in this study corroborate recent evidence suggesting that RAG significantly outperforms standalone fine-tuning in minimizing medical hallucinations, particularly when navigating rapidly evolving clinical guidelines²³.

The superior performance of the proposed framework over baseline models highlights the critical role of decoupling knowledge retrieval from logical reasoning in specialized medical AI. General-purpose LLMs frequently conflate outdated protocols with current standards. This conflation often leads to dangerous “hallucinations” regarding quantitative cut-offs. However, our results indicate that anchoring the generation process in a curated evidence base effectively mitigates this clinical risk. The strong alignment observed between “Factuality” and “Safety” scores suggests that the hybrid retrieval strategy successfully captured the pharmacological nuances essential for DKD management, such as precise eGFR thresholds for metformin dose adjustment or discontinuation, which purely semantic searches might miss²⁴. Furthermore, the explicability afforded by CoT prompting did not merely improve diagnostic accuracy but aligned the AI’s reasoning trajectory with human clinical logic. This structural transparency proved decisive in fostering physician trust, as evidenced by the system’s robustness against the scrutiny of senior specialists.

A comparative analysis of the validation results reveals critical insights regarding the cost-benefit trade-off in medical AI deployment. While the cloud-based GPT-o3 agent achieved the highest scores in handling complex, multi-morbidity cases, the locally deployed DeepSeek-RAG agent demonstrated comparable performance in over 90% of routine clinical queries. This finding nuances the prevailing assumption that larger models are invariably superior for all medical tasks. Instead, it supports the growing consensus regarding Small Language Models (SLMs) in clinical settings. When grounded in high-quality external knowledge, SLMs can achieve “sufficient clinical intelligence” for specific medical domains. Furthermore, this localized deployment inherently ensures patient data privacy and significantly reduces operational costs²⁵. The strong inter-rater reliability (ICC = 0.863, 95% CI: 0.821–0.894) observed across residents and chief physicians further underscores the system’s robustness; even rigorous evaluators acknowledged that the RAG-enhanced outputs consistently met the safety standards required for clinical practice. This stability implies that the hierarchical architecture can reliably supply guideline-grounded recommendations, offering a consistent reference that may help mitigate the variability often inherent in human clinical judgment across different levels of seniority²⁶. A direct comparative assessment of RAG versus standalone LLM approaches reveals distinct trade-offs. Standalone LLMs offer the advantage of holistic reasoning across diverse medical domains without retrieval latency, yet they are fundamentally limited by static parametric knowledge that becomes outdated as guidelines evolve, and they lack the citation transparency essential for clinical verification. RAG-enhanced models, by contrast, ground their outputs in retrievable, updatable evidence, providing both factual accuracy and traceability—attributes critical for safe pharmacotherapy. However, pure RAG systems are constrained by retrieval dependency: when the knowledge base lacks coverage for atypical presentations, or when queries require cross-domain synthesis beyond the retrieved passages, RAG outputs may be incomplete. The hybrid architecture proposed in this study addresses these limitations by combining RAG-grounded reasoning (Tier 2) with deep chain-of-thought analysis (Tier 3) for edge cases, while preserving the efficiency of lightweight retrieval (Tier 1) for routine queries. Regarding SLMs specifically, while locally deployed models such as DeepSeek-v3.2 achieve “sufficient clinical intelligence” when augmented with high-quality external knowledge, they face inherent limitations in care coordination scenarios that require reasoning across multiple organ systems and longitudinal patient trajectories—tasks where larger reasoning models (e.g., GPT-o3) retain a performance advantage. The hierarchical design thus represents a pragmatic compromise: deploying SLMs where they suffice while reserving more capable models for the subset of cases where their additional reasoning capacity is clinically warranted.

The clinical implications of these findings suggest potential utility in primary care environments where immediate access to nephrology sub-specialists is limited. In the current state of DKD management, the patient journey from diagnosis to optimized management is often fragmented: primary care physicians may lack the nephrology-specific expertise to interpret nuanced eGFR trajectories or adjust medications according to the latest KDIGO guidelines, resulting in diagnostic delays, suboptimal pharmacotherapy, and delayed referrals to specialists. The proposed RAG-LLM system addresses these gaps by functioning as a real-time, guideline-grounded decision support tool at each stage of the patient journey. At the point of initial screening, the system can flag at-risk patients based on UACR and eGFR criteria; during treatment initiation, it ensures that first-line agents such as SGLT2 inhibitors are prescribed in concordance with eGFR-based eligibility thresholds; and during longitudinal follow-up, it provides dynamic dose adjustment recommendations as renal function evolves. By supporting primary care providers in more consistently applying specialist guideline recommendations without requiring immediate nephrology consultation, the system has the potential to compress the diagnostic-to-management interval, reduce iatrogenic errors, and improve the overall quality and continuity of DKD care. The system’s ability to provide real-time, guideline-compliant recommendations acts as a sophisticated clinical safeguard, potentially reducing the incidence of iatrogenic acute kidney injury caused by improper drug dosing. The system automates the cross-referencing of complex guidelines, such as the KDIGO and American Diabetes Association consensus statements. This capability may help primary care physicians apply guideline recommendations more consistently when managing early-stage DKD. Whether such support can improve treatment timeliness or delay disease progression requires prospective clinical validation. Moreover, the “citation-aware” interface design aligns with the principles of evidence-based medicine, encouraging clinicians to verify rather than blindly accept AI-generated advice, thereby maintaining the “human in the loop” for critical therapeutic decisions ²⁷.

While the current iteration of the system is optimized for a Chinese clinical context utilizing localized expert consensuses and a bilingual foundation model, the underlying hierarchical RAG architecture is fundamentally content-agnostic. Adapting this framework to other international healthcare systems requires tier-specific considerations: (1) Tier 1 (Fast Response Agent): portability is straightforward, as foundational DKD definitions and standard screening intervals are largely universal; replacing the knowledge base with translated generic medical content requires minimal effort. (2) Tier 2 (Enhanced Reasoning Agent with RAG): this tier requires the most substantial adaptation, as the vector database must be repopulated with region-specific clinical corpora—such as the American Diabetes Association (ADA) Standards of Medical Care or the European Association for the Study of Diabetes (EASD) consensus statements—and the retrieval scoring weights may need recalibration to account for differences in guideline structure and terminology across healthcare systems. Additionally, local hospital formularies and drug availability must be incorporated into the knowledge base to ensure prescribing recommendations are executable within the target institution’s constraints. (3) Tier 3 (Deep Analysis Agent): GPT-o3 operates as a cloud-based general-purpose reasoning engine; however, institutions with strict data governance requirements may need to replace this tier with a locally compliant reasoning engine, potentially at the cost of reduced reasoning depth for the most complex cases. Therefore, portability should be considered technical adaptability rather than immediate deployability, and site-specific validation remains necessary before clinical use. Despite these tier-specific considerations, the modular design of the framework facilitates systematic adaptation to diverse healthcare infrastructures, though site-specific validation and knowledge-base customization remain prerequisites for deployment in new clinical settings.

Despite these promising results, several limitations warrant consideration. A structured risk assessment of the system, categorized by severity and likelihood, is presented in Table 3 and summarized as follows: (1) Diagnostic risk (Severity: High; Likelihood: Low–moderate): while the system achieved high Accuracy and Diagnosis scores, the 2.1% metformin-related safety omission rate and 3.3% hyperkalemia monitoring omission rate in high-risk subgroups indicate that residual diagnostic risk persists, particularly in atypical DKD presentations that fall outside standard guideline algorithms; (2) Medication safety risk (Severity: High; Likelihood: Low in the proposed system): the safety-error audit demonstrated that the RAG-enhanced system substantially reduced but did not eliminate prescribing errors, underscoring the necessity for clinician review of all pharmacotherapy recommendations; (3) Care delivery risk (Severity: Moderate; Likelihood: Moderate): the system’s current on-demand nature means that it does not proactively monitor patients or alert clinicians to emerging safety signals (e.g., acute eGFR decline), limiting its utility as a continuous safety net; (4) Stakeholder risk (Severity: Moderate–high; Likelihood: Moderate): over-reliance on AI-generated recommendations without critical appraisal could inadvertently undermine clinical autonomy, particularly among less experienced practitioners who may defer to the system even in edge cases where guideline-based recommendations may not apply; (5) Privacy risk (Severity: High; Likelihood: Low): the Tier 3 cloud escalation pathway involves transmitting de-identified clinical data to an external API, which, while mitigated by the local-first routing strategy and de-identification protocols, warrants ongoing attention as data governance frameworks evolve. To mitigate these risks, the system explicitly positions itself as a decision support tool rather than a decision-making agent, maintaining the clinician as the final arbiter. First, the validation was conducted retrospectively using textual clinical records, which may not fully capture the dynamic complexity and temporal evolution characteristic of prospective, real-time patient interactions. Second, the current iteration relies exclusively on structured text and laboratory values, omitting critical imaging data such as renal pathology slides or Doppler ultrasound, which are essential for differentiating atypical DKD phenotypes. Future research will focus on evolving this text-based agent into a multimodal system capable of integrating imaging features with genomic data to support precision medicine. Furthermore, the presence of explicit citation markers in the outputs of the proposed retrieval-augmented system may have introduced a degree of trust bias during the physician evaluation. Although evaluators were blinded to model identities, responses accompanied by authoritative references inherently appear more trustworthy, which could subtly elevate subjective ratings in dimensions such as Factuality or Safety. While traceability is a core functional advantage of the RAG architecture designed to foster clinical trust and enable rapid evidence verification in real-world settings, future validation protocols should decouple the user interface from the clinical content—such as by masking citations during the review process—to isolate and assess the pure semantic accuracy of the generated reasoning. Additionally, it is important to note that the current system functions as an on-demand assistant, meaning its clinical utility depends on the physician’s initiative to query it. This retrospective study specifically measures the safety and guideline-concordance of the generated text, rather than actual changes in physician prescribing behavior or real-world error reduction. Prospective randomized controlled trials integrated into real clinical workflows are necessary to determine whether such tools can actively improve true guideline adherence and long-term patient outcomes, such as the preservation of renal function and the reduction of cardiovascular events, to pave the way for widespread clinical adoption ²⁸. Table 3 summarizes the structured risk assessment, including severity, likelihood, and mitigation strategies for each identified risk category.

Table 3.

Structured risk assessment of the proposed system

Risk Category	Example	Severity	Likelihood	Mitigation
Diagnostic risk	Atypical DKD/NDKD misclassification	High	Low–moderate	MDT review, citation verification
Medication safety risk	Missing eGFR-based dose adjustment	High	Low in proposed system	Safety-error rules, clinician review
Care delivery risk	Lack of proactive monitoring	Moderate	Moderate	Integration with EHR alerts in future work
Stakeholder risk	Over-reliance by junior physicians	Moderate–high	Moderate	Human-in-the-loop design, mandatory review
Privacy risk	Cloud escalation of sensitive data	High	Low	Local-first routing, de-identification

This study supports the potential clinical utility and technical robustness of a hierarchical multi-agent system for DKD management. Our multi-center validation confirms that a tiered architecture, which dynamically allocates computational resources based on query complexity, successfully reconciles the tension between real-time responsiveness and deep clinical reasoning. Crucially, the results demonstrate that domain-specific grounding through authoritative knowledge bases is a more significant determinant of clinical reliability than model parameter size alone, as evidenced by the system’s superior adherence to safety protocols compared to unaugmented baselines. The high inter-rater reliability observed across diverse physician seniority levels indicates that this framework functions with the transparency and consistency required for clinical integration. As a scalable solution bridging the gap between computational efficiency and expert-level decision support, this framework may support clinicians by producing guideline-grounded answers when queried. Ultimately, prospective clinical trials are necessary to demonstrate whether such on-demand assistance translates into tangible improvements in real-world prescribing adherence, safety events, and patient outcomes. Future work will focus on evolving this text-based architecture into a multimodal system, integrating renal pathology and longitudinal data to facilitate the transition toward comprehensive, individualized nephrology.

Methods

Construction of the DKD-specific knowledge corpus

To ensure the clinical authority and evidentiary depth of the system, we constructed a specialized knowledge corpus focused on DKD. The curation process prioritized high-quality, peer-reviewed resources intended for clinical professionals, rather than general health information.

The primary source materials were derived from recognized academic and professional organizations, encompassing clinical practice guidelines, expert consensus statements, professional monographs, and high-impact research articles. To maintain the integrity and density of the information ingested by the RAG framework, we implemented a rigorous multi-stage filtering protocol. Documents limited to introductory content or executive summaries were excluded to avoid redundancy with full-text versions. Furthermore, we prioritized temporal validity by including the most recent guidelines and their updated versions while systematically excluding obsolete documents that no longer align with current standards of care, such as those pre-dating the widespread clinical adoption of sodium–glucose cotransporter 2 (SGLT2) inhibitors and non-steroidal mineralocorticoid receptor antagonists (MRAs).

The final curated corpus comprises 139 foundational documents, including 1 pivotal clinical practice guideline [e.g., Kidney Disease: Improving Global Outcomes (KDIGO) 2024], 4 multi-institutional expert consensus statements, 2 authoritative monographs, and 132 high-quality peer-reviewed research articles (the full list of incorporated literature is detailed in Supplementary Table 1. This specialized library serves as the external “long-term memory” for the agent, providing the necessary evidence base for the subsequent retrieval, reasoning, and clinical decision support tasks described in the hierarchical framework.

LLM architecture design

Hierarchical multi-agent framework

To reconcile the trade-off between real-time responsiveness and the reasoning depth required for complex DKD management, we developed a hierarchical multi-agent architecture²⁹ rather than relying on a monolithic large language model. This framework employs a dynamic routing mechanism that stratifies clinical queries based on semantic complexity and the requisite depth of reasoning. As illustrated in Fig. 5, the router directs inputs to one of three specialized tiers: (1) The Fast Response Agent, which handles foundational definitions and standard inquiries with low latency; (2) The Enhanced Reasoning Agent, activated for intermediate clinical scenarios requiring evidence verification against clinical guidelines; and (3) The Deep Analysis Agent, which processes intricate cases involving multi-morbidity analysis or research-grade hypothesis generation. This stratified approach optimizes computational resource allocation, restricting high-cost inference to tasks explicitly demanding complex logical deduction. To ensure operational transparency and prevent unpredictable routing behaviors, the dynamic router was implemented utilizing a lightweight heuristic and keyword-matching strategy rather than a secondary complex language model. Specifically, the router continuously evaluates query complexity against explicit trigger criteria. Foundational queries lacking patient-specific metabolic data and strictly containing terminologies such as “definition,” “standard screening interval,” or “normal range” are classified as low-complexity and directed to Tier 1. Queries presenting structured clinical vignettes with standard metabolic parameters (e.g., exact eGFR values, HbA1c levels) that require direct guideline cross-referencing for pharmacotherapy adjustments are classified as moderate-complexity and processed by Tier 2 (DeepSeek-RAG). Conversely, high-complexity triggers are defined by the presence of keywords indicating refractory multi-organ complications (e.g., “concurrent severe heart failure,” “dialysis preparation”), atypical non-diabetic kidney disease (NDKD) presentations, or conflicting systemic comorbidities that fall outside standard guideline algorithms. Such cases are escalated to the Tier 3 (GPT-o3) cloud-based reasoning engine. This rule-based triage effectively decouples standard clinical workflows from edge-case research inquiries, ensuring that the cloud-based reasoning engine is exclusively reserved for scenarios exceeding routine nephrology protocols.

Fig. 5. Schematic of the hierarchical multi-agent architecture.

The system employs a dynamic router to classify queries into three complexity tiers: Tier 1 (Fast Response), Tier 2 (Enhanced Reasoning with RAG), and Tier 3 (Deep Analysis with GPT-o3). This design optimizes the trade-off between computational cost and reasoning depth.

Base Model Selection and Inference Strategy. Foundation models were selected to balance performance within the Chinese medical context with strict data privacy requirements. We employed DeepSeek-v3.2³⁰, a high-performance open-source model optimized for Chinese-English bilingual contexts, as the backbone for the local inference layers (Tier 1 and Tier 2). Deployment on a local high-performance GPU cluster ensures that sensitive patient data remains within the hospital intranet during routine diagnostic tasks. For the third tier, addressing highly complex research inquiries, we integrated GPT-o3 via a secure API. GPT-o3 was selected for its advanced reasoning capabilities—demonstrated on medical benchmarks such as MedQA³¹—which are essential for analyzing complex pathophysiological mechanisms and formulating personalized management strategies for refractory DKD cases. A “local-first” inference strategy was implemented, wherein approximately 88% of routine queries are processed locally by DeepSeek-v3.2 (Tier 1: 11.61%; Tier 2: 76.03%), with cloud-based GPT-o3 invoked only when the router detects complex query patterns (Tier 3: 12.36%), thereby balancing operational costs with diagnostic precision.

Hybrid RAG mechanism

To mitigate hallucinations—a prevalent issue in generative AI characterized by the generation of coherent but factually incorrect medical information—we implemented a hybrid RAG mechanism³² specifically for the Enhanced Reasoning Agent. Unlike standard RAG approaches relying solely on vector similarity, our system employs a dual-path “dense-sparse” retrieval strategy. We utilize the Milvus vector database for dense retrieval based on high-dimensional semantic embeddings to capture conceptual nuances (e.g., associating “foam in urine” with “proteinuria”). Concurrently, a keyword-based sparse retrieval engine ensures exact matches for specific pharmacological terms (e.g., “Dapagliflozin” or “eGFR thresholds”) that may be diluted in vector space. Retrieved knowledge chunks are processed through a structured prompt engineering framework incorporating CoT reasoning³³. As depicted in Fig. 6, context comprising retrieved excerpts from the curated DKD knowledge corpus is dynamically injected into the model’s prompt³⁴. The model follows a structured three-step reasoning protocol: (1) assessing the patient’s chronic kidney disease (CKD) staging and risk stratification; (2) cross-referencing specific contraindications within the retrieved guidelines (e.g., eGFR-dependent dosage adjustments for SGLT2 inhibitors); and (3) synthesizing a therapeutic recommendation with explicit citations. This constraint ensures the output represents a grounded synthesis of authoritative medical evidence rather than probabilistic prediction.

Fig. 6. Workflow of the hybrid RAG mechanism.

The pipeline integrates dual-path retrieval (semantic vector search + keyword matching) with Chain-of-Thought (CoT) prompting to ground DeepSeek-v3.2 responses in clinical guidelines.

LLM clinical validation and evaluation

Patient enrollment and baseline characteristics

To rigorously evaluate the clinical performance of the proposed hierarchical RAG-LLM system, we retrospectively enrolled a cohort of 267 patients diagnosed with DKD between January 2020 and December 2024. This multi-center study derived patient data from three tertiary medical institutions: The First Affiliated Hospital of Fujian Medical University (n = 166, 62.17%), Sanya Central Hospital (Hainan Third People’s Hospital, n = 62, 23.22%), and The Second Attached Hospital of Fujian Medical University (n = 39, 14.61%). The study population represented a broad spectrum of DKD progression and clinical complexity. The mean age of the cohort was $63.6\pm 11.9$ years, with a male predominance (71.91%). According to the KDIGO guidelines, the participants spanned all renal functional stages, with the majority in Stage 1-2 (32.59%) and Stage 3 (36.33%), while Stage 4 and Stage 5 (ESRD) accounted for 14.61% and 10.49%, respectively. Notably, the cohort presented a high prevalence of systemic comorbidities, including hypertension (80.15%), cardiovascular disease (56.18%), anemia (43.07%), and diabetic retinopathy (35.21%), providing a challenging and representative dataset for testing the agent’s multi-factorial management capabilities. Detailed baseline characteristics are summarized in Table 4. This study protocol was approved by the ethics committee of Sanya Central Hospital (Approval no. LLKY2307145), the ethics committee of the Second Affiliated Hospital of Fujian Medical University (Approval no. 2025-087), and the ethics committee of the First Affiliated Hospital of Fujian Medical University (Approval no. 2022-454) in accordance with the Declaration of Helsinki. The requirement for informed consent was waived by the ethics committee, as all patient identifiers were de-identified prior to analysis to ensure data privacy and confidentiality. Cases were consecutively enrolled to minimize selection bias. The three participating hospitals were selected to capture geographic and practice-pattern variability: the Fujian sites represent large tertiary referral centers in southeastern China with diverse inpatient and outpatient populations, while the Sanya site serves a distinct regional demographic in southern Hainan Province. Disease duration ranged from newly diagnosed DKD to long-standing disease exceeding 15 years (mean: 8.7 ± 5.3 years), ensuring that the validation cohort encompassed the full temporal spectrum of DKD management challenges.

Table 4.

Baseline characteristics of the study population

Characteristics	Total Patients (N = 267)
Age, years	63.6 ± 11.9
Sex, n (%)
Male	192 (71.91)
Female	75 (28.09)
Hospital, n (%)
The First Affiliated Hospital of Fujian Medical University	166 (62.17)
The Second Attached Hospital of Fujian Medical University	39 (14.61)
Sanya Central Hospital · Hainan Third People’s Hospital	62 (23.22)
CKD Stages, n (%)
Stage 1-2	87 (32.59)
Stage 3 (3a + 3b)	97 (36.33)
Stage 4	39 (14.61)
Stage 5 (ESRD)	28 (10.49)
Unknown	16 (5.99)
Comorbidities, n (%)
Hypertension
Yes	214 (80.15)
No	53 (19.85)
Anemia
Yes	115 (43.07)
No	152 (56.93)
Diabetic Retinopathy
Yes	94 (35.21)
No	173 (64.79)
CVD
Yes	150 (56.18)
No	117 (43.82)

Comparative validation strategy

To evaluate the clinical utility of the hierarchical architecture, we designed a multi-center, retrospective, blind validation study. The comparative analysis encompassed four distinct experimental arms to benchmark performance against both baseline and state-of-the-art systems: (1) DeepSeek-Base, the locally deployed foundational model without external knowledge retrieval; (2) DeepSeek-RAG (the proposed framework), incorporating the hybrid retrieval mechanism and dynamic routing; (3) GPT-4o, representing an established industry benchmark for generalized medical reasoning; and (4) GPT-o3, utilized as a high-level reference for reasoning-intensive tasks. To ensure an equitable and rigorous comparative analysis, all evaluated models (DeepSeek-Base, GPT-4o, GPT-o3, and the Proposed System) were subjected to a standardized inference protocol. Every model received the exact same de-identified clinical vignettes (as exemplified in Fig. 2a) without any model-specific manual rephrasing or formatting bias. Furthermore, to mitigate variations stemming from decoding strategies, we maintained uniform hyperparameters across all models where applicable. The generation parameters were uniformly set to a temperature of 0.1, a top-p of 0.90, a frequency penalty of 0.0, and a maximum token limit of 2048 to encourage highly deterministic and focused clinical responses. For the unaugmented baseline models, a standardized zero-shot system prompt assigning the role of an expert nephrologist was consistently applied. The exact prompt templates, including the standardized clinical instructions and the context-injected structures utilized within the RAG framework, are comprehensively documented in Supplementary Note 1. To illustrate the tiered routing mechanism, representative prompt examples for each tier are as follows: (1) Tier 1 example: “What is the normal range of UACR for screening DKD?”—a foundational query requiring no patient-specific data, handled by the Fast Response Agent with minimal latency; (2) Tier 2 example: “A 58-year-old male with type 2 diabetes, eGFR 42 ml/min/1.73 m², and UACR 350 mg/g. Should SGLT2 inhibitor be initiated, and is metformin dose adjustment required?”—a standard clinical vignette requiring guideline cross-referencing, processed by the Enhanced Reasoning Agent with RAG; (3) Tier 3 example: “A 62-year-old female with DKD Stage 4 (eGFR 18), concurrent heart failure (NYHA III), recurrent hyperkalemia on RAAS blockade, and recent acute kidney injury. How should RAAS, MRA, and SGLT2i therapy be reconciled?”—a complex multi-morbidity scenario exceeding standard guideline algorithms, escalated to the Deep Analysis Agent (GPT-o3) for exhaustive pathophysiological reasoning.

The validation cohort comprised 267 retrospective clinical cases of DKD consecutively recruited from three tertiary academic medical centers. Cases were selected to represent a diverse spectrum of disease progression, ranging from early-stage hyperfiltration (DKD Stage I) to ESRD, ensuring evaluation against varying degrees of clinical complexity. To minimize evaluator bias, a double-blind review process was implemented wherein all model outputs were anonymized and randomized. The expert evaluation panel consisted of 12 physicians stratified by professional seniority (residents, attending physicians, and chief specialists), facilitating a granular analysis of AI-clinician alignment across different expertise levels.

Multidimensional physician evaluation framework

Transcending binary accuracy metrics, we developed a comprehensive seven-dimensional evaluation framework specifically tailored to the complexities of nephrology management³⁵. As detailed in Table 5, model performance was quantified using a 5-point Likert scale (1 = Severe Deviation/High Risk, 5 = Excellent/Strict Conformity).

Table 5.

Multidimensional clinical evaluation metrics for the Diabetic Kidney Disease (DKD) intelligent system

Dimension	Evaluation focus	Scoring Criteria & Clinical Benchmarks	Score Anchors (1 = Critical Flaw vs. 5 = Clinical Excellence)
Accuracy	Alignment with KDIGO/CDS guidelines and staging precision.	Evaluates adherence to guidelines regarding DKD staging (G1-G5, A1-A3) and therapeutic choices. Focuses on the precision of first-line drug selection (e.g., SGLT2i, GLP-1RA).	1: Severe deviation (e.g., incorrect staging or obsolete therapy); 3: Generally accurate but misses precision in dosing or A-stage; 5: Strictly conforms to latest evidence; treatment plan matches expert consensus.
Comprehensiveness	Holistic multifactorial management coverage.	Assesses whether the response addresses the “whole-patient” needs: glycemic control, blood pressure, lipid profiles, and lifestyle interventions, beyond just renal protection.	1: Fragmented response (e.g., kidney focus only); 3: Covers basic metrics but neglects comorbidities; 5: Comprehensive management plan covering all metabolic risk factors.
Factuality	Evidence grounding and logical coherence.	Verifies the medical reasoning chain connecting lab values to decisions. Checks for “hallucinations” regarding drug mechanisms or non-existent guideline clauses.	1: Contains fabricated evidence or false logic; 3: Logical but lacks strong evidentiary support; 5: Rigorous logic chain with accurate citation of authoritative sources.
Response Efficiency	Information density and clinical usability.	Measures conciseness. Evaluates if the output allows a busy clinician to grasp key decision points within seconds, avoiding “textbook-style” redundancy.	1: Verbose, repetitive, or disorganized; 3: Information is buried but retrievable; 5: Highly structured, concise, and immediately actionable.
Safety (Critical)	Contraindication management and adverse risk mitigation.	Evaluates the identification of eGFR-based dose-adjustment requirements and contraindications and electrolyte status. Specifically assesses the risk stratification for hyperkalemia when initiating steroidal or non-steroidal mineralocorticoid receptor antagonists (e.g., finerenone) alongside RAAS inhibitors.	1: Life-threatening error (e.g., failing to recommend metformin dose adjustment/reassessment at eGFR 30–45 ml/min/1.73 m², failing to discontinue metformin at eGFR <30 ml/min/1.73 m², or initiating SGLT2i when eGFR <20 ml/min/1.73 m²); 3: Safe but lacks nuance in monitoring advice; 5: Flawless safety profile; proactively identifies risks and monitoring plans.
Further Diagnostic Tests	Relevance for differential diagnosis (DKD vs. NDRD).	Evaluates the utility of suggested workups (e.g., Renal Biopsy indications, fundoscopy) to differentiate DKD from Non-Diabetic Renal Disease (NDRD).	1: Irrelevant or wasteful testing suggestions; 3: Standard tests ordered but lacks specificity; 5: Highly targeted workup strategy that aids differential diagnosis cost-effectively.
Diagnosis	Concordance with MDT consensus.	Assesses the primary diagnostic conclusion. Focuses on the ability to identify the primary pathology and specific complications accurately.	1: Incorrect diagnosis; 3: Correct general category but lacks specific phenotyping; 5: Diagnosis fully matches the Multidisciplinary Team (MDT) gold standard.

These metrics were designed to capture critical dimensions of clinical decision-making beyond simple diagnostic concordance. While Accuracy and Diagnosis assess alignment with Multidisciplinary Team (MDT) consensus, Safety was prioritized as a core dimension given the high risks associated with renal pharmacotherapy (e.g., hyperkalemia from renin-angiotensin-aldosterone system inhibitors). Furthermore, Factuality and Response Efficiency evaluated the system’s reliability and practical usability in time-constrained clinical settings, respectively. Specifically, Response Efficiency was operationalized by evaluating the structural conciseness (e.g., the use of bulleted actionable insights versus dense paragraphs) and the objective output length, ensuring that key decision points could be rapidly extracted by a busy clinician. Each physician independently scored the randomized responses relative to de-identified patient records to generate a robust dataset for quantitative analysis³⁶. The MDT reference standard was established independently of the 12 evaluation physicians to avoid circular assessment. A dedicated three-member consensus panel—comprising a senior nephrologist (≥15 years clinical experience), a senior endocrinologist (≥12 years), and a clinical pharmacist specializing in renal pharmacotherapy—convened prior to the evaluation phase to generate reference answers for each of the 267 clinical cases. For each case, panel members independently drafted guideline-concordant management recommendations, which were subsequently reconciled through structured discussion to produce a single consensus reference. This pre-evaluation standardization ensured that physician evaluators assessed model outputs against a consistent, authoritative benchmark rather than their individual clinical judgment.

To transcend the inherent subjectivity of Likert-scale evaluations, we further established an objective, binary metric for identifying safety-critical errors. This safety-error audit was added during revision to provide an objective safety endpoint. The audit rules were defined a priori before re-reviewing the outputs and were then applied uniformly and retrospectively across all model outputs. In alignment with the 2024 KDIGO Clinical Practice Guideline and established pharmacological references, a safety-critical error was strictly defined as any generation instance containing one of the following specific clinical contraindications or omissions: (1) recommending metformin without dosage adjustment when eGFR is 30–45 ml/min/1.73 m², or failing to discontinue metformin when eGFR is strictly below 30 ml/min/1.73 m² (per KDIGO 2024 guidelines); (2) failing to issue an explicit hyperkalemia monitoring warning when initiating renin-angiotensin-aldosterone system (RAAS) inhibitors concurrently with mineralocorticoid receptor antagonists (MRAs, e.g., finerenone); and (3) inappropriately initiating sodium-glucose cotransporter-2 (SGLT2) inhibitors in patients with severe renal impairment (eGFR < 20 ml/min/1.73 m²). The absolute frequencies of these predefined errors were then audited across all model outputs by independent reviewers blinded to model identity.

Statistical analysis

Quantitative data from Likert scale evaluations were treated as continuous variables and presented as mean ± standard deviation (SD). Given the repeated-measures structure of the evaluation design—where each clinical case was assessed across multiple model outputs by multiple raters—the primary comparative analyses were conducted using linear mixed-effects models (LMM) with model type as a fixed effect and random intercepts for case and rater. The LMM was specified as: Score ~ Model + (1 | Case) + (1 | Rater), where Model was treated as a fixed effect and Case and Rater as random effects to account for the non-independence of observations arising from the crossed design. Pairwise model contrasts were estimated from the marginal means of the fitted mixed-effects models, with multiplicity adjustment for multiple comparisons. One-way ANOVA was retained only as a sensitivity analysis and did not alter the direction of the findings; accordingly, LMM-based P-values are reported as the primary results. The LMM-based pairwise comparisons confirmed that the Proposed System significantly outperformed DeepSeek-Base and GPT-4o across Accuracy, Factuality, and Safety (all LMM-based P < 0.001), with effect sizes (Cohen’s d) remaining consistent with those obtained from the sensitivity ANOVA framework. Additionally, Cohen’s d was calculated to determine the effect sizes for key performance differences, providing a standardized measure of the magnitude of improvement beyond mere statistical significance.

Author contributions

X.J.L., J.Q.L., Z.H.C., Y.N.W., Z.Y.Z., Y.H.L., and J.Y.X. collected clinical data. C.H.F., and J.H.L. performed methodology and formal analysis. X.T., J.H.L., J.Q.L., X.J.L., Z.H.C., Y.N.W., Y.H.L., Z.Y.Z., M.X.W., X.Y.S., W.X.C., and H.C. performed model evaluation and scoring on the entire dataset independently. X.T., L.T., and J.Q.L. prepared and wrote the draft of the manuscript. H.C., C.H.F., and L.T. designed the study and revised the manuscript. H.C. oversaw the study and secured funding. All authors subsequently edited the manuscript and approved its final version for publication.

Data availability

Due to ethical and privacy restrictions imposed by the Institutional Review Boards (IRBs) of the participating institutions, the raw patient data cannot be made publicly available. All patient identifiers were de-identified prior to analysis to ensure data privacy and confidentiality. Researchers who meet the criteria for access to confidential data may request access through the corresponding author (Hong Chen, fychenhong@126.com), subject to approval by the relevant ethics committees and completion of appropriate data use agreements.

Code availability

To support the findings of this study and ensure academic transparency, the custom code and analytical scripts utilized for the evaluation in this research have been deposited in a public repository. Due to strict institutional data privacy regulations, ethical constraints, and the proprietary nature of the hospital-deployed backend environment, the raw patient database and full deployment configurations cannot be made publicly available. The reproducibility package containing the relevant analysis scripts can be accessed via GitHub at: https://github.com/cfang6/Diabetic-nephropathy-agent-

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s41746-026-03035-0.