Table 3.

Error-Evaluation of Document Retrieval

Type of error in document retrieval	LLaMA 3.1 405B with RAG	GPT-4o with RAG
Overly specific protocols	10	8
Incorrect protocol in the correct subgroup	5	4
Misunderstanding of medical terms	4	4
Incorrect body region	2	2
Protocol retrieval based on pre-diagnosis	1	1

Overly specific protocols = vague clinical question requires a general protocol for broad differential diagnoses, disease-specific protocols are chosen instead; Incorrect protocol in the correct subgroup = confusion between protocols within the same category (e.g., tumor follow-up vs. recurrence); Misunderstanding of medical terms = misinterpretation of medical language; Incorrect body region = brain and spine protocols mistakenly interchanged; Protocol retrieval based on pre-diagnosis = focus on past diagnosis rather than current symptoms; RAG = Retrieval-Augmented Generation