Evaluating the accuracy of advanced language learning models in ophthalmology: A comparative study of ChatGPT-4o and Meta AI's Llama 3.1

Dear editor,

Artificial intelligence (AI) has rapidly evolved over the past few years, impacting various disciplines, including medicine. Initially focused on pattern recognition, data processing, and medical imaging analysis, AI has advanced to the level where it can affect clinical decision-making, reduce diagnostic errors, and improve patient outcomes, particularly in specialized fields like ophthalmology.^1,2

The role of AI in linking visual evaluations with diagnostic algorithms makes it particularly promising in ophthalmology. AI algorithms have already demonstrated proficiency in detecting diabetic retinopathy,³ retinal detachment,⁴ age-related macular degeneration,⁵ and glaucoma⁶ through image analysis. Such capabilities are likely to further enhance diagnostic efficiency when combined with physician judgment. As AI's role in image-based diagnostics is developing, the potential of large language models (LLMs), which can comprehend and generate human-like text, has remained largely unexplored.

LLMs represent a significant milestone in AI, beginning with OpenAI's release of Chat Generative Pre-trained Transformer 3 (GPT-3) in June 2020.⁷ GPT-3, trained on 410 billion words from the internet, features 175 billion parameters. Three years later, OpenAI launched ChatGPT-4, followed by the launch of ‘ChatGPT-4o-latest’ (ChatGPT) in August 2024. The fastest and most efficient model yet, dynamically updated from its predecessors, some experts hypothesize that the latest iteration could contain around 1.8 trillion parameters, although the exact number remains uncertain.8, 9, 10 Similarly, Meta (formerly Facebook) introduced Llama 3.1 in July 2024 as the largest open-source LLM to date.¹¹ Llama 3.1 is considered a new standard, outperforming ChatGPT and other LLMs in reasoning, multilingual capabilities, long-context handling, math, and evaluation metrics.¹² Llama 3.1 is available in three sizes: 8 billion (8B), 70 billion (70B), and 405 billion (405B) parameters.

Parameters are the internal variables and weights learned during training, enabling models to perform complex tasks, including natural language processing, medical diagnostics, and decision support. Increasing parameters generally enhance performance, allowing for better model capacity, generalization, fine-tuning, and contextual understanding. However, more parameters also increase the burden on processing power, computational cost, and training time, making the development and deployment of larger models both resource and technologically demanding.^13,14

The progression of LLMs, such as GPT-4o and Llama, represents a significant step forward in AI's capabilities. Recent LLMs have been applied successfully in various medical contexts, including passing standardized exams like the United States Medical Licensing Examination and providing insights into clinical decision-making.¹⁵ However, the application of these models in specialized fields, such as ophthalmology, in terms of accuracy, reliability, and utility compared to human experts remains to be addressed. As these models may not have specific pre-training in ophthalmic knowledge, the ability of LLMs to process and respond to textual, knowledge-based medical queries is still under investigation.11, 12, 13 Understanding such performance characteristics will be critical for the employment of LLMs in medical education and the training of physicians.¹⁴

Given the implications for medical education and clinical practice, this study aims to evaluate the accuracy of two of the most advanced LLMs to date -- ChatGPT-4o and Llama 3.1 -- in responding to ophthalmology-related test questions. While the performance of LLMs has previously been explored, it is unknown how much it has advanced with the updated model and new models, regarding ChatGPT and Meta AI, respectively.

1. Methods

Based on the general examination method of other investigators, a test set consisting of 254 questions was created from the Review Questions in Ophthalmology, Third Edition, a resource recommended by the American Academy of Ophthalmology (AAO) Committee for Resident Education as preparation for the Ophthalmic Knowledge Assessment Program (OKAP) examination.16, 17, 18 Since this book is not indexed by search engines, it is unlikely to have been included in LLM pre-training datasets.

The 254 questions were randomly selected from a total of 1062 questions. The current OKAP examination is structured into 13 sections (20 questions each) while the textbook has 12 (it does not include "general medicine"). The OKAP sections in this study were thus populated by 22 randomly selected questions from the corresponding source chapter. Questions requiring visual interpretation were excluded due to current limitations in LLMs' ability to process image-based inputs. Notably, only 12 pathology questions met this criterion as the majority contained visual material.

To replicate the OKAP examination format, we adhered to a multiple-choice question structure, where each question includes four options with one correct answer. The ChatGPT, Llama 70B, and Llama 405B models were independently prompted with the following preface: "Please select the correct answer and provide an explanation:" followed by the question-and-answer choices. To ensure that no memory or learning bias influenced the outcomes, a new session was initiated for each LLM prior to processing each question.

To determine the reliability of outputs from the tested LLMs, multiple trials were performed. Recognizing the probabilistic characteristics of LLMs, some variation in response was anticipated with each run. To evaluate this possibility, the same set of test questions were submitted three times, using identical prompts.

The textbook's question bank did not provide explicit difficulty ratings for each item. To address this, each question was categorized by cognitive level before testing, thereby enabling an analysis of each LLM's performance on questions of varying complexity. A two-tiered cognitive level system was adopted, simplifying the three-level system used by the AAO on real OKAP examinations.¹⁹ Low cognitive level questions focused on assessing simple factual recall and basic concept recognition, such as identifying the nerve responsible for a specific ocular movement. High cognitive questions, by comparison, required more advanced reasoning, such as determining the correct intraocular lens power calculation or selecting the best management approach for neovascular age-related macular degeneration.

This experimental design allowed for an evaluation of LLM performance across varying levels of cognitive demand and provided insights into the reproducibility of their outputs in a controlled, clinically relevant context. Three composite variables were created: ChatGPT4oCombinedAnswer, Llama70BCombinedAnswer, and Llama405BCombinedAnswer. These variables represent the common answer selected by the respective LLMs at least twice out of three trials. In cases where the LLMs provided different answers in all trial runs, the final response was recorded as the combined answer.

Logistic regression was conducted to determine significant associations between AI model accuracy and both the cognitive level and category of the questions, with P-values of <0.05 considered statistically significant. Logistic regression is a statistical modeling tool that estimates the relationship between a binary dependent variable and one or more independent variables. It allows for the control of multiple confounding variables and provides results in the form of adjusted odds ratios, which indicate the estimated risk of an outcome associated with the independent variables. The utility of this model lies in its capacity to account for complex interactions while studying a binary outcome measure.

The accuracy of each LLM was determined by verifying its responses against the answer provided in the book. Cohen's kappa was employed for reliability analysis to assess the agreement among three responses within each LLM, as well as for the agreement between the LLM combined response and the textbook's correct answer. The kappa statistic, as proposed by Cohen, measures the agreement between two binary variables while accounting for the probability of chance agreement. The kappa coefficient is calculated as the observed agreement minus expected agreement, divided by one minus expected agreement.²⁰ The strength of agreement was classified according to the following thresholds: poor (k ​≤0.20), fair (0.21≤ ​k ​≤0.40), moderate (0.41≤ ​k ​≤0.60), good (0.61≤ ​k ​≤0.80), very good (0.81≤ ​k ​≤0.90), and excellent (0.91≤ ​k ​≤1.00).²¹ Associations between categorical variables were analyzed using the Chi-square test. All analyses were performed using SPSS v29.

2. Results

The ChatGPT model achieved an accuracy of 75.9% ​± ​1.1%. Llama 3.1 models varied with the 405B model reaching 77.3% ​± ​0.6% accuracy and the 70B model at 64.9% ​± ​1.0%. The differences between the 405B and 70B models, between ChatGPT and the 70B model were statistically significant, but not between 405B and ChatGPT (Table 1). All models performed similarly across categories, with the weakest performances occurring in lens and cataract, neuro-ophthalmology, and optics categories (Fig. 1).

Table 1.

Accuracy results by model and test set.^a

AI Model	Test 1 (%)	Test 2 (%)	Test 3 (%)	Mean Accuracy ​± ​SD (%)
ChatGPT-4o	78	76	74	75.9 ​± ​1.1
Llama 70B	66.5	65.4	63	64.9 ​± ​1.0
Llama 405B	77.2	78.3	76.4	77.3 ​± ​0.6

Results are presented as a percentage of correct answers. Each model completed three identical test sets, comprising 254 questions each, totaling 762 questions (n ​= ​762). The overall mean and standard deviation were calculated across models' three test sets.

^aChatGPT-4o and Llama 405B significantly outperformed Llama 70B (P < 0.001) but did not differ significantly from each other (P > 0.05).

Fig. 1.

Model Accuracy Comparison

Comparison of accuracy scores across 12 categories for three models: ChatGPT-4o, Llama 70B, and Llama 405B. Error bars represent standard errors.

Questions of cognitive level 2 had a 35.8% decrease in the odds of Llama 3.1 70B answering correctly compared to questions of cognitive level 1 (Table 2). In contrast, there was no significant association between question cognitive level with 405B or ChatGPT accuracy. No significant association existed between the question category and any LLMs' accuracy.

Table 2.

Accuracy by cognitive level for AI models.

AI Model	Odds Ratio	95% Confidence Interval	P-value
ChatGPT-4o	1.166	[0.522, 2.605]	0.709
Llama 70B	0.358	[0.170, 0.754]	0.007a
Llama 405B	1.379	[0.630, 3.018]	0.421

Only Llama 3.1 70B showed a significant decrease in odds of answering questions belonging to cognitive level 2 (n ​= ​61) correctly compared to cognitive level 1 (n ​= ​193).

^aStatistically significant.

All three LLMs demonstrated good internal agreement (test-retest) (Table 3). 405B and ChatGPT had good agreement with the correct answer, whereas 70B exhibited moderate agreement. The 405B and 70B models displayed the least amount of agreement between their answers, while 405B and ChatGPT produced had highest amount of agreement between each model (Table 4).

Table 3.

Intra-AI agreement.

AI Model	Comparison	Agreement Level	Kappa (k) Valuea
ChatGPT-4o	Answers 1 & 2	Very good	0.879
	Answers 1 & 3	Very good	0.879
	Answers 2 & 3	Very good	0.874
	Combined vs. Correct	Good	0.679
Llama 70B	Answers 1 & 2	Very good	0.942
	Answers 1 & 3	Very good	0.921
	Answers 2 & 3	Very good	0.926
	Combined vs. Correct	Moderate	0.526
Llama 405B	Answers 1 & 2	Very good	0.858
	Answers 1 & 3	Very good	0.853
	Answers 2 & 3	Very good	0.900
	Combined vs. Correct	Good	0.700

Agreement levels and kappa values for comparisons between combined answers from Llama 70B, Llama 405B, and ChatGPT-4o. Values indicate the level of agreement between answers, with higher kappa values indicating stronger agreement.

^aall Kappa values P < 0.001.

Table 4.

Cross-model agreement.

	Agreement Level	Kappa (k) Valuea
Llama 70B vs. ChatGPT-4o	Good	0.611
Llama 70B vs. Llama 405B	Moderate	0.595
Llama 405B vs. ChatGPT-4o	Good	0.726

Agreement levels and kappa values for comparisons between combined answers from Llama 70B, Llama 405B, and ChatGPT-4o. Values indicate the level of agreement between models, with higher kappa values indicating stronger agreement.

^aall Kappa values P < 0 .001.

3. Discussion

Other studies that evaluated various LLMs on OKAP practice test questions found ChatGPT-3.5 demonstrated an accuracy range of 46%–59.4%,^22,23 while ChatGPT-4 achieved between 70% and 81%.^16,24 Our results are promising, aligning well with or even surpassing the reported accuracies of previous models assessed on OKAP test results. There is currently no published research evaluating the performance of any Llama model on standardized medical knowledge-test accuracy, which limits direct comparisons. A 71% accuracy on the OKAP exam was reported for other LLMs such as Google Gemini and Bard, further supporting the validity of our findings in relation to existing LLM benchmarks.²⁵ When considering general medicine standardized exams, such as the USMLE, ChatGPT-3 showed an accuracy of 60%,²⁶ and ChatGPT-4 achieved 77% on other specialty board questions.²⁷ These comparisons further underscore the recent advancements in LLMs noted earlier.

The analysis also revealed that the performance of the Llama 3.1 70B model decreased significantly on questions classified as cognitive level 2 compared to level 1, while no such association was observed for the 405B model or ChatGPT-4o (Table 2). This suggests that smaller models may struggle more with complex reasoning tasks, which likely require a deeper understanding of content and emphasizes the importance of model size in handling more sophisticated medical questions.

Meta AI's 70B model significantly underperformed compared to its more complex sibling in terms of accuracy (Fig. 2). This disparity, attributable solely to the difference in the number of parameters, aligns with computational studies indicating that increasing the number of parameters generally improves performance, as the model can better capture and leverage complex patterns within the data. Additionally, the unexpected finding that the 70B model's responses aligned more closely with ChatGPT's than with 405B further underscores the possibility that smaller models may adopt different generalization strategies. Due to its reduced complexity, the 70B model may rely on reasoning pathways that are more akin to those used by ChatGPT, rather than its larger counterpart. All of this indicates that models of varying sizes process and generalize medical knowledge differently.

Fig. 2.

AI Model Parameters vs. Accuracy Percentage

Relationship between model size and accuracy for three large language models: ChatGPT-4o, Llama 70B, and Llama 405B. Asterisk (∗) indicates that the 1.8 trillion parameter count for ChatGPT-4o is an estimate. This figure shows that larger models generally achieve higher accuracy, the performance similarity between the 405B model and ChatGPT-4o suggests diminishing returns as models scale, indicating a potential parameter-performance plateau.

The absence of significant differences between the 405B model and ChatGPT-4o suggests that simply increasing model size may lead to diminishing returns, particularly as models reach a certain scale. While the parameter count of ChatGPT-4 remains uncertain, its comparable performance to the 405B model hints at a parameter-performance plateau. This is consistent with research indicating that beyond a certain point, performance gains from increased parameters become nonlinear.^10,14,26 The analysis also highlighted limitations across all models in certain ophthalmology categories such as lens and cataract, neuro-ophthalmology, and optics. This suggests that despite overall strong performance, LLMs struggle with narrowly focused areas of expertise. The findings underscore the potential need for domain-specific pretraining or fine-tuning of LLMs to enhance their utility in ophthalmology. Incorporating public-domain ophthalmology resources like EyeWiki for pretraining could improve the accuracy and reliability of AI-generated responses in clinical settings.

It is essential to recognize that the observed differences in performance may stem from factors beyond merely parameter count. These variations may reveal inherent limitations related to model design and training methodologies. Elements such as a model's architecture, the diversity and quality of its training dataset, and the optimization techniques employed during training can significantly influence its capacity to generalize across complex medical domains. This is particularly relevant as previously discussed in the context of the 70B model, which may rely heavily on the specific training data to compensate for its smaller parameter count. On the other hand, models like ChatGPT may benefit from access to a more diverse, high-quality training dataset and employ advanced optimization techniques that enhance their reasoning abilities beyond just the size of its parameters.

Understanding these nuances underscores that, while parameter size is significant, strategic improvements in training methodologies and architectural design are crucial for developing models that excel in specialized domains, such as ophthalmology. However, it is important to note that the proprietary nature of these design and training elements complicates direct comparisons and comprehensive discussions between LLMs.

The implications of these findings for medical education and practice are significant. As LLMs are increasingly integrated into educational tools and clinical decision-support systems, understanding their strengths and limitations is crucial. The increased accuracy rates achieved by the larger models suggest that these tools could potentially be valuable in complementing traditional educational resources, particularly in helping prepare students and professionals for standardized examinations. LLMs hold tremendous potential by making education more accessible: they provide cost-effective, personalized explanations of exam material to students and create practice test questions.²⁸ However, it's important to note that larger models often require more advanced computing resources and may be behind paywalls, potentially limiting access for some users. Additionally, the observed deficiencies in specific categories and the variable performance depending on cognitive complexity highlight the need for cautious integration. Overreliance on LLMs without proper oversight could lead to gaps in learning or even propagate misinformation, particularly in areas where the models are less accurate.

This study has several limitations. First, the models were evaluated using questions based on 2015 compendium that could have impacted the results, but since the questions tend to reflect core and basic factual knowledge and clinical practice patterns that are time-vetted, that effect is likely small. The pretraining dataset for Llama 3.1 encompasses information up to December 2023, whereas ChatGPT's training data only extends to October 2023, potentially enabling both models to provide more contemporary and different, yet accurate responses.^10,12 That said, the LLMs still displayed superior accuracy to other studies. Another limitation might be that the tested question set may not accurately represent a real OKAP exam.

In conclusion, while larger LLMs such as Meta AI's Llama 3.1 405B and ChatGPT-4o-latest models demonstrate improved accuracy in ophthalmology-related questions in comparison to their predecessors and smaller models, the marginal gains observed with increasing model size suggest that further scaling may not significantly enhance performance. Likewise, the persistent deficiencies seen in certain ophthalmology categories indicate that domain-specific pretraining may be necessary to fully realize the potential of LLMs in ophthalmology. These findings contribute to the growing body of literature exploring the capabilities and limitations of LLMs in specialized medical fields like ophthalmology. The integration of written and image content into future models could address some of these limitations. Such models would likely enhance the validity and utility of LLMs in ophthalmic education, training, and exam preperation, where multimodal analysis is critical for practice skills such as interpreting optical coherence tomography (OCT) scans, fundoscopic images, or interpreting ocular pathology alongside clinical reasoning.

Study approval

There was no requirement for IRB study approval as this study did not involve human subjects.

Author contributions

TL: Conceptualization, Data curation, Investigation, Methodology, Visualization, Writing ​– ​original draft. RL: Writing - Review and Editing. RM: Writing - Review and Editing, Formal Analysis. CM: Writing - Review and Editing, Supervision, Project Administration. All authors reviewed the results and approved the final version of the manuscript.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.