Skip to main content
Arthroscopy, Sports Medicine, and Rehabilitation logoLink to Arthroscopy, Sports Medicine, and Rehabilitation
. 2026 Nov 9:e70086. Online ahead of print. doi: 10.1002/ars2.70086

Large Language Models Aligned With Medical Experts on Established Topics but Were Less Aligned on Emerging or Debated Topics in Posterolateral Corner Treatment: An Artificial Intelligence‐Augmented Delphi Study

Yinan Yu 1,2,3, Luke V Tollefson 4, Robert F LaPrade 4,✉, Jorge Chahla 5, Felix C Oettl 6, Kristian Samuelsson 7,8,9, Robert Feldt 1,2
PMCID: PMC13748481  PMID: 43011399

Abstract

Purpose

To evaluate how Large Language Models (LLMs) align with a concurrently performed human medical expert‐modified Delphi study on the posterolateral corner and to qualitatively analyze LLM behavior.

Methods

Four commercially available and current LLMs (Gemini‐2.5‐Pro. GPT‐4o, o4‐mini, and o3) were tasked with rating posterolateral corner consensus statements on a 5‐point Likert scale and estimating the level of expert consensus for each statement, mirroring a human expert Delphi process. LLM performance was evaluated based on alignment with expert majority opinion, accuracy in predicting expert consensus, and self‐consistency across multiple LLM invocations. Deep research to improve LLM performance was also explored.

Results

LLM‐generated ratings showed moderate correlation with average expert ratings, with some models exhibiting tendencies toward overly positive ratings. Consensus derived from LLM self‐consistency did not reliably approximate expert consensus. LLM estimates of consensus were informative, with some models showing improved alignment with actual expert consensus. Qualitative analysis revealed distinct behavioral patterns across different LLMs regarding justification generation, certainty expression, and handling of conflicting evidence. Preliminary findings suggest that deep research capabilities can enhance rating accuracy as well as consensus prediction. Evidence of hallucination in LLM reasoning is observed and requires further verification.

Conclusions

Although LLMs show potential in reflecting expert medical knowledge and estimating consensus, their direct ratings and self‐consistency may not perfectly align with expert panels. Different LLMs display unique strengths and weaknesses, suggesting that model selection or a combination can be beneficial for applications like modified Delphi studies. Improved consensus estimation was seen by providing the models access to web search, enabling them to find recent, relevant research articles.

Level of Evidence

Level V, expert opinion.


graphic file with name ARS2-9999-e70086-g002.webp


The posterolateral corner (PLC) of the knee is composed of 3 main static stabilizers: the lateral (fibular) collateral ligament, popliteus tendon, and popliteofibular ligament. These structures combine to provide stability to varus and external rotation forces and function as secondary stabilizers to anterior and posterior translation of the knee. Historically, the PLC has been recognized as the “dark side” of the knee because of its more complex anatomy and less optimal surgical outcomes. 1 , 2 Modified Delphi studies are being increasingly used in orthopaedic surgery where limited knowledge is available and where expert consensus is desired. They have been used to establish minimal reporting standards for biologics treatments and to obtain expert consensus for the treatment of complex orthopaedic pathology. 3 , 4 , 5 In 2018, a modified Delphi expert consensus statement of the diagnosis, classification, treatment, and rehabilitation of the PLC was published. 2 To evaluate the progression toward the recommended treatments of this previous work, a new modified expert Delphi PLC statement was commissioned and published. 6

Large Language Models (LLMs) have emerged as potentially powerful tools capable of processing and generating human‐like text, sparking significant interest in their diverse applications within medicine, including clinical decision support, medical education, and administrative task automation. 7 , 8 The potential for LLMs to analyze vast medical literature and assist in complex medical reasoning is rapidly being explored. 9 , 10 This burgeoning capability necessitates a thorough evaluation of how LLM‐generated information aligns with established expert knowledge and the nuanced processes of medical consensus. Such an assessment serves a dual purpose: first, to critically appraise the general medical understanding and reliability of these artificial intelligence (AI) models 11 and second, to investigate their potential utility in supporting and possibly enhancing medical consensus‐building exercises like modified Delphi studies. 9 , 12 , 13 Although LLMs could offer benefits such as accelerating consensus or summarizing evidence, they also carry risks, including the perpetuation of data biases, generation of inaccurate information, or a deficiency in the sophisticated reasoning that characterizes human medical expertise.

The authors believe that it would be time to evaluate the ability of LLMs to align with expert knowledge and consensus processes. It was desired to determine if LLMs align with expert ratings and consensus on complex medical pathology (like the PLC), can accurately estimate the level of expert consensus, have self‐consistency, and determine if grounding techniques (like search and summarize with deep research) 14 can improve performance through justification and certainty. Thus, the purpose was to evaluate how LLMs align with a concurrently performed human medical expert‐modified Delphi study on the PLC and to qualitatively analyze LLM behavior. The hypothesis was that LLM consensus estimation would be more informative than raw data evaluation, that LLM self‐consistency would be lower for those statements that lacked human expert consensus, and that deep research would improve LLM alignment with the human PLC‐modified Delphi consensus statements.

METHODS

In the concurrent human medical expert‐modified Delphi study, participants were medical experts who rated each statement on a 5‐point Likert scale, indicating their level of agreement to the statement. Expert consensus was operationally defined using standard thresholds in a modified Delphi methodology: In rounds 1 and 2, consensus to agree was considered reached when ≥70% of respondents rated a statement as “Agree” (4) or “Strongly Agree” (5) and ≤20% rated it as “Disagree” (2) or “Strongly Disagree” (1); for the final round, the threshold was raised to ≥75% agreement with ≤20% disagreement to identify the final consensus, and a consensus was considered “not achieved” otherwise. This dataset served as both the input and ground truth for evaluating the LLM responses and to evaluate the ability of LLMs to replicate expert judgment and predict consensus in a modified Delphi study by performing structured reasoning, rating, and justification tasks. For convenience, relevant AI terminology is provided in Table 1.

TABLE 1.

A Brief Summary of AI Concepts

AI Concept Explanation Relevance to This Study
LLM A neural network trained on massive text corpora to predict the next token, enabling fluent language generation and comprehension Core technology supplying automated ratings, justifications, and consensus estimates
Reasoning model A LLM whose fine‐tuning objective emphasizes logically coherent generation rather than conversational style alone o4‐mini and o3 are tuned for structured reasoning, affecting how they justify ratings
Deep research An workflow in which the model first queries external sources (e.g., PubMed and web search) and conditions its answer on the retrieved texts Used with o4‐mini to ground ratings and consensus predictions in up‐to‐date clinical literature
Temperature Sampling parameter (0‐1+) that controls output randomness; typically, lower values yield consistent answers, and higher values encourage diversity Varied to test whether response diversity improves self‐consistency and consensus accuracy
Hallucination Generation of content that is plausible‐sounding but factually incorrect, unverifiable, or fabricated Observed when reasoning models cited vague or nonexistent studies, which highlights the need for expert review
Grounding Requiring the model to cite or rely on verifiable external evidence rather than its internal knowledge alone Implemented through deep research to improve transparency and reduce unsupported claims
Self‐consistency Degree to which a model repeats the same answer across independent runs with identical prompts Evaluated as a proxy for expert consensus
Recall (sensitivity) Proportion of true positives correctly identified: TP/(TP + FN) Indicates a model's ability to detect statements where surgeons reached consensus
Precision Proportion of positive predictions that are correct: TP/(TP + FP) Complements recall to show how often a model's “consensus achieved” predictions were right
F1 score Harmonic mean of precision and recall, balancing false‐positive and false‐negative errors in one metric Primary number used to compare overall consensus prediction accuracy across models

AI, artificial intelligence; FN, false negative; FP, false positive; LLM, Large Language Model; TP, true positive.

To explore this, 4 LLMs were chosen to represent a range of model sizes, architectures, and reasoning capabilities: GPT‐4o, 15 Gemini 2.5 Pro preview version available April to May 2025, 16 o4‐mini, 17 and o3. 17 GPT‐4o and Gemini 2.5 are state‐of‐the‐art generalist models trained for broad applicability across tasks, whereas o3 and o4‐mini are optimized specifically for reasoning performance, with o4‐mini being a significantly smaller model in terms of parameters and computational footprint (Figure 1). Each model was presented with statements from the modified Delphi study in May 2025, one at a time, and asked to perform a sequence of tasks designed to emulate the evaluative and interpretive processes used by human experts (Supporting Information).

FIGURE 1.

FIGURE 1

Description of the LLMs used in this study. The name, developer, size, and characteristics of each LLM are listed. (LLMs, Large Language Models.)

Each separate LLM was first instructed to rate the statement using the same 5‐point Likert scale as human participants. The LLM then provided a brief, 1‐ or 2‐ sentence justification for its rating, referencing the strength or type of evidence. After rating, the model predicted whether medical experts would reach a consensus to agree on the statement based on the defined Delphi thresholds. It selected 1 of 3 possible outcomes, consensus to agree (i.e., consensus achieved), consensus to disagree (i.e., consensus not achieved), or unclear, along with a justification explaining its rationale. Each model completed 5 independent runs per statement.

Deep research, which is augmented LLMs with access to external tools such as web search or scientific databases, enabled them to retrieve, synthesize, and integrate relevant evidence beyond their static training data into reasoning and decision‐making. In this study, a deep research mode of o4‐mini was adopted for analyzing the first round of the Delphi study, in which the model was explicitly instructed to perform evidence‐based literature retrieval prior to responding to a given query. This setup aimed to improve the relevance and reliability of model justifications by grounding them in up‐to‐date clinical literature and to enhance the accuracy of consensus predictions in areas where guideline clarity and consistency of published evidence are especially critical. The prompts used in the study can be found in Supporting Information.

To guide our investigation, 3 Research Questions (RQs) were formulated. RQ1 evaluated to what extent LLM‐generated ratings aligned with those provided by human medical experts. This question examined whether LLMs can reproduce human expert judgments on the Likert scale in a manner consistent with an expert medical consensus. RQ2 assessed how accurately LLMs can predict whether medical experts will reach a consensus on specific PLC treatment statements. This assesses the capacity of LLMs to anticipate the outcome of scientific consensus, operationalized through predefined Delphi thresholds. RQ3 evaluated how self‐consistent LLMs across multiple runs were for the same PLC statement and if self‐consistency correlated with expert consensus. This explored whether a model's internal agreement across runs was indicative of expert agreement and whether self‐consistency was a viable proxy for expert medical consensus. Additionally, a qualitative analysis was performed to evaluate the models’ rating behaviors and reasoning styles using example responses to selected statements.

To evaluate rating alignment and address the RQs, the Pearson correlation coefficient was used to compare model‐generated ratings with expert ratings. The model rating was averaged across multiple runs when available, and the expert rating was averaged across responses from 40 medical experts. This metric provided a measure of how closely the model's quantitative judgments aligned with those of human experts.

For consensus evaluation, 2 approaches were considered. In the first, the model explicitly predicted whether consensus among experts would be achieved for a given statement. When there were multiple runs, the most frequent prediction was identified as the final estimation. In the second, consensus was inferred from the model's own outputs by examining the consistency of its responses across multiple independent runs. Sensitivity, specificity, and the F1 score were used to assess how well the model distinguished between cases where consensus was achieved and cases where it was not. When using explicit model predictions, both “no consensus” (indicating diverse opinions among experts) and “consensus to disagree” were mapped to a single label, “consensus not achieved,” whereas only “consensus to agree” was labeled as “consensus achieved.”

RESULTS

Rating Alignment (RQ1)

We compared the ratings generated by different LLMs to expert ratings. For each question, the expert rating was computed as the mean value across ratings from 40 individual experts. For each model, 5 independent runs were conducted per question, and the average model rating across these runs was used for analysis (Table 2).

TABLE 2.

Model Performance of Rating Alignment for Posterolateral Knee Injuries

Delphi Round Model Pearson Correlation P Value
1 GPT‐4o −0.12 .351
Gemini‐2.5 −0.033 .8
o4‐mini −0.027 .832
Deep Research o4‐mini 0.596 <.001
o3 −0.024 .854
2 GPT‐4o −0.107 .516
Gemini‐2.5 0.096 .56
o4‐mini 0.056 .735
o3 −0.008 .959
3 GPT‐4o 0.15 .362
Gemini‐2.5 −0.018 .913
o4‐mini 0.235 .15
o3 0.228 .163

As shown in Figure 2, the dynamics of rating behavior across modified Delphi rounds revealed clear differences between models. In later rounds (rounds 2 and 3), which included questions with higher expert agreement, both GPT‐4o and o4‐mini showed a shift in rating distributions: Their ratings became higher on average and exhibited lower standard deviations. This suggests that these models were somewhat sensitive to the increasing clarity or consensus in the questions and adjusted their ratings accordingly. In contrast, Gemini‐2.5 displayed little change across rounds. Its rating distribution remained centered around a Likert scale 4 grading with consistently moderate variance, suggesting a strong positivity bias that was relatively insensitive to the nature of the questions. The o3 model, optimized for reasoning, exhibited a more cautious rating pattern overall, with generally lower ratings and slight shifts in distribution across rounds.

FIGURE 2.

FIGURE 2

Graphical representation of the average rating of the 4 LLMs used across all 3 rounds, including the use of Deep Research (o4‐mini) in round 1. All LLMs were compared against doctor ratings on the same set of consensus statements related to the diagnosis, treatment, and rehabilitation of PLC injuries. The consensus statement on PLC has been published by LaPrade et al. 6 (LLMs, Large Language Models; PLC, posterolateral corner.)

Consensus Prediction (RQ2)

The accuracy of the models in predicting expert consensus was evaluated. Expert consensus was defined per question based on majority agreement among the 40 medical experts. For each model, 5 independent runs per question were performed, and the model's direct consensus estimate (rather than an aggregation of ratings) was used to generate the consensus prediction using the most frequent prediction across the 5 runs. Performance was evaluated using the F1 score, specificity, and sensitivity (Table 3).

TABLE 3.

Model Performance of Consensus Prediction

Delphi Round Model F1 Score Specificity Sensitivity
1 GPT‐4o 0.725 0.6 0.781
Gemini‐2.5 0.727 0.433 0.875
o4‐mini 0.722 0.533 0.812
Deep Research o4‐mini 0.72 0.467 0.844
o3 0.697 0.633 0.719
2 GPT‐4o 0.847 0.833 0.758
Gemini‐2.5 0.839 0.5 0.788
o4‐mini 0.935 1 0.879
o3 0.9 1 0.818
3 GPT‐4o 0.825 1 0.703
Gemini‐2.5 0.882 0.5 0.811
o4‐mini 0.93 0.5 0.892
o3 0.879 1 0.784

Note: Deep research does not help with consensus estimation.

All models tended to have high sensitivity, indicating that they are generally good at detecting questions where expert consensus existed (true positives). Model performance improved noticeably across Delphi rounds, which was expected because later rounds contained questions with higher expert agreement, making the task easier for the models. There was no clear pattern, indicating a single best‐performing model across all rounds. Notably, Gemini‐2.5 consistently exhibited low specificity alongside high sensitivity, indicating a tendency to overpredict agreement. The deep research variant of o4‐mini did not yield improved consensus predictions; if anything, it showed reduced specificity, meaning it became less reliable at identifying cases where expert disagreement existed. This suggests that adding a research phase made the model more verbose or variable in its reasoning but did not translate into more calibrated or accurate expert consensus estimation.

Self‐Consistency Versus Medical Expert Consensus (RQ 3)

Analysis of LLM self‐consistency, deriving consensus by voting across multiple independent runs, was performed to determine if it could approximate expert consensus. For this analysis, GPT‐4o was used with 30 independent runs per question. For each question, the same logic used to define expert consensus was applied, and the most frequent prediction across the 30 runs was used as the predicted consensus. The results in Table 4 show that consensus derived from self‐consistency voting was not a reliable approximation of expert consensus.

TABLE 4.

Sampled Statements for Qualitative Inspection

Statement Experts Avg. Rating Experts Agreeing Experts Disagreeing Expert Consensus o4‐mini Rating o4‐mini Consensus Prediction
59 3.85 30 8 Yes 2 No False negative
56 2.65 9 22 No 3 No True negative
26 3.8 29 9 No 4 Yes False positive
27 4.35 35 2 Yes 5 Yes True positive

Qualitative Results

To evaluate the models’ rating behaviors and reasoning styles, a qualitative analysis using example responses to selected statements was conducted. Outputs from multiple models were sampled, and these responses were analyzed to identify patterns in reasoning quality, evidence use, and consistency with expert consensus.

First, one example statement along with its ratings and reasoning to illustrate model performance and interpretation is presented below. The statement is: “The external rotation recurvatum test is not specific for a complete PLC injury.” The average rating from the experts was 3.6, with 67.5% agreeing to the statement (no consensus was reached). Representative responses from 4 different models to this statement to illustrate their rating behavior and reasoning style are then presented:

  • •

    Gemini‐2.5 (rating 5): “Strongly agree based on established orthopedic literature and biomechanical studies which show this test indicates severe combined ligamentous instability (PLC + cruciate), not specific to an isolated complete PLC injury.”

  • •

    GPT‐4o (rating 4): “Agree based on systematic reviews and clinical trials that indicate the external rotation recurvatum test lacks specificity and sensitivity for diagnosing complete PLC injuries due to the complexity of knee pathology associated with PLC injuries.”

  • •

    o3 (rating 2): “Disagree: multiple recent systematic reviews and cohort studies (e.g., Geeslin and LaPrade 2016; Devitt et al. 2019) consistently report very high specificity (>90%) of the external rotation recurvatum test for high‐grade/complete PLC injuries, although its sensitivity is low.”

  • •

    o4‐mini (rating 4): “Agree based on recent systematic reviews and meta‐analyses showing that while the external rotation recurvatum test is sensitive for combined PCL‐PLC injuries, it also yields positive findings in isolated PCL tears or hyperextensible knees, limiting its specificity for complete PLC lesions.”

Although all models provided on‐topic responses, the evidence they refer to was often arbitrary. The reasoning tended to be superficial or overly general, with no clear linkage to actual high‐quality evidence. For instance, Gemini‐2.5 and GPT‐4o relied on vague references to “orthopaedic literature” and “systematic reviews,” whereas o3 cited specific papers but misrepresented their content or overstated the specificity, with the Devitt et al. article in 2019 nonexistent (likely hallucination). o4‐mini provided somewhat more nuanced phrasing but still lacked concrete evidence and inaccurately reported an association between the PCL and PLC for a positive external rotation recurvatum test. In the literature, there is an association between the anterior cruciate ligament and the PLC for a positive external rotation recurvatum test. It was found that o3 consistently uses references as evidence, often backing its arguments with concrete numerical values. However, the references provided are frequently inaccurate. Although they may touch on a similar topic, they are often not specific to the exact question at hand, and in some cases, the references are clearly fabricated, sometimes citing a journal name and year without providing an actual article or correct citation. This pattern indicates a high level of hallucination: The model generates authoritative‐sounding references to strengthen its responses, but these references were not grounded in verifiable sources.

In addition, a closer look at how the deep research approach component behaves was analyzed, because adding this component improved the accuracy of rating predictions (Supporting Information). To further explore its performance, 4 statements were sampled for a qualitative analysis (Table 4). In Statement 56 (Weightbearing after isolated FCL reconstruction), the model predicted no consensus (false), whereas experts also did not reach consensus (false). The model's output was reasonable but missed a key Level 1 study reporting no clinical difference between non‐weightbearing and partial weightbearing for isolated FCL reconstructions. 18 This key reference omission likely contributed to the lack of consensus and limited the explanation's clarity. In Statement 26 (Meniscal tears and surgical timing in acute grade 3 PLC injuries), the model incorrectly predicted consensus (true), whereas experts did not reach consensus (false). Although most experts agreed on the early treatment of major meniscal tears, variation in surgical practices across centers (e.g., single‐stage vs 2‐stage surgical procedures) and the lack of definitive evidence explain the nonconsensus. In Statement 59 (Return to sport timeline after isolated PLC surgery by 6 months postoperatively), the model predicted no consensus (false), whereas experts did achieve consensus (true). This statement applies only in optimal cases, where the patient has full strength, knee confidence, and objective evidence of graft healing. However, most patients do not meet these optimal conditions at 6 months postoperatively. Many have additional concomitant ligament injuries or face delays in rehabilitation. For them, 9 months or more is typically needed to return to full activities. The model may have missed consensus because of this nuance and the high frequency of combined ligament injuries in PLC cases. Finally, in Statement 27 (Impact of neurovascular injury on surgical timing of grade 3 PLC tears), both the model and experts agreed on a consensus (true). Neurovascular injuries clearly influence surgical timing, especially with popliteal artery injuries being surgical emergencies. This case showed general clinical alignment and understanding. Overall, disagreements arose mostly because of nuances in concomitant ligament injuries, overlooked studies, and variations in clinical practices.

DISCUSSION

In this study, we found that LLMs primarily aligned with human experts in consensus predictions for established PLC topics but were overall less aligned with emerging and debated topics. The hypothesis of the study was confirmed in that LLM consensus estimation was more informative than raw data evaluation, LLM self‐consistency was lower for those statements that lacked human expert consensus, and deep research improved LLM alignment with the human PLC‐modified Delphi consensus statements. When further applying the deep research approach to the initial queries, the models showed further alignment with expert ratings. Thus, it appears that the grounding provided by deep research enabled the models to articulate their reasoning based on the available literature, underscoring the importance of using AI deep research techniques in enhancing both model performance and transparency for responding to medical queries.

In interpreting these findings, statements with clearer and more consistent representation in the literature were easier for LLMs to evaluate accurately, whereas more nuanced or controversial topics led to greater variability in their interpretation. The use of deep research significantly improved alignment in such cases, because it allowed the models to explicitly retrieve and summarize relevant literature rather than relying solely on pretrained internal representations. The complexity of the PLC as an anatomical and clinical entity likely further challenged the models; despite this, they performed well on statements addressing the core aspects of PLC diagnosis and treatment but were less consistent on statements reflecting emerging or debated concepts.

In comparison to prior expert consensus efforts on PLC injuries, this work highlights both the alignment and limitations of AI assistance. Although modified Delphi studies have been used previously to establish consensus on PLC diagnosis and treatment, 2 there is limited published work applying LLMs and AI to this specific area. In other areas of orthopaedics, for example, in the shoulder with rotator cuff injuries, a study by Megafu et al. 19 reported inconsistent concordance with the American Academy of Orthopaedic Surgeon Clinical Practice Guidelines when prompting with ChatGPT and Gemini at 81% and 75%, respectively. These findings are similar to those found in the present study in which it was found that LLMs were typically aligned with medical experts in published topics but less aligned in the emerging and debated topics. This highlights a key limitation in that current LLMs do not differentiate well between systematic reviews, expert opinion, or lower‐tier studies unless explicitly prompted or grounded via external data. Therefore, while helpful for breadth, LLMs are not yet sufficient for automated evidence grading or guideline development in emerging and debated topics.

Beyond consensus statements, these findings suggest broader opportunities for LLM use in orthopaedic workflows. For example, LLMs could support literature reviews by surfacing key themes and controversies or generate structured summaries to inform early‐stage drafting of clinical guidance. 20 , 21 In education, LLM‐generated explanations, especially when grounded with external data sources, may provide learning tools for trainees navigating complex topics such as PLC injuries. 8 , 20 Moreover, researchers may explore further into integrating LLMs into scoping reviews, registry analysis, or structured data abstraction tasks, particularly in data‐scarce or rapidly evolving subfields.

In terms of practical implementation and future directions, these findings suggest several potential LLM applications. For researchers, LLMs may serve as useful tools to accelerate the early stages of consensus development, helping to highlight key areas of agreement and controversy for expert review. 22 For patients, current LLM outputs are not yet reliable enough to serve as standalone clinical guidance, given the known limitations in evidence evaluation and reasoning. 23 , 24 However, as models improve, they may eventually contribute to patient‐facing education tools or shared decision‐making aids. For research purposes, grounded LLM outputs appear sufficiently accurate to inform literature reviews or to assist in drafting consensus statements; however, these outputs may not properly cite the literature or may provide vague references, as highlighted in the results with the outputs from the o3 model on the external rotation recurvatum test. Depending on the model, topic, and prompting, the outputs may be accurate; however, for others, especially for complex topics, expert validation is still required. More broadly, the use of LLMs in other modified Delphi processes across medical specialties represents an important future direction, with potential to streamline consensus‐building in complex, or rapidly evolving, fields.

Overall, LLMs’ estimated consensus was informative and could accelerate the initial screening in modified Delphi processes, but it was found that they cannot yet replace expert deliberation on specific topics. In addition, deep research showed a potential to improve rating performance. However, differences between LLMs exhibited distinct behaviors, so comparisons between different LLMs may allow for a more comprehensive assessment of the literature for a specific pathology.

Limitations

The authors acknowledge some limitations of this study. First, this study was specific to PLC injuries, so its ability to relate to other orthopaedic pathologies is not yet known. Second, only commercial LLMs were used over open‐source models to be able to show current state‐of‐the‐art use of LLMs. Furthermore, present LLMs do not appear to differentiate information published in higher‐ versus lower‐tiered peer‐reviewed literature, and further LLMs should strive to differentiate among higher versus lower levels of evidence in publications. Finally, the methodology for deep research would be anticipated to require further refinement over time.

CONCLUSIONS

Although LLMs show potential in reflecting expert medical knowledge and estimating consensus, their direct ratings and self‐consistency may not perfectly align with expert panels. Different LLMs display unique strengths and weaknesses, suggesting that model selection or a combination can be beneficial for applications like modified Delphi studies. Improved consensus estimation was seen by providing the models access to web search, enabling them to find recent, relevant research articles.

GPT‐4o, 15 Gemini 2.5 Pro preview version available April to May 2025, 16 o4‐mini, 17 and o3 17 were used to compare expert consensus to LLM models. These LLM models were not used in writing the manuscript.

SUPPORTING INFORMATION

Additional supporting information can be found online in the Supporting Information section.

DISCLOSURES

The authors (R.F.L., J.C., K.S., R.F.) declare the following financial interests/personal relationships that may be considered as potential competing interests: R.F.L. reports relationships with Ossur Americas that includes consulting or advisory and funding grants; Smith & Nephew that includes consulting or advisory and funding grants; Linvatec Europe that includes consulting or advisory; Responsive Arthroscopy that includes consulting or advisory; the Arthroscopy Association of North America that includes funding grants; and the American Orthopaedic Society for Sports Medicine that includes funding grants. J.C. is a board or committee member of the American Orthopaedic Society for Sports Medicine, the Arthroscopy Association of North America, and ISAKOS; is a paid consultant for Arthrex, ConMed Linvatec, Ossur, RTI Surgical, Smith & Nephew, and Vericel; receives hospitality payments from Breg, DePuy Synthes Sales, the Joint Restoration Foundation, Medical Device Business, Pacira Pharmaceuticals, SI‐Bone, and Vericel; receives educational support from Midwest Associates; and is a paid presenter or speaker for Smith & Nephew. K.S. is a member of the Board of Directors of Getinge AB (publ) and medtech advisor to Carl Bennet AB. R.F. is CEO of Accelerandium AB. The other authors (Y.Y., L.V.T., F.C.O.) declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this article.

Supporting information

Supplementary Material

ARS2-9999-e70086-s001.pdf (174.3KB, pdf)

REFERENCES

  • 1. Chahla J, Moatshe G, Dean CS, LaPrade RF. Posterolateral gcorner of the knee: Current concepts. Arch Bone Joint Surg. 2016;4:97‐103. [PMC free article] [PubMed] [Google Scholar]
  • 2. Chahla J, Murray IR, Robinson J, et al. Posterolateral corner of the knee: An expert consensus statement on diagnosis, classification, treatment, and rehabilitation. Knee Surg Sports Traumatol Arthrosc. 2019;27:2520‐2529. [DOI] [PubMed] [Google Scholar]
  • 3. Murray IR, Chahla J, Safran MR, et al. International expert consensus on a cell therapy communication tool: DOSES. J Bone Joint Surg Am. 2019;101:904‐911. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4. Murray IR, Makaram NS, Geeslin AG, et al. Multiligament knee injury (MLKI): An expert consensus statement on nomenclature, diagnosis, treatment and rehabilitation. Br J Sports Med. 2024;58:1385‐1400. [DOI] [PubMed] [Google Scholar]
  • 5. Murray IR, Geeslin AG, Goudie EB, Petrigliano FA, LaPrade RF. Minimum information for studies evaluating biologics in orthopaedics (MIBO): Platelet‐rich plasma and mesenchymal stem cells. J Bone Joint Surg Am. 2017;99:809‐819. [DOI] [PubMed] [Google Scholar]
  • 6. LaPrade RF, Gerhold C, Kunze KN, et al. A contemporary international expert consensus statement on the evaluation, diagnosis, treatment, and rehabilitation of injuries to the posterolateral corner of the knee. Arthroscopy. 2025;41:4630‐4640. [DOI] [PubMed] [Google Scholar]
  • 7. Shool S, Adimi S, Saboori Amleshi R, Bitaraf E, Golpira R, Tara M. A systematic review of large language model (LLM) evaluations in clinical medicine. BMC Med Inform Decis Mak. 2025;25:117. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8. Vrdoljak J, Boban Z, Vilovic M, Kumric M, Bozic J. A review of large language models in medical education, clinical decision support, and healthcare administration. Healthcare. 2025;13:603. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9. Schifano J, Niederberger M. How Delphi studies in the health sciences find consensus: A scoping review. Syst Rev. 2025;14:14. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10. Yun H, Marshall I, Trikalinos T, Wallace BC. Appraising the potential uses and harms of LLMs for medical systematic reviews. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing . Singapore: Association for Computational Linguistics; 2023:10122‐10139. [Google Scholar]
  • 11. Bedi S, Liu Y, Orr‐Ewing L, et al. Testing and evaluation of health care applications of large language models: A systematic review. JAMA. 2025;333:319‐328. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12. Lange T, Kopkow C, Lutzner J, et al. Comparison of different rating scales for the use in Delphi studies: Different scales lead to different consensus and show different test‐retest reliability. BMC Med Res Methodol. 2020;20:28. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13. Wang D, Zhang S. Large language models in medical and healthcare fields: Applications, advances, and challenges. Artif Intell Rev. 2024;57:299. [Google Scholar]
  • 14. Jesudason D, Gao C, Seth I, Chan WO, Bacchi S. OpenAI's ‘deep research’ for the generation of comprehensive referenced medical text: Uses and cautions. ANZ J Surg. 2025;95:2250‐2252. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15. GPT‐4o System Card . OpenAI. https://openai.com/index/gpt‐4o‐system‐card/. Accessed May, 2025.
  • 16. Gemini 2.5 Pro‐Model Card . https://deepmind.google/models/gemini/pro/. Accessed May, 2025.
  • 17. o3 and o4‐mini System Card . Open AI. https://openai.com/index/o3‐o4‐mini‐system‐card/. Accessed May, 2025.
  • 18. LaPrade RF, DePhillipo NN, Cram TR, et al. Partial controlled early postoperative weightbearing versus nonweightbearing after reconstruction of the fibular (lateral) collateral ligament: A randomized controlled trial and equivalence analysis. Am J Sports Med. 2018;46:2355‐2365. [DOI] [PubMed] [Google Scholar]
  • 19. Megafu M, Guerrero O, Yendluri A, et al. ChatGPT and Gemini are not consistently concordant with the 2020 American Academy of Orthopaedic Surgeons Clinical Practice Guidelines when evaluating rotator cuff injury. Arthroscopy. 2025;41:2753‐2757. [DOI] [PubMed] [Google Scholar]
  • 20. Ramkumar PN, Kunze KN, Haeberle HS, et al. Clinical and research medical applications of artificial intelligence. Arthroscopy. 2021;37:1694‐1697. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21. Wascher DC, Ollivier M. Large language models in orthopaedic publications: The good, the bad and the ugly. Am J Sports Med. 2024;52:2193‐2195. [DOI] [PubMed] [Google Scholar]
  • 22. Hasan SS, Woo JJ, Cote MP, Ramkumar PN. Generative versus nongenerative artificial intelligence. Arthroscopy. 2025;41:545‐546. [DOI] [PubMed] [Google Scholar]
  • 23. Kodra JD, Saroyan A, Darby F, et al. ChatGPT‐generated responses across orthopaedic sports medicine surgery vary in accuracy, quality, and readability: A systematic review. Arthrosc Sports Med Rehabil. 2025;7:101210. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24. Spaan J, Streepy J, Hodakowski A, et al. Individuals frequently search Google with questions about the management of meniscal tears and the indications for and technical details of surgery but the quality of the information is suboptimal. Arthrosc Sports Med Rehabil. 2025;7:101061. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary Material

ARS2-9999-e70086-s001.pdf (174.3KB, pdf)

Articles from Arthroscopy, Sports Medicine, and Rehabilitation are provided here courtesy of John Wiley & Sons on behalf of the Arthroscopy Association of North America

RESOURCES