A Multi-Model LLM Consensus Framework to Identify EHR-Predictable Eligibility Criteria in NSCLC Immunotherapy Trials

Abstract

Clinical trial participation in oncology remains low, with complex and granular eligibility criteria limiting timely accrual and excluding many patients who could otherwise benefit from novel therapies. We propose a structured framework that utilizes a multi-model large language model consensus pipeline to convert free-text eligibility from Phase III PD-1/PD-L1 non–small cell lung cancer trials into standardized umbrella concepts and to classify each concept into tiers of Electronic Health Record predictability. Applying this framework, we consolidated hundreds of heterogeneous eligibility traits into a manageable set of clinical concepts and found that roughly half are both clinically important and plausibly inferable from routine structured and unstructured EHR data, while the remainder either add little value to prediction or depend on specialized testing. This taxonomy provides a prioritized “predictable target” list for future NLP and machine-learning models and a practical blueprint for designing more computable, pragmatic eligibility criteria and EHR-driven prescreening tools.

Introduction

Clinical trials are essential for advancing cancer care, yet participation remains persistently low. In the United States, fewer than 5% of adult cancer patients enroll in trials, and structural and clinical barriers contribute to this under-enrollment¹. Across oncology, an estimated 20–40% of trials fail to meet accrual targets or close early due to poor recruitment^2,3, delaying evaluation of new therapies and limiting patient access to potentially beneficial treatments⁴. Although lack of awareness, geographic access, and mistrust remain important contributors, several reports have highlighted trial design factors—particularly restrictive or complex eligibility criteria—as a major barrier to timely accrual and to the generalizability of trial findings⁵.

These concerns are especially pronounced in immunotherapy and precision oncology. Eligibility sections in contemporary thoracic and solid tumor trials now often include dozens to hundreds of inclusion and exclusion clauses spanning performance status, comorbidities, organ function thresholds, prior lines of systemic therapy, and increasingly granular biomarker requirements⁶. Longitudinal analyses of National Cancer Institute–affiliated trials have shown that eligibility content has grown substantially over time and that greater eligibility “content density” is associated with higher odds of accrual failure. Narrow and layered criteria can exclude older adults, patients with comorbidities, and racial and ethnic minorities who more closely resemble the real-world population, thereby limiting both accrual and external validity⁶. In practice, complex eligibility requirements also introduce workflow burden: clinicians and coordinators must reconcile multiple data sources, interpret nuanced temporal and disease-status conditions, and often order additional tests to resolve uncertainty, even when many patients will ultimately prove ineligible⁷.

Electronic Health Records (EHRs) offer a natural foundation for enhancing this process. EHR-based prescreening systems use structured data and clinical notes to automatically identify potentially eligible patients, reduce manual chart review, and generate alerts to research staff⁸. Numerous NLP and machine learning approaches have been developed for computational phenotyping and trial matching, ranging from rule-based extraction of key attributes to deep learning systems that process free-text eligibility criteria and clinical notes^9–11. Formal representation work has aimed to encode eligibility criteria in computable formats, through logical templates, ontologies, and annotated corpora such as CHIA, to enable semi-automated cohort identification and decision support^12,13. However, the real-world impact of these systems is constrained by data availability and quality: many high-value eligibility traits (for example, extent of metastatic involvement, autoimmune history, or prior lines of therapy) are incompletely captured in structured fields, embedded inconsistently in free text, or missing altogether, which limits the performance and reliability of automated prescreening^14,15. As a result, it remains unclear which eligibility criteria are realistic targets for EHR-based inference, and which effectively require new testing or data collection.

Recent advances in large language models (LLMs) offer new capabilities for understanding and transforming clinical text at scale. Domain-scale models such as GatorTron trained on billions of words of de-identified clinical notes have achieved state-of-the-art performance on a range of EHR NLP tasks, including concept extraction, relation identification, and phenotyping^13,16. More general-purpose frontier LLMs have demonstrated strong performance on medical question answering benchmarks and licensing-style examinations, suggesting they encode substantial clinical knowledge and reasoning ability¹⁷. At the same time, evaluations also emphasize that LLM outputs can be inconsistent, overconfident, or sensitive to prompt phrasing, underscoring the need for careful validation and methods that reduce reliance on any single model¹⁸. Prior work has explored LLMs and earlier NLP systems for parsing eligibility criteria, mapping them to standardized representations, and generating patient-matching queries, but they depend on the availability of computable data; in practice, many high-value eligibility traits are incompletely captured or missing, limiting the impact of automated approaches^12,19.

To address these gaps, we propose a structured framework that audits and classifies trial eligibility traits into EHR predictability tiers, explicitly distinguishing criteria that can plausibly be inferred from existing structured and unstructured EHR content from those that require new testing or specialized assays. Using phase III PD-1/PD-L1 non–small cell lung cancer (NSCLC) trials as a case study, we leverage a multi-model LLM consensus strategy to (1) standardize heterogeneous eligibility expressions into a set of umbrella concepts and (2) estimate, for each concept, whether it is realistically predictable from routine EHR data in a typical health system. This approach yields a prioritized set of high-value “predictable targets” that can guide the design of future NLP and machine-learning systems for EHR-based prescreening and inform efforts to make eligibility criteria more computable, pragmatic, and aligned with real-world data capture.

Method

Overall workflow

The workflow consisted of four modules (Figure 1): (1) identification and extraction of trial eligibility traits, (2) consolidation of semantically related traits into umbrella concepts, (3) development of an EHR predictability framework, and (4) application of a multi-model LLM consensus pipeline to assign each umbrella concept to a predictability category. Together, these modules move from raw free-text eligibility criteria posted on ClinicalTrials.gov to a standardized taxonomy of eligibility concepts annotated with their putative EHR predictability.

Figure 1.

Overall workflow transforming free-text eligibility criteria into standardized umbrella concepts with assigned EHR predictability tiers via a four-module LLM-based pipeline.

Study setting and data source

This retrospective, protocol-level study analyzed eligibility criteria for phase III immune checkpoint inhibitor (ICI) trials in non–small cell lung cancer (NSCLC) registered on ClinicalTrials.gov, a public registry that provides structured protocol elements (e.g., design, arms, outcomes) alongside free-text eligibility criteria²⁰. We queried ClinicalTrials.gov using the expression: “Non-Small Cell Lung Cancer” AND (“PD-1” OR “PD-L1” OR “Programmed Death-1” OR “Programmed Death-Ligand 1”) AND “Phase 3” AND “United States”, and retrieved 107 trials that met the following conditions: (1) interventional, phase III studies, (2) adult NSCLC as the target population, (3) a PD-1 or PD-L1 inhibitor as a study or comparator drug, and (4) English-language inclusion and exclusion criteria available in the “Eligibility Criteria” field. Prior work has used similar ClinicalTrials.gov–based sampling strategies to analyze eligibility criteria complexity and representation across large trial cohorts^21,22.

For each included study, we crawled the eligibility section and retained only the free-text inclusion and exclusion criteria. Trial identifiers (NCT numbers), phase, sponsor type, and intervention names were stored for contextual analyses but were not used directly in the LLM classification pipeline.

LLM environment and consensus implementation

Automated text processing and classification were performed using five contemporary frontier LLMs, representing multiple vendors and training paradigms: Claude Opus 4.5 (Anthropic), GPT-5.1 (OpenAI), Gemini 3 Pro (Google), DeepSeek Reasoner (DeepSeek), and Grok 4.1 (xAI). These models were selected to capture diversity in architecture and training data while remaining accessible via stable APIs. We intentionally selected general-purpose frontier LLMs rather than medically fine-tuned models (e.g., Med-PaLM, MedGemma) for two reasons. First, the primary task in this study—segmenting, grouping, and classifying eligibility criteria text drawn from ClinicalTrials.gov protocols— is fundamentally a language understanding and categorization task rather than a clinical reasoning or diagnostic task. The input consists of semi-structured protocol language describing study requirements, not patient-level clinical narratives requiring deep medical inference. General-purpose models with broad language comprehension are well suited to such text processing tasks. Second, recent work from our group has demonstrated that general-purpose LLMs, when guided by well-structured prompts and domain-specific instructions, can achieve strong performance on healthcare NLP tasks without requiring domain-specific pretraining²³. This evidence supports the premise that careful prompt engineering and task decomposition can compensate for the absence of domain-specific pretraining, particularly when the task centers on language structure rather than clinical decision-making. Where available, extended reasoning modes were enabled to promote deliberative classification, consistent with recommended practices for clinical and biomedical applications of LLMs²⁴.

Prompt design and tuning

Prompts were designed to be model-agnostic and followed a consistent structure across all tasks: (1) a role definition establishing the model as a clinical informaticist, (2) explicit task instructions, (3) category definitions with positive examples, and (4) strict JSON output format requirements. For the classification task, each category was defined with both inclusion criteria and representative examples (e.g., “ECOG performance status, measurable disease per RECIST” for P&N; “HIV status, pregnancy test, basic liver function labs” for P-NN; “PD-L1 expression, EGFR mutation status” for NP). Extended thinking/reasoning modes were enabled for all models to promote deliberative classification. No few-shot examples beyond the category definitions were provided, ensuring that classifications reflected each model’s inherent clinical reasoning rather than pattern matching to exemplars. For each task, all five models processed the same input payload. We then aggregated their outputs using a pre-specified consensus rule: for each item, we recorded the output selected by each model, applied majority vote to assign a provisional label, and derived a confidence level based on the degree of agreement—Green (≥4 of 5 models in agreement), Yellow (3 of 5), or Red (<3 agreeing). This type of ensemble strategy has been proposed as one way to mitigate idiosyncratic biases and instability in individual LLMs while leveraging their complementary strengths. All implementations are available at: https://github.com/abfnu090/clinical-trial-eligibility-classification. All LLM calls were orchestrated using scripted workflows, which logged prompts, responses, timestamps, and token usage. No trial-specific patient-level data was used; only public protocol text from ClinicalTrials.gov was processed.

Eligibility trait extraction

Eligibility sections in ClinicalTrials.gov records are written as semi-structured free text, often using lists and subclauses. Our goal in the first module was to segment these sections into “traits,” defined as the smallest clinically meaningful unit that could plausibly correspond to a single eligibility feature for EHR-based patient matching (e.g., “ALK fusion,” “ECOG performance status 0–1,” “no active autoimmune disease requiring systemic therapy”).

For each eligible trial, we programmatically retrieved the eligibility text and passed it to an LLM-based segmentation prompt. The prompt instructed the model to: (1) split the text into atomic traits, (2) avoid duplicating identical phrases, and (3) remove purely structural tokens (e.g., bullet labels) while preserving clinical semantics. Compared with purely rule-based sentence splitting, this approach allowed us to handle nested bullet lists and complex clause structures that are common in oncology protocols. Related work on eligibility information extraction has also exploited multi-step NLP pipelines to recognize entities, attributes, and relations in criteria text^25,26.

Across the 107 trials, the initial segmentation step yielded 606 trait mentions. We then applied basic preprocessing, such as conversion to lowercase, whitespace normalization, and removal of exact duplicate strings, to reduce trivial redundancy. This resulted in 543 unique trait strings, which constituted the input vocabulary for subsequent modules.

Umbrella concept induction

Eligibility criteria frequently express the same underlying concept in slightly different ways across trials and even within the same protocol (e.g., “ALK rearrangement,” “ALK fusion,” “translocation of ALK”)²¹. To enable a meaningful analysis of predictability and downstream tool development, we sought to normalize this heterogeneity by grouping semantically equivalent traits into umbrella concepts that represent distinct clinical ideas.

The full list of 543 trait strings was provided to each of the five LLMs with instructions to: (1) group traits that describe the same underlying clinical concept into approximately 40–80 “umbrella” categories, (2) assign a concise, standardized clinical label to each umbrella, and (3) ensure that every trait was assigned to exactly one umbrella. Using the unified umbrella list derived from these proposals, the models were then prompted to (4) map each original trait string to exactly one umbrella category. This two-step procedure is analogous in spirit to concept normalization and ontology-driven grouping in prior eligibility representation work, but here we rely on LLM-based semantic grouping rather than an external ontology²⁷.

We reconciled these five model-specific schemas into a unified set of umbrella concepts through a two-step consolidation process. First, we prompted Claude Opus 4.5 to examine all umbrella proposals from the five models and identify semantic overlaps by analyzing shared trait membership. The prompt provided each model’s proposed umbrellas along with representative traits under each category, and instructed it to merge categories that described the same underlying clinical concept despite differing terminology (for example, “ALK alteration,” “ALK rearrangement,” and “ALK mutation status” were consolidated into a single umbrella). When multiple labels existed for the same concept, a standardized clinical term was selected. The model was instructed to ensure that every original trait mapped to exactly one umbrella in the final consolidated set, targeting 40–80 categories to balance granularity with interpretability.

Second, the consolidated umbrella assignments were reviewed for quality assurance. Traits that were inconsistently grouped across models or placed into ambiguous categories were flagged for manual review. These flagged cases were examined and reassigned to the most appropriate umbrella based on clinical definitions, creating new umbrella categories only when a genuinely distinct concept was identified. This process resulted in 92 umbrella concepts, each linked to one or more original trait strings and serving as the unit of analysis for subsequent EHR predictability classification.

EHR predictability framework

In the third module, we developed an EHR predictability taxonomy to categorize umbrella concepts based on how realistically they could be inferred from routine EHR data in a typical large healthcare system. Prior literature on computable phenotypes and eligibility representation has emphasized that not all clinically meaningful attributes are equally available in structured form; some require complex NLP over clinical notes, while others depend on specialized tests that may not be routinely performed²⁸. Guided by this work and by clinical experience with NSCLC ICI trials, we defined three mutually exclusive categories:

Predictable and Necessary (P&N): These concepts are frequently required for ICI eligibility but are not reliably available as structured fields. Examples include performance status (ECOG/WHO), documentation of brain or CNS metastases, measurable disease per RECIST, prior immune-related adverse events, certain autoimmune conditions, and patterns of prior systemic therapy (e.g., number and type of prior lines).

Predictable but Not Necessary (P-NN): These concepts are typically available as discrete data in the EHR or easily obtained in routine care; examples include basic demographics, many standard laboratory values, and common viral serologies when ordered. While they may also be predictable from EHR data, the marginal benefit of building prediction models for these concepts is lower because the information is generally readily accessible or easily obtainable.

Not Predictable (NP): These concepts typically rely on specialized testing, tissue availability, or protocol-specific procedures that cannot be assumed to be present in the EHR at the time of prescreening. Examples include the PD-L1 expression level as determined by a specific assay, rare oncogenic driver alterations or fusion panels that are not universally tested in the target population, and detailed tissue submission or archival requirements. These traits are critical to trial design but are not realistic targets for pre-existing EHR-based inference alone.

The definitions, along with representative examples for each category, were embedded directly into the prompts used in the final classification module to standardize how models interpreted “predictable” and “necessary.” Examples of these categories are shown in Table 1.

Table 1.

EHR predictability categories with representative examples.

Category	Definition	Example Umbrellas
Predictable & Necessary (P&N)	Concepts frequently required for ICI trial eligibility that are not reliably captured in structured EHR fields but can plausibly be inferred from existing clinical documentation. These traits are often embedded in oncology progress notes, radiology reports, pathology narratives, or treatment summaries, requiring natural language processing or chart abstraction to extract. Because they are commonly missing yet clinically important for eligibility determination, they represent high-value targets for imputation models that could expand the pool of identifiable trial candidates.	Performance Status, Brain/CNS Metastases, Prior Systemic Therapy, Disease Stage, Autoimmune Disease, Measurable Disease, Disease Progression, Interstitial Lung Disease/Pneumonitis, Prior Immunotherapy/Checkpoint Inhibitors, Treatment-Related Toxicity
Predictable but Not Necessary (P-NN)	Concepts are typically available as discrete structured data in the EHR or easily obtained through routine clinical care. These include standard laboratory values, basic demographics, and common screening tests that are ordered as part of the usual practice. While these traits may also be predictable from clinical documentation, the marginal benefit of building imputation models is low because the information is generally readily accessible or can be obtained through simple intake questions or routine orders.	Age, Hepatic Function, Renal Function, Hematologic Function, HIV Status, Hepatitis B/C Status, Pregnancy/Lactation Status, Thyroid Function, Smoking Status, Blood Pressure/Hypertension
Not Predictable (NP)	Concepts that cannot be reliably inferred from standard EHR data require specialized molecular testing, tissue-based assays, or prospective specimen collection. These traits depend on results from specific diagnostic platforms or protocols that may not be universally performed in the target population prior to trial screening. While critical for trial eligibility, they are not realistic targets for EHR-based prediction and must be obtained through dedicated testing as part of the screening process.	PD-L1 Expression Status, EGFR Mutation Status, ALK Mutation Status, ROS1 Rearrangement Status, KRAS/STK11/KEAP1 Mutation Status, HER2 Status, Tumor Tissue/Biospecimen, Genetic/Germline Testing, Actionable Genomic Alterations, MRD Status

Multi-model LLM consensus for predictability classification

In the final module, each of the 92 umbrella concepts was represented by (1) its umbrella label, (2) a short natural-language description synthesized from its constituent traits, and (3) representative trait strings. These concept summaries, together with the formal definitions and examples of P&N, P-NN, and NP described above, were provided to each of the five LLMs. Models were instructed to assume a typical large health-system EHR environment containing both structured data (e.g., demographics, laboratory values, medication orders, diagnosis codes) and commonly available unstructured documentation (e.g., oncology clinic notes, radiology reports, pathology reports, treatment summaries), and to assign each umbrella concept to exactly one predictability category.

For each umbrella, we collected the five model votes and applied the consensus rule described earlier to determine a provisional category and confidence level (Green, Yellow, or Red). All Red-flag concepts (i.e., those lacking a 3-of-5 majority) were then reviewed by an investigator with oncology informatics expertise. For each such concept, the reviewer examined the label, underlying traits, and category definitions, and assigned a final category based on how the information is typically captured in EHRs and whether it is realistically inferable from existing data elements. Green and Yellow concepts were not routinely modified, but obvious misclassifications identified during spot checks were corrected. We have conducted error analysis to identify mismatches across the models and misclassifications.

Results

The multi-model pipeline was applied to eligibility criteria from 107 phase III PD-1/PD-L1 NSCLC trials. LLM-based segmentation and preprocessing yielded 543 unique eligibility traits, which were consolidated into 92 umbrella concepts representing distinct clinical ideas. Across the two main classification steps—trait-to-umbrella mapping and umbrella-to-predictability assignment—inter-model agreement was high. The large majority of traits and umbrellas fell into the Green or Yellow confidence bands, with only a small minority flagged as Red and requiring manual adjudication (Table 2). All Red items were resolved through investigator review; no eligibility concept remained unlabeled.

Table 2.

Agreement across pipeline tasks.

Task	Items	Green (≥4/5)	Yellow (3/5)	Red (<3/5)
Trait-to-umbrella mapping	543	501 (92.3%)	35 (6.4%)	7(1.3%)
Umbrella-to-category classification	92	65 (70.7%)	26 (28.3%)	1 (1.1%)

Final predictability labels were assigned at the umbrella level and propagated to all constituent traits. Across the taxonomy, approximately half of all traits were classified as P&N, with the remainder divided between P-NN and NP (Table 3). A similar pattern held at the umbrella level. P&N umbrellas were dominated by concepts that are central to ICI decision-making but often incompletely represented in structured data, including performance status (ECOG/WHO), disease stage, brain/CNS metastases, prior systemic therapy patterns, autoimmune disease history, interstitial lung disease or pneumonitis, and prior ICI exposure or toxicity. P-NN umbrellas largely captured demographic and routine laboratory criteria (for example, age, organ function thresholds, common viral serologies), whereas NP umbrellas were primarily biomarker- and tissue-related (for example, specific oncogenic driver alterations). Full list of traits is available at: https://github.com/abfnu090/clinical-trial-eligibility-classification/blob/main/data

Table 3.

Final predictability distribution.

Category	Umbrellas	Traits	% of Traits
Predictable & Necessary (P&N)	35	252	46.4%
Predictable but Not Necessary (P-NN)	43	212	39.0%
Not Predictable (NP)	14	79	14.6%
Total	92	543	100%

Model-level voting in the predictability classification step showed systematic differences across the five LLMs (Table 4). GPT-5.1 assigned the largest fraction of umbrellas to P&N. In contrast, Grok 4.1 predominantly favored P-NN. Claude Opus 4.5 and DeepSeek Reasoner showed intermediate patterns, with a majority of votes in P-NN and smaller but non-trivial fractions in P&N and NP. Gemini 3 Pro exhibited the most balanced distribution. These heterogeneous voting profiles contributed to the Yellow-confidence cases (3 of 5 agreement) in the predictability classification phase. However, the majority-vote consensus, combined with manual review of Red cases, yielded a single, reconciled label for each umbrella concept.

Table 4.

Model voting patterns in predictability classification

Model	P&N	P-NN	NP
Claude Opus 4.5	24 (26.1%)	51 (55.4%)	17 (18.5%)
GPT-5.1	56 (60.9%)	25 (27.2%)	11 (12.0%)
Gemini 3 Pro	37 (40.2%)	45 (48.9%)	10 (10.9%)
DeepSeek Reasoner	30 (32.6%)	43 (46.7%)	19 (20.7%)
Grok 4.1	10 (10.9%)	66 (71.7%)	16 (17.4%)

To assess the validity of the consensus labels, we performed a manual chart review–style evaluation on a subset of traits sampled from each confidence tier and compared the model-assigned predictability category with expert judgment from an oncology informatics reviewer. For each of the Green, Yellow, and Red groups, we randomly selected traits, re-examined their underlying eligibility text and umbrella assignments, and independently assigned a reference predictability tier blinded to the model output. This monotonic pattern aligned with the expected relationship between inter-model agreement and correctness, suggesting that the consensus mechanism is not merely capturing arbitrary agreement, but is meaningfully correlated with expert assessment.

Discussion

This work delivers an upstream map of ICI trial eligibility: a set of umbrella concepts derived from heterogeneous criteria text, each labeled with an EHR predictability tier by a multi-model LLM ensemble. The most informative part of the study is not only the overall distribution of categories, but where the framework struggled—those concepts that exposed genuine gray zones in EHR documentation and model reasoning.

Error analysis showed that disagreements clustered in a few recurring areas rather than being random. First, traits with ambiguous temporal scope (for example, “active infection,” “recent significant bleeding,” or “prior malignancy within X years”) consistently split the models. Some implicitly assumed these should be inferred from existing notes, labs, and timelines (leaning toward P&N), while others implicitly treated them as questions clarified at screening (leaning toward PNN). Second, concepts at the boundary between structured and unstructured data, such as the number and type of prior systemic therapy lines, prior radiotherapy, or brain/CNS metastases, were frequent sources of Yellow or Red flags. These reflect real-world uncertainty about how completely such information is captured in structured treatment and staging fields versus being buried in free-text documentation. Third, investigational study participation emerged as an extreme case, with votes spread across all three tiers, highlighting that some eligibility concepts depend more on local documentation habits than on inherent clinical complexity. Model-specific tendencies also contributed to these error modes. Some models had a bias toward labeling many concepts as P&N, while others had a bias toward PNN, especially for borderline cases. The ensemble and confidence tiers (Green/Yellow/Red) did not eliminate disagreement, but they transformed it into a signal: high-confidence concepts were generally stable, while Yellow and Red items reliably indicated where human review and local context are most critical.

In a production setting, several strategies could address these gray zone cases. First, evidence from our prior work suggests that fine-tuning LLMs with domain-specific instructions can effectively resolve many such ambiguities by embedding contextual rules for temporal interpretation and data source expectations directly into the model’s behavior²³. Second, the multi-model consensus structure itself provides a built-in triage mechanism: concepts receiving Green-level agreement (≥4 of 5 models) can be processed automatically with high confidence, while Yellow and Red concepts can be routed to human reviewers for adjudication. This tiered routing would allow production systems to handle the majority of concepts automatically while focusing expert attention on the genuinely ambiguous cases. Combined with a final domain expert review step, this framework offers a practical and scalable approach to managing edge cases without requiring manual adjudication of every eligibility concept.

While a traditional gold standard for EHR predictability classification of eligibility concepts does not yet exist in the literature, several design features of our framework serve analogous validation functions. First, the multi-model consensus approach itself acts as an internal validation mechanism: by requiring agreement across five independently queried LLMs with diverse architectures and training data, the framework reduces the risk of any single model’s biases or errors propagating to the final labels. This ensemble strategy is conceptually similar to multi-annotator agreement in gold standard construction, where convergence among independent raters is used as a proxy for correctness. Second, we performed a blinded expert review in which an investigator with oncology informatics expertise independently classified a stratified sample of traits across confidence tiers and compared these reference labels with the consensus output. The monotonic relationship between confidence tier and expert agreement—with Green-tier concepts showing the highest concordance and Red-tier concepts the lowest—provides empirical evidence that the consensus mechanism captures meaningful signal rather than arbitrary agreement. Prior work on multi-model LLM ensembles for clinical NLP tasks has similarly demonstrated that aggregating outputs across diverse models improves reliability relative to any single model alone^17,18.

This study relies primarily on LLM-based consensus with limited direct clinical validation of the umbrella concept taxonomy. The final list of 92 umbrella concepts and their predictability assignments was reviewed by a domain expert with oncology informatics experience, who adjudicated all Red-flag cases and spot-checked Green and Yellow assignments for clinical plausibility. This review confirmed that the LLM-derived groupings were clinically coherent and that the predictability labels aligned with how these data elements are typically captured in EHR systems. However, a more comprehensive clinical validation process would strengthen the framework. Future work will include a multidisciplinary review panel comprising at least one practicing oncologist and one clinical research coordinator with direct experience in NSCLC immunotherapy trial screening to enable a formal assessment of inter-rater agreement and identification of concepts where clinical judgment diverges from model consensus.

Several future directions follow directly from these findings. First, we will operationalize the P&N taxonomy by linking it to real-world EHR data and building imputation models for each high-priority P&N umbrella concept. By quantifying how frequently these traits are documented in structured and unstructured EHR fields—and how accurately they can be inferred when missing—we aim to convert the current conceptual taxonomy into a concrete, data-driven prescreening pipeline. Second, we aim to evaluate the framework prospectively in trial prescreening workflows, using P&N-driven tools alongside standard manual review to measure effects on workload, time to identify candidates, and concordance with final eligibility decisions. Third, we will extend and stress-test the framework in other disease areas and trial types, and explore richer taxonomies (for example, decomposing predictability into documentation prevalence, structuredness, and dependence on new testing) rather than a single three-tier label. Finally, we envision using these results not just to drive EHR-based prediction but to inform future eligibility design, encouraging criteria that are both scientifically justified and realistically verifiable from routine data, thereby reducing avoidable screening effort and missed opportunities for enrollment.

Several limitations should be acknowledged. The analysis is restricted to phase III interventional trials evaluating PD-1/PD-L1 inhibitors in NSCLC registered on ClinicalTrials.gov, and observed patterns may differ for other phases, cancer types, or non-interventional designs. Predictability assignments reflect consensus judgment from general-purpose LLMs and a single domain expert reviewer, without direct empirical validation against EHR data. Practical performance will depend on local documentation practices, data completeness, and the downstream NLP or machine-learning methods employed. Although the proposed approach mitigates single-model biases, it cannot eliminate shared blind spots across models trained on overlapping corpora. Finally, the study analyzed only English-language eligibility criteria from U.S.-registered trials; applicability to non-English criteria or international trial registries remains untested.

Conclusion

We developed a multi-model LLM framework that transforms free-text ICI trial eligibility into standardized umbrella concepts and assigns each concept to an EHR predictability tier, revealing where routine data can realistically support automated prescreening and where it cannot. By showing that a substantial subset of eligibility traits in phase III PD-1/PD-L1 NSCLC trials are both clinically important and plausibly inferable from existing EHR content, this work provides a concrete, prioritized “target list” for future NLP and prediction models. In doing so, it provides a practical blueprint for designing more computable eligibility criteria and for developing EHR-based tools that reduce the screening burden and expand access to trials for patients whose data already exist in their records.