COPE: Chain-of-Thought Prediction Engine for open-source large language model-based stroke outcome prediction from clinical notes

Abstract

Objective

To develop and evaluate Chain-of-Thought Outcome Prediction Engine (COPE), a reasoning-enhanced large language model, for predicting 90-day functional outcomes after acute ischaemic stroke (AIS) from unstructured clinical notes.

Methods

We included 464 patients with AIS who had discharge summaries and 90-day modified Rankin Scale (mRS) outcomes. COPE uses a two-step chain-of-thought (CoT) framework based on sequential open-source models (LLaMA-3-8B): the first generates intermediate clinical reasoning and the second outputs an mRS prediction. We compared COPE’s performance with GPT-4.1, ClinicalBERT, a structured variable-based machine learning model (XGBoost) and a single-step large language model (LLM) without CoT. Performance was evaluated using mean absolute error (MAE), accuracy within ±1 mRS point (±1 ACC) and exact accuracy (ACC).

Results

COPE achieved an MAE of 1.01 (95% CI 0.92 to 1.11), ±1 ACC of 74.4% (95% CI 69.9% to 78.8%) and ACC of 32.8% (95% CI 28.0% to 37.6%), comparable to GPT-4.1 (MAE, 1.00 (95% CI 0.90 to 1.09); ±1 ACC, 77.9% (95% CI 73.7% to 82.0%); ACC, 32.5% (95% CI 28.0% to 37.4%); p=0.72, 0.11 and 0.96). COPE demonstrated performance comparable to a strong structured-data baseline using XGBoost (MAE, 1.03 (95% CI 0.93 to 1.13); ±1 ACC, 73.9% (95% CI 69.4% to 78.5%); ACC, 33.3% (95% CI 28.5% to 38.2%); p=0.77, 0.87 and 0.89), and outperformed ClinicalBERT and the single-step LLM. Subgroup analyses showed consistent performance across sex and age, with higher error among older patients, those undergoing thrombectomy and those with longer summaries.

Conclusions

COPE, a reasoning-enhanced framework using lightweight, open-source LLMs, achieved performance comparable to a proprietary model and to strong traditional baselines while operating without model retraining or manual feature engineering. It offers an accurate and privacy-preserving solution for outcome prediction from unstructured clinical text.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Accurate prediction of functional outcomes after acute ischaemic stroke is clinically important, but existing approaches typically rely on manually extracted structured variables or supervised models requiring large labelled datasets, limiting scalability and use of unstructured clinical notes.

WHAT THIS STUDY ADDS

• This study demonstrates that a zero-shot, reasoning-enhanced, open-source large language model framework can predict 90-day stroke outcomes directly from discharge summaries with performance comparable to proprietary models and strong structured-data machine learning baselines, without feature engineering or model fine-tuning.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• These findings support the feasibility of privacy-preserving, locally deployable large language models for clinical outcome prediction from routine clinical documentation and may enable broader use of unstructured clinical text in research and future decision-support tools.

Introduction

Predicting patient outcome is an important part of the clinical decision-making process, helpful for patient expectations and resource management.1 Free-text clinical notes, such as discharge summaries, offer comprehensive documentation of a patient’s hospital stay, encompassing clinical narratives, treatments, laboratory and imaging results and follow-up plans. Despite their rich contextual and linguistic detail, these documents remain underused in predictive modelling.2 3 One clinical domain where the utilisation of free-text clinical notes holds particular promise is stroke, which is a leading cause of death globally. Many survivors endure significant functional impairments, which can greatly diminish their quality of life.4 Given the high frequency and morbidity of acute ischaemic stroke (AIS) coupled with a standardised, well-accepted and patient-centric outcome scoring system (the 90-day modified Rankin Scale (mRS)),3–5 AIS outcome prediction is an ideal test case for evaluating methods to predict outcome from free-text clinical notes.5

Conventional approaches to outcome prediction often require manual extraction of structured clinical variables (eg, demographics, baseline stroke severity scores, imaging findings, treatment variables and comorbidities) from unstructured discharge summaries, followed by regression-based or tree-based machine learning models—a time-consuming process that precedes most traditional machine learning (ML) algorithms.6 7 These methods are not only resource-intensive but also risk omitting nuanced clinical context embedded in free-text narratives. Recent advances in natural language processing, including language modelling techniques, have enabled more automated extraction of information from clinical text, improving stroke prediction performance. For example, Sung et al 5 used a domain-adapted transformer model—Bio+Discharge Summary Bidirectional Encoder Representations from Transformers (BERT)—to predict functional outcomes after ischaemic stroke, achieving comparable or superior accuracy to traditional structured-data models. However, many of these approaches rely on supervised fine-tuning, which requires large volumes of annotated clinical data that are often unavailable due to privacy, access and cost constraints.8

Large language models (LLMs) offer a potential solution by enabling zero-shot or few-shot inference directly from raw clinical text.9 While proprietary models such as GPT-4 have demonstrated promise in healthcare applications, their adoption is limited by concerns around data privacy, cost and lack of adaptability.10 11 In contrast, open-source LLMs provide transparent, customisable and cost-effective alternatives that can be deployed locally within institutional infrastructure, enabling inference without external data transfer and allowing protected health information to remain under institutional governance. Despite these advantages, relatively limited work has examined the use of open-source LLMs for clinical outcome prediction in the stroke setting, particularly using reasoning-enhanced or zero-shot frameworks, with most prior studies focusing on information extraction rather than prognostic modelling.12 13

Recent work has demonstrated that LLMs can improve performance on complex tasks through explicit reasoning strategies, such as chain-of-thought (CoT) prompting, which encourages models to generate intermediate reasoning steps before producing a final answer.14 CoT and related reasoning-oriented prompting methods have been reported to improve performance on multistep reasoning tasks across a range of benchmarks by eliciting intermediate reasoning steps.14 15 However, most prior applications of LLM reasoning have focused on general benchmarks or knowledge-based tasks, with limited exploration in clinical outcome prediction from unstructured medical text.

To address this gap, we introduce the CoT Outcome Prediction Engine (COPE), a zero-shot, reasoning-enhanced framework built on the open-source LLaMA-3-8B model and present stroke outcome prediction as a demonstrative use case. A zero-shot design was intentionally adopted to avoid reliance on large, annotated clinical datasets and to enable deployment without site-specific retraining in privacy-constrained clinical settings, allowing outcome prediction without any task-specific fine-tuning. COPE employs a two-step architecture that uses CoT prompting to first generate intermediate clinical reasoning and then produce a structured outcome prediction. This design enhances interpretability and output reliability compared with standard prompting approaches, which often struggle with rigid output formats or limited clinical context. By leveraging open-source LLMs, COPE provides a transparent, scalable and privacy-preserving alternative to proprietary systems. We evaluate its performance against a state-of-the-art proprietary model (GPT-4.1), a conventional language model (ClinicalBERT), a traditional clinical variable-based machine learning model (Clinical ML) and a single-step LLM without reasoning—demonstrating that open-source, CoT-enhanced models can match or exceed the performance of closed systems while avoiding manual feature extraction through prompt-based model guidance and enabling zero-shot prediction without fine-tuning.

Methods

Data collection and preparation

The dataset consisted of 464 patients with AIS, enrolled from a single institutional registry at Stanford University Hospital. All patients were evaluated for thrombectomy eligibility between 2010 and 2023. For each patient, the discharge summary—an unstructured clinical document capturing the patient’s hospital course, diagnostics, interventions and discharge plan—was used as the primary input for modelling. Long-term functional outcomes were assessed using the mRS, an ordinal patient-centred measure of post-stroke disability that ranges from 0 (no symptoms) to 6 (death).16 The mRS score was recorded at a median of 90 days following discharge (range: 60–120 days). Patients who died during the hospitalisation were excluded, as death is explicitly documented in the discharge summary and therefore represents an observed outcome that can be directly extracted from the text, rather than a predictive inference task. We acknowledge that excluding these patients may introduce selection bias and potentially inflate performance estimates, as the model is evaluated only among survivors with available functional outcomes. A flowchart detailing the inclusion and exclusion process is shown in figure 1. Prior to analysis, all discharge summaries were de-identified by removing protected health information, in accordance with Health Insurance Portability and Accountability Act guidelines.

Figure 1.

Flowchart showing patient inclusion and exclusion criteria. Patients were excluded if they lacked a 90-day mRS score (n=937) or experienced in-hospital death (n=61). The remaining 464 patients were included and stratified by their mRS scores, then randomly divided into a prompt exploration set (20%) and a testing set (80%). mRS, modified Rankin Scale; SUH, Stanford University Hospital.

Chain-of-thought Outcome Prediction Engine

Model structure: Figure 2 illustrates the overall framework of COPE and provides an example of the two-step process. COPE is a two-step pipeline built with two off-the-shelf, open-source LLaMA-3-8B-Instruct models17 run purely in inference mode; no additional fine-tuning or weight updates were performed (ie, zero-shot inference). LLaMA-3-8B was selected to balance reasoning capacity, computational efficiency and feasibility of local deployment, enabling single graphics processing unit (GPU) inference while demonstrating sufficient performance for stroke outcome prediction in this study.

Figure 2.

Overview of Chain-of-Thought Outcome Prediction Engine. The discharge summary was first fed into an LLM, using a reasoning prompt to support the outcome prediction task. The output of the first LLM was then fed to the framework’s second LLM step, which used an extraction prompt to derive a single mRS score from the reasoning provided by the first step. The sample discharge summary and reasoning LLM output above represent excerpts of the input (discharge summary text) and intermediary output (reasoning). LLM, large language model; mRS, modified Rankin Scale; NIHSS, National Institutes of Health Stroke Scale; s/p, status post; SVT, supraventricular tachycardia; tPA, tissue plasminogen activator.

The first model, referred to as the ‘reasoning LLM’, receives the entire discharge summary and produces a structured CoT output containing key clinical cues, provisional reasoning and a tentative prognosis. LLaMA-3-8B supports a maximum context window of 8192 tokens. In practice, the vast majority of discharge summaries in our cohort fell within this limit; when inputs exceeded the maximum context length, truncation was applied. The second model, the ‘extraction LLM’, receives this reasoning text and returns a single mRS score between 0 and 6. By decoupling free-form clinical reasoning from final score generation, COPE not only enforces strict output formatting but also improves interpretability and predictive performance.

Prompt design and optimisation: The patient dataset was randomly divided into a 20% prompt-exploration subset, used to refine prompts for both LLMs, and an 80% held-out test set, stratified by mRS scores. Within the exploration subset, candidate prompts were iteratively refined by adjusting instruction specificity and clinical framing (eg, specifying a stroke-care setting and clinician role), clarifying the prediction task and output constraints and including a brief definition of the mRS. The final prompts were selected based on predictive accuracy and the validity of outputs (ie, consistent generation of a single integer mRS score) on the exploration subset and were subsequently evaluated on the independent test set. The final prompt texts and associated details are provided in the online supplemental Material 1.

Ablation studies and comparison models

To evaluate the effectiveness of this framework, we conducted a series of ablation and comparative studies, including comparisons with GPT-4.1, ClinicalBERT and a clinical variable-based machine learning model (Clinical ML). To isolate the contribution of reasoning, we also compared COPE against a single-step LLM without reasoning.

ClinicalBERT baseline: We fine-tuned the publicly available Bio_ClinicalBERT model18 specifically for the mRS score prediction task. Since discharge summaries often exceed the 512-token input limit of BERT-based models, each summary was split into overlapping 512-token segments with a 50-token overlap. Each segment was treated as an independent instance during training. During inference, we applied the same chunking strategy and aggregated the model’s predictions across segments using the method described by Huang et al 3 for scalable readmission prediction.

Clinical ML baseline: To benchmark against traditional structured-data machine learning approaches, we implemented a gradient-boosted decision tree model (XGBoost) trained on structured features extracted from discharge summaries using an LLM-assisted workflow with human verification for consistency. These variables included demographic information (age, sex), prior medical history (stroke, hypertension, diabetes and atrial fibrillation), transfer status, clinical severity scores (baseline, 24-hour and discharge National Institutes of Health Stroke Scale (NIHSS)), relevant laboratory results (glycated haemoglobin and low-density lipoprotein), treatment indicators (eg, intravenous tissue plasminogen activator administration and endovascular thrombectomy (EVT)), procedural and hospital course information (thrombolysis in cerebral infarction score and procedure-related complications), and discharge destination. Missing structured values were handled using rule-based encoding and median imputation. Entries recorded as ‘Missing’ were replaced with the median value of the corresponding feature, while binary indicators (eg, comorbidities and procedure-related complications) were conservatively encoded as 0 (absent) when missing.

GPT-4.1: The closed-source GPT-4.1 model was evaluated using the same optimised prompt and two-step CoT structure developed for the COPE framework. The first step generated intermediate clinical reasoning from the discharge summary, followed by a second step that extracted the final mRS using a structured parsing prompt. This design allows for a controlled comparison by holding the reasoning process and prompt engineering constant. The model was deployed in a secure, in-hospital environment at Stanford University Hospital, maintained by Stanford Medicine’s Technology and Digital Solutions team, in compliance with institutional policies for handling protected health information. GPT-4.1 was evaluated solely as a reference benchmark under institutional governance, whereas COPE is designed for local deployment using open-source models to avoid reliance on proprietary systems.

Single-step LLM (no reasoning): In this baseline, the same open-source LLM used in COPE directly predicted the mRS from the discharge summary without generating intermediate reasoning, allowing evaluation of whether explicit reasoning improves performance beyond a straightforward end-to-end prediction. Prompt exploration was performed using the same exploration subset as COPE to guide the LLM toward strong performance while ensuring it directly output a single integer mRS score in the correct format.

Performance evaluation

To assess model performance in predicting 90-day outcomes, we evaluated mean absolute error (MAE), exact accuracy (ACC) and within-one-grade accuracy (±1 ACC). MAE quantifies the average absolute difference between predicted and reference mRS scores, while ACC reflects the proportion of exact matches. Given the ordinal nature of the mRS, we additionally report ±1 ACC, a commonly used ordinal-tolerance metric in prior stroke outcome prediction studies,19–21 which is reported alongside MAE and ACC as a complementary measure.

Subgroup and sensitivity analysis design

To assess model robustness and potential performance disparities of COPE, subgroup analyses were conducted across demographic, clinical and documentation-related factors. Evaluation was stratified by sex (male vs female), treatment status (endovascular thrombectomy (EVT) vs non-EVT), discharge summary length (quartile-based word count) and patient age group (<46, 46–64, 65–80 and >80 years), with age stratified into quartiles to ensure adequate sample size within each subgroup. For each subgroup, model performance was quantified using MAE with 95% CIs derived via bootstrap resampling. Sensitivity analyses were performed to examine whether model error patterns remained consistent across strata and to identify potential sources of bias in prediction accuracy.

Statistical analysis

To evaluate statistical significance across performance metrics, several tests were employed. A bootstrap-based hypothesis test22 was used to assess differences in MAE, ACC and ±1 ACC. P values<0.05 were considered statistically significant. To correct for multiple comparisons, the Benjamini-Hochberg procedure was applied, controlling the false discovery rate at 0.05. Bootstrap resampling was also used to estimate 95% CIs for all metrics.

Patient and public involvement

Patients and the public were not involved in the design, conduct, reporting or dissemination plans of this research.

Results

Patient characteristics

A total of 1462 patients with AIS were initially identified from the Stanford University Hospital registry. After applying exclusion criteria—specifically, missing 90-day mRS data (n=937) and in-hospital death (n=61)—a final cohort of 464 patients (median age, 70 years; IQR 60–80 years; 254 male patients) was included in the study. Demographic characteristics are summarised in table 1. The dataset was then randomly divided into two subsets: 20% (n=92) for prompt exploration and 80% (n=372) for model testing.

Table 1.

Summary of demographics for all patients with acute ischaemic stroke (n=464) included in the study

		Prompt exploration set (n=92)	Testing set (n=372)
Male		48 (52.2)	206 (55.4)
Age, years Median ( IQR )		71 (60–81)	70 (60–79)
Hypertension		60 (65.2) 4.3 %	243 (65.3) 0.5%
Diabetes		27 (29.3)	88 (23.7) 1.1%
Baseline NIHSS Median ( IQR )		15 (8–20)	14 (9–20) 0.5%
IV-tPA		35 (38.0) 4.3%	177 (47.6) 2.9%
EVT		76 (82.6)	294 (79.0)
mRS scores at 90 days3	0	10 (10.9)	42 (11.3)
	1	14 (15.2)	56 (15.1)
	2	11 (12.0)	45 (12.1)
	3	21 (22.8)	83 (22.3)
	4	19 (20.7)	75 (20.2)
	5	7 (7.6)	30 (8.1)
	6	10 (10.9)	41 (11.0)

Unless otherwise stated, data are presented as the number of patients and their corresponding percentage of the total.

Grey-shaded percentages indicate the proportion of missing values for the corresponding variable in each dataset split. If no data are missing, no grey percentage is shown.

EVT, endovascular therapy; IV-tPA, intravenous tissue plasminogen activator; mRS, modified Rankin Scale; NIHSS, National Institutes of Health Stroke Scale.

Performance analysis

Table 2 presents a comparative evaluation of models for stroke outcome prediction, including GPT-4.1, Clinical BERT, a structured-data machine learning baseline (XGBoost), a single-step LLM and the proposed COPE framework. The COPE model achieved an MAE of 1.01 (95% CI 0.92 to 1.11), ±1 ACC of 74.4% (95% CI 69.9% to 78.8%) and exact ACC of 32.8% (95% CI 28.0% to 37.6%). These results were statistically indistinguishable from those of GPT-4.1 (MAE=1.00 (95% CI 0.90 to 1.09); ±1 ACC=77.9% (95% CI 73.7% to 82.0%); ACC=32.5% (95% CI 28.0% to 37.4%); p=0.72, 0.11 and 0.96, respectively).

Table 2.

Performance of different models for stroke outcome prediction. Comparisons include a state-of-the-art proprietary large language model (GPT-4.1), a conventional language model (ClinicalBERT), a structured clinical variable-based machine learning model (XGBoost) and the proposed COPE. Performance metrics include MAE, accuracy within ±1 mRS point (±1 ACC) and exact accuracy (ACC)

Model type	MAE	±1 ACC (%)	ACC (%)
GPT 4.1	1.00 (0.90 to 1.09) p = 0.72	77.9 (73.7 to 82.0) p = 0.11	32.5 (28.0 to 37.4) p = 0.96
Clinical BERT	1.24 (1.13 to 1.36) p < 0.001	64.5 (59.7 to 69.4) p < 0.001	28.5 (23.9 to 33.1) p = 0.13
XGBoost	1.03 (0.93 to 1.13) p = 0.77	73.9 (69.4 to 78.5) p = 0.87	33.3 (28.5 to 38.2) p = 0.89
Single-step LLM	1.20 (1.09 to 1.33) p < 0.001	67.5 (62.9 to 72.0) p < 0.001	29.7 (25.4 to 34.0) p = 0.09
COPE	1.01 (0.92 to 1.11)	74.4 (69.9 to 78.8)	32.8 (28.0 to 37.6)

Lower MAE and higher accuracy values indicate better performance. Values in parentheses indicate 95% CIs. P values were calculated using bootstrap resampling to assess statistical significance relative to the COPE model.

Bold values indicate the best-performing result for each evaluation metric.

ACC, exact prediction accuracy of the mRS score; ±1 ACC, prediction accuracy within±1 mRS score; BERT, Bidirectional Encoder Representations from Transformers; COPE, Chain-of-Thought Outcome Prediction Engine; LLMs, large language models; MAE, mean absolute error.

COPE also demonstrated performance comparable to a strong structured-data baseline using XGBoost (MAE=1.03 (95% CI 0.93 to 1.13); ±1 ACC=73.9% (95% CI 69.4% to 78.5%); ACC=33.3% (95% CI 28.5% to 38.2%); p=0.77, 0.87 and 0.89, respectively), with overlapping CIs across all metrics, despite operating in a zero-shot setting without any model retraining or feature engineering. In contrast, Clinical BERT exhibited higher prediction error and lower accuracy (MAE=1.24 (95% CI 1.13 to 1.36); ±1 ACC=64.5% (95% CI 59.7% to 69.4); ACC=28.5% (95% CI 23.9% to 33.1%); p<0.001 for MAE and ±1 ACC; p=0.13 for ACC). The single-step LLM baseline also underperformed relative to COPE (MAE=1.20 (95% CI 1.09 to 1.33); ±1 ACC=67.5% (95% CI 62.9% to 72.0%); ACC=29.7% (95% CI 25.4% to 34.0%); p<0.001 for MAE and ±1 ACC; p=0.09 for ACC).

Two samples of discharge summaries and their corresponding model outputs are shown in the online supplemental Material 2. A per-class error analysis using a row-normalised confusion matrix is provided in online supplemental Material 3, illustrating the distribution of under-prediction and over-prediction across mRS categories.

Subgroup and sensitivity analyses of COPE

Subgroup analysis examined model error across demographic, clinical and documentation strata, including sex, treatment status, note length and age (figure 3). Performance was similar between males (MAE, 0.99; 95% CI 0.86 to 1.12; n=206) and females (MAE, 1.04; 95% CI 0.90 to 1.18; n=166), with overlapping 95% CIs. By treatment status, non-EVT cases showed lower error (0.80; 95% CI 0.62 to 0.99; n=84) compared with EVT cases (1.07; 95% CI 0.97 to 1.18; n=288). Error increased with note length across quartiles: Q1 <1958 words (0.87; 95% CI 0.69 to 1.05), Q2 1958–2389 words (1.03; 95% CI 0.86 to 1.21), Q3 2390–2843 words (1.06; 95% CI 0.86 to 1.27) and Q4 ≥2843 words (1.09; 95% CI 0.90 to 1.28). By age, MAE was 1.03 (95% CI 0.70 to 1.40; n=30) for <46 years, 0.90 (95% CI 0.76 to 1.05; n=106) for 46–64 years, 0.91 (95% CI 0.77 to 1.07; n=153) for 65–80 years and 1.32 (95% CI 1.11 to 1.55; n=83) for >80 years, indicating the highest error among the oldest patients. These trends suggest that increased clinical complexity and documentation burden are associated with higher prediction error, a pattern further examined in the Discussion.

Figure 3.

Mean absolute error and 95% CIs are shown across subgroups defined by sex, treatment status, documentation length (quartiles of note word count) and patient age. Model error was similar between male and female patients but higher among those who underwent EVT compared with non-EVT cases. Prediction error increased with longer documentation length and was highest among patients older than 80 years, suggesting that clinical complexity and documentation burden contribute to reduced model accuracy. EVT, endovascular thrombectomy.

Discussion

We developed and evaluated COPE, a CoT reasoning–enhanced LLM framework for predicting patient outcomes from free-text clinical notes, demonstrating its performance for stroke outcomes based on discharge summaries. Outcome prediction is a fundamentally important task, with benefits to patients, clinicians, resource managers and researchers and methods to perform this using free-text, unstructured clinical notes remain underexplored.

The open-source COPE approach demonstrated performance comparable to a state-of-the-art proprietary LLM, GPT-4.1, across multiple outcome prediction metrics. Notably, this performance was achieved using LLaMA 3–8B, a substantially smaller open-source model with approximately 60–100×fewer parameters and lower computational requirements than GPT-4.1. These results highlight the effectiveness of task-specific design choices—such as incorporating CoT-based prompting—in enabling strong performance from lightweight models. Beyond predictive accuracy, COPE is built on open-source architectures and explicit prompt-based reasoning, which support reproducible experimentation and facilitate alignment with privacy-conscious deployment constraints common in clinical settings.

Unlike traditional single-step prompting, COPE separates the reasoning process into two sequential stages: first generating an intermediate clinical rationale, then producing the final prediction. This two-step design consistently outperformed the single-step baseline, indicating that prompting the model to reason explicitly through the clinical context before prediction enhances both accuracy and robustness. During development, we observed that single-step LLMs often produced free-text or mixed-format outputs when given detailed clinical context or reasoning instructions, even when explicitly instructed to return only an integer between 0 and 6. Only highly constrained prompts reliably yielded valid outputs (see online supplemental Material 1 for an illustrative example). Paradoxically, more elaborate prompts, which often improve reasoning in other domains, degraded performance in this structured clinical prediction setting. These findings underscore a fundamental limitation of single-step LLMs: they struggle to balance complex reasoning with rigid output constraints. By decoupling reasoning from prediction, the COPE framework addresses this challenge, producing more reliable and clinically actionable predictions.

Previous approaches to outcome prediction have primarily relied on clinical variable-based machine learning models.23–25 In comparison, our LLM-based framework achieved performance that was comparable to, and in some cases modestly better than, strong structured-data baselines, while operating in a zero-shot setting without manual feature extraction or model retraining. This competitive performance likely stems from the model’s ability to process rich, unstructured clinical narratives, which provide contextual cues and nuanced insights beyond what structured data alone can offer. Traditional models depend on manually extracted clinical features—a time-consuming and labour-intensive process that limits scalability. In contrast, our approach leverages open-source, pretrained LLMs to perform representation learning directly from unstructured clinical text, avoiding manual feature extraction while using prompt design and refinement to guide how the model performs the prediction task. We did not include long-context encoder-only models (eg, GatorTron26) as comparators, as these architectures are primarily designed for supervised fine-tuning and representation learning rather than zero-shot, generative reasoning from unstructured clinical text, which was the focus of this study.

We found that COPE’s prediction errors were concentrated in subgroups characterised by higher clinical complexity and greater documentation burden. The higher error observed among EVT cases likely reflects case-mix and outcome heterogeneity: patients undergoing EVT are typically more severely affected (higher baseline NIHSS), experience variable reperfusion success and are at risk for peri-procedural complications—all of which broaden the range of possible outcomes and increase prediction difficulty. Similarly, model performance decreased with longer discharge notes, which may be associated with medical case complexity and comorbidities, procedural templates, radiology addenda and multiservice narratives that may dilute or obscure salient clinical cues within the model’s effective context window—a limitation noted in prior LLM-based clinical applications.27 Finally, the higher error among patients older than 80 years may reflect wider variability in recovery trajectories, frailty, comorbidities and discharge disposition, all of which likely contribute to less predictable functional outcomes.

Our study has several limitations. First, the models were evaluated using data from a single centre, which may limit generalisability to other healthcare settings with different patient populations, clinical workflows or documentation styles. However, the dataset encompasses diverse clinical presentations and real-world discharge documentation, supporting its relevance as a proof of concept. Future multicentre validation will be essential to confirm external generalisability and evaluate performance across heterogeneous practice environments. Second, the quality of prompt engineering likely influenced how the large language model made predictions, playing a critical role in optimising stroke outcome estimation. In this study, prompt design was largely manual and guided by iterative trial and error, reflecting the current limitations of prompt standardisation in clinical applications. This effort represents a non-trivial form of engineering and should be considered a trade-off relative to traditional feature engineering. Because prompts encode assumptions about clinical language and documentation structure, model performance may be sensitive to differences in note style across institutions, requiring prompt adaptation in new settings. Developing systematic or automated prompt optimisation approaches may help reduce reliance on manual tuning and improve consistency in model reasoning. Third, after applying inclusion and exclusion criteria, the final cohort was relatively small and derived from a single centre, which may limit representativeness and generalisability to broader stroke populations. As in routine clinical practice, 90-day mRS outcomes were not available for all eligible patients due to loss to follow-up or refusal, further contributing to sample size reduction. Accordingly, the findings should be interpreted as a proof-of-concept evaluation of the proposed framework rather than definitive evidence of population-level performance. Lastly, the study did not assess clinician interaction with the model or its interpretability in clinical decision-making. Evaluating usability and trust among care providers will be critical for future translation. In addition, practical deployment would require demonstration in different medical environments, integration with electronic health record systems, alignment with clinical workflows and prospective validation to ensure safe and effective use in real-world settings.

In conclusion, we developed and evaluated COPE, a reasoning-enhanced large language model framework built on LLaMA-3-8B, which demonstrated strong performance in predicting 90-day stroke outcomes from free-text clinical discharge summaries. By integrating a two-step CoT architecture, COPE achieved performance comparable to the state-of-the-art proprietary model GPT-4.1 and to a strong structured-data baseline, XGBoost, while remaining lightweight and privacy-preserving. These findings underscore the potential of open-source, reasoning-based LLMs to enable transparent and scalable clinical outcome prediction from unstructured text.

Funding Statement

This work was supported by the National Institute of Neurological Disorders and Stroke (NINDS), National Institutes of Health, US Department of Health and Human Services (grant number K99NS140547).

Data availability statement

Data may be obtained from a third party and are not publicly available. The data used in this study were obtained from Stanford University Hospital and contain protected health information. Due to institutional and ethical restrictions, these data are not publicly available. Access may be considered upon reasonable request to the corresponding author and subject to appropriate institutional approvals.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

This study was conducted in accordance with the United States Health Insurance Portability and Accountability Act (HIPAA) of 1996 and was approved by the local Institutional Review Board (IRB-77549). In compliance with local IRB guidelines, as this is a retrospective study using deidentified clinical notes, the requirement for patient consent was waived by the Institutional Review Board.