Understanding uncertainty in large language model predictions of early death in critically ill patients: a conformal prediction approach

Abstract

Background

Early prediction of in-hospital death remains a significant challenge due to the limited availability of structured data during initial admission. Unstructured clinical notes, which often contain important observations and impressions, are an underutilized resource for real-time risk stratification. While leveraging recent advances in large language models (LLMs) is a promising approach to use this unstructured information, the lack of understanding of the uncertainty of LLM predictions, at the patient level, for such critical forecasts is a serious deterrence for their use in clinical settings.

Objective

This study aims to demonstrate the use of conformal prediction to evaluate the effectiveness and confidence in predicting in-hospital death probability for an individual patient using LLMs, specifically GPT-4o and unstructured clinical notes.

Methods

We applied conformal prediction to quantify the uncertainty of GPT-4o’s zero-shot predictions for in-hospital death, leveraging concatenated clinical notes documented from the first 24 h of intensive care unit (ICU) admission in MIMIC-III for patients with acute kidney failure who were admitted through the emergency department (ED).

Results

Across both classes “in-hospital death” and “in-hospital survive,” the GPT model performed better on the in-hospital death class, achieving precision of 0.52 (95% CI 0.48-0.56), recall of 0.93 (95% CI, 0.90-0.95), and F1-score 0.66 (95% CI, 0.63-0.70). The conformal prediction (CP) framework provided an overall empirical coverage of 90.4%, exceeding the target threshold of 90%. However, class-specific coverage was imbalanced, with 99.7% for the death and 81.1% for the survived class.

Conclusions

The model’s outputs exhibit overconfidence, particularly in cases of incorrect predictions. Integrating conformal prediction provides a promising approach to quantifying and calibrating uncertainty in large language model outputs for individual patient predictions, thereby enhancing their potential applicability for clinical decision-making.

Introduction

In the early hours following emergency admission, resource constraints and sparse structured data make continuous risk assessment difficult, particularly for patients at risk of life-threatening organ dysfunction or unexpected death.^1–4 Predicting death among critically ill patients is also valuable for evaluating the effectiveness of treatments, clinical guidelines, surgeries, and other interventions using electronic health record (EHR) data.⁵ These records typically include demographic details, physiological data, early lab results, clinical observations, and treatment information.^6,7 Various scoring systems, including Emergency Severity Index,⁸ APACHE (Acute Physiology and Chronic Health Evaluation),⁹ Death Probability Models,¹⁰ and the Sequential Organ Failure Assessment (SOFA) score¹¹—have been developed to assess illness severity and predict patient outcomes. However, despite their widespread use, these models often have limited accuracy and are primarily used as benchmarking tools.^12,13 Moreover, the relative accuracy varies substantially among patients and methods to quantify prediction reliability at the patient level remain poorly established.

Machine learning (ML) has promised better performance by leveraging electronic health records (EHRs) and publicly available databases such as Medical Information Mart for Intensive Care (MIMIC-III)¹⁴ and High Time Resolution ICU Dataset (HiRID),¹⁵ and community challenges have showcased ML’s ability to learn complex, nonlinear relationships among physiological variables^16,17 and predict patient outcomes, including ICU death. However, many ML pipelines rely on curated, structured input features and assume stable data availability, conditions often unmet during early stages of patient care.^18,19 Moreover, ML systems typically fail to provide clinicians with calibrated, case-level uncertainty estimates, making it difficult to determine when predictions are trustworthy.^20,21 Large language models (LLMs) change this equation by circumventing restrictive data assumptions and directly consuming unstructured clinical notes, often the richest source of information available in the first 24 h, while requiring minimal task-specific training.^22–25 Unlike traditional ML, LLMs can capture nuanced clinical context through advanced language comprehension.^26–34 Models like the Generative Pre-trained Transformer (GPT) have shown promise in this space, performing at or above human level on standardized medical licensing exams and demonstrating the ability to extract relevant insights from complex clinical documentation.²⁹ Their success is largely attributed to their vast pretraining data, scale of parameters, and fine-tuning through reinforcement learning with human feedback (RLHF),³⁰ making them a compelling addition to predictive modeling in healthcare.^22,35–43 Building on this, authors in Williams et al³³ evaluated GPT for real-world ED acuity assessment and found that a general-purpose model could approximate clinician triage decisions, while authors in Sarbay et al³⁴ assessed GPT in scenario-based ED triage tasks and reported feasible but variable performance These studies underscore the growing applicability of LLMs in emergency care settings, where rapid, text-based clinical reasoning can augment early decision support. Extending this direction, our study leverages LLMs to predict in-hospital death based solely on unstructured clinical notes documented within the first 24 h of an ICU patient’s admission, aiming to enable early risk stratification and clinical prioritization in critically ill patients admitted through the ED. Prior work has explored death prediction using structured data and text-derived variables from the first 48 h of hospitalization²³ and compared performance of a few traditional ML models, with and without unstructured text, and reported improved discrimination when including manually engineered text features. Another study²⁴ relied exclusively on notes and traditional ML pipelines to predict short-, mid-, and long-term ICU death, using feature extraction methods like frequency-based and embedding-based representations. In contrast, our work evaluates a general-purpose, non-fine-tuned LLM model operating in a zero-shot setting, consuming raw clinical narratives without any handcrafted preprocessing or retraining. Moreover, since LLMs are known to occasionally produce information that is not factually accurate or grounded in reality, commonly referred to as “hallucinations”,⁴⁴ quantitatively assessing the reliability of their outputs at the patient level is critical. To responsibly support clinical decision-making, we must quantify the uncertainty associated with LLM predictions at the patient level.²¹ To quantify this uncertainty, we propose to adopt the conformal prediction (CP) framework, a model-agnostic, distribution-free approach that offers rigorous coverage guarantees.^45–47 CP allows us to set a desired error rate (α) and generates prediction sets that contain the true label with a guaranteed level of confidence. By using nonconformity scores—such as $1-f{\left(X\right)}_Y$, where $f{\left(X\right)}_Y$ is the softmax score for the true label⁴⁸ CP provides a practical and interpretable method to express and control the uncertainty in LLM-generated predictions, which is essential for supporting trustworthy clinical decision-making. Using nonconformity scores, CP yields interpretable error control at the patient level, directly addressing the gap left by traditional scores and standard ML.

Building on this rationale, we demonstrate the use of conformal prediction to evaluate the effectiveness and confidence of an LLM (GPT-4o) when predicting in-hospital death from clinical notes recorded within the first 24 h of admission and, critically, whether CP can quantify and communicate prediction uncertainty in a clinically meaningful way. Unlike prior work that relies on structured data or lacks calibrated uncertainty, our approach operates directly on unstructured intensive care unit (ICU) clinical notes from the first 24 h of care for patients admitted through the ED, providing explicit, controllable error guarantees via conformal prediction. To our knowledge, no prior study has jointly evaluated GPT-4o for this task and paired it with a CP framework to provide calibrated, patient-level uncertainty on unstructured notes.

Methods

Dataset

The dataset used in this study is MIMIC-III, a widely recognized and comprehensive repository of de-identified health data.¹⁴ It encompasses clinical information from approximately 60 000 intensive care unit patients treated at Beth Israel Deaconess Medical Center in Boston, Massachusetts, across nearly a decade. This dataset includes a wide range of information, such as electronic health records, laboratory results, medication prescriptions, and clinical notes. As a first step to demonstrate the application of our methods, we focused on patients who were initially admitted through the emergency department, with an admission type labeled as “Emergency,” admission location recorded as “Emergency Room Admit” and an initial diagnosis of Acute kidney failure (584.9). We centered our study on patients diagnosed with acute kidney failure because this condition is both clinically significant and prevalent among critically ill patients, often serving as a marker of multi-organ dysfunction and poor prognosis. Early detection of deterioration in this population is particularly valuable, as timely interventions such as renal replacement therapy or hemodynamic optimization can meaningfully alter outcomes. Moreover, acute kidney failure’s heterogeneous presentation in unstructured clinical notes provides a rigorous and challenging test case for evaluating language models’ ability to extract prognostic signals—and, through conformal prediction, to quantify uncertainty at the patient level—from narrative documentation within the first 24 h of ICU care (following ED admission). For these patients, we retrieved the clinical notes documented within the first 24 h of ICU patient admission. All available notes from this period were concatenated and submitted to the LLM. For consistency in the following sections, we will address this concatenated set of clinical notes as the “clinical note.” The final study cohort consisted of 4714 unique admissions, among whom 910 patients (19.3%) died during their hospital stay, forming the “in-hospital death” class, while 3804 patients (80.7%) survived, forming the “in-hospital survived” class. To address the class imbalance in the dataset, we under-sampled the patients who survived to create a balanced cohort. After removing the outliers, the final cohort consisted of 1354 records. To maintain balance in the calibration and test sets for conformal prediction, we performed a stratified split followed by majority class under-sampling where necessary.

Conformal prediction

Most current CP methods for LLMs depend on access to model logits to compute nonconformity scores. Model logits are the raw, unnormalized output scores that reflect the model’s confidence in each possible token and are produced by the model before applying the softmax function to obtain probabilities. Authors in Kumar et al⁴⁹ defined nonconformity scores using softmax outputs over logits for multiple-choice question answering (MCQ) tasks, while references^50,51 apply the conformal risk control framework, a generalization of CP, to LLMs by leveraging log-probabilities derived from the model. However, for API-only LLMs, end users typically do not have access to model logits. Even when logits are available, they are often misscalibrated, which can impair the effectiveness of CP by yielding unreliable prediction sets or intervals, often resulting in large set sizes and thus reduced efficiency.^52–54 The framework introduced in Lee et al⁵⁵ offers a complementary approach by avoiding logit dependence altogether and instead quantifies uncertainty using entropy calculated from model-assigned confidence scores. This entropy-based method enabled staged decision-making with selective human oversight.

In our work, we implemented the framework introduced in Su et al,⁵⁴ with a slight modification. To implement conformal prediction in the absence of direct access to model logits, we initially sampled 30⁵⁴ randomly independent responses for each patient input and used the empirical frequency of each response as a coarse-grained measure of uncertainty. The sampling rate of 30 responses per input was selected following prior work,⁵⁴ which acknowledges that while thousands of samples are theoretically required to approximate true predictive probabilities with high precision, such exhaustive sampling is computationally infeasible. As a result, a smaller, tractable number, such as 30, offers a pragmatic balance between uncertainty estimation and computational efficiency. For each prediction, the LLM reports 1 answer, either “Yes” or “No,” but this frequency-based approach introduces a key limitation: multiple responses may share the same frequency count despite differing in underlying uncertainty levels.⁵⁴ This results in a concentration of nonconformity scores on a limited set of values, thereby reducing the efficiency of the resulting prediction sets. To address this limitation, we enhanced the approach by capturing the top 10 log-probabilities generated for each sample.

Figure 1 shows an overview of the proposed conformal prediction pipeline, divided into 3 main stages: Prompting & Sampling, Post-Processing, and Nonconformity Score. In the first stage, concatenated clinical notes documenting the first 24 h of admission per patient are combined with a structured prompt and submitted to GPT-4o. From this point onward, any reference to a “note” refers to this concatenated set of notes from the first 24 h of admission for each patient. To approximate uncertainty without access to internal logits, the model is queried $R=30$ times, receiving 30 randomly independent responses per patient $i$ input. Each response is stored together with the top-K = 10 candidate tokens $\left\{t_{ir1},\dots , t_{\mathit{irK}}\right\}$predicted at the response position and their associated log-probabilities $\left\{l_{ir1},\dots , l_{\mathit{irK}}\right\}$. We convert and (slightly) renormalize the probabilities. The cumulative sum of top 10 probabilities consistently reached a threshold of .99, enabling uniform and reliable normalization across all samples.

Figure 1.

Overview of the conformal prediction pipeline. (Prompting & Sampling) Concatenated ICU notes from the first 24 h of admission are prompted to GPT-4o. Each note is queried 30 times; for every response, the top 10 tokens and log-probabilities are captured, exponentiated, and normalized. (Post-Processing): Tokens are lower-cased, mapped to classes via indicator functions, and aggregated class probabilities are calculated for each response. (Nonconformity Score): For each response, class-specific nonconformity scores are computed.

$$p_{\mathit{irj}}=\text{exp}\hspace{0.17em}\left(l_{\mathit{irj}}\right),\mathrm{ }{S}_{ir}=\sum _{j=1}^K{p}_{\mathit{irj}}\mathrm{ }\left(\mathit{empirically}\mathrm{ }{S}_{ir}\ge 0.99\right),\mathrm{ }{q}_{\mathit{irj}}=\frac{p_{\mathit{irj}}}{S_{ir}}.$$

To ensure consistency during post-processing, we normalized the tokens by converting all candidates to lowercase. In the post-processing stage, probabilities are converted into class-level probabilities by mapping each token to a class using indicator functions:

$${\chi }_Y\left(t\right)=1\left\{”\text{yes}\mathrm{”}\mathrm{ }in\mathrm{ }t\right\},\mathrm{ }{\chi }_N\left(t\right)=1\left\{”\text{no}\mathrm{”}\mathrm{ }in\mathrm{ }t\right\},\mathrm{ }{\chi }_O\left(t\right)=1\left\{\text{neither} “\text{no}\mathrm{”}\mathrm{ }\mathit{nor}\mathrm{ }\mathrm{“}\mathit{yes}\mathrm{”}in\mathrm{ }t\right\},$$

$$P_{ir}^{\left(c\right)}=\sum _{j=1}^K{q}_{\mathit{irj}}\mathrm{ }{\chi }_c\left(t_{\mathit{irj}}\right),\mathrm{ }c\in \left\{Y,N,O\right\}.$$

Specifically, we computed the total probability mass assigned to responses containing the word “yes” as the probability of the Y=“in-hospital death” class, and similarly aggregated probability mass assigned to responses containing the word “no” for the N=“in-hospital survived” class. Probabilities for responses not containing either keyword have been aggregated and, for illustration purposes in Figure 1, are represented under a class named O=“other.” It is worth noting that we have only 2 main classes in this task: “in-hospital survived,” illustrated as “No” in this figure, and “in-hospital death,” illustrated as “Yes.” The high probabilities are concentrated within the top 10 tokens, cumulatively summing to approximately .99, while the remaining token probabilities are negligibly small. This confirms that the probability mass is effectively captured within the top 10 candidates. The one-word point label for that query is:

$$\widehat{y_{ir}}={{\arg \mathit{max}}_{c\in \left\{Y,N\right\}}P}_{ir}^{\left(c\right)}.$$

In the final stage, nonconformity scores $s_{ir}^{\left(c\right)}={1-P}_{ir}^{\left(c\right)}$ are computed for each prediction. This method provided a more fine-grained and probabilistically grounded estimate of uncertainty while remaining compatible with API-based LLMs.

LLM methods

In this study, we used the GPT-4o model for generative question answering.⁵⁶ We accessed this model through the chat completions API endpoint in a zero-shot setting. In zero-shot prediction, a model is presented with tasks or queries for which it has not received explicit training. It is expected to extrapolate knowledge from its pre-existing understanding of language and context to generate meaningful responses.

Since prompt engineering is essential when interacting with any LLM to obtain high-quality responses, we first experimented with and formulated prompts to elicit the desired responses from the model. The examined prompts are provided in Appendix S1. Then we used our finalized prompt in a zero-shot prediction setting. Our finalized prompt was selected to be as follows:

You are a clinical risk prediction model. I will provide you with a clinical note recorded on the first day of a patient’s hospital admission. Based solely on the information provided in the note, your task is to predict whether the patient is at high risk of in-hospital death during their current admission. Please carefully analyze the content of the note for clinical indicators such as vital signs, symptoms, laboratory results, and any other documented clinical findings that might signal a critical condition. Your response must be exactly one word: "Yes" if the note suggests that the patient is at risk of in-hospital death, or "No" if it does not. Do not include any additional text or explanation: <clinical note text here>.

Those clinical notes for which the LLM answers “Yes” will be associated with the “in-hospital death” class, while those receiving a “No” answer will be associated with the “in-hospital survived” class. To align with emerging standards for transparency and reproducibility in LLM research, we followed the checklist introduced in Tripathi et al,⁵⁷ which provides structured guidance for reporting applied studies using pretrained models with minimal modifications.

Evaluation was conducted on a balanced dataset, and calibration was achieved using an α of 0.1 to ensure 90% coverage. The calibration set was constructed by randomly selecting 50% of the cohort, with the remaining 50% used for testing.

Results

For the LLM-only (pre-CP) evaluation, aggregating performance across both classes (in-hospital death and survival), the GPT model achieved an overall accuracy of 53%, precision of 58%, recall of 53%, and an F1 score of 44%. When stratified by class, recall was 93% for in-hospital death and 13% for in-hospital survival, showing imbalance in class-specific sensitivity. This observed asymmetry indicates that the model preferentially flags high-risk (death) cases, an operating point that can be beneficial for early warning in high-risk critical‐care settings (as in our cohort of ED-admitted patients with acute kidney failure requiring ICU care).

To assess potential demographic biases in our dataset, we examined the distribution of in-hospital death across ethnicity, gender, and marital status, as summarized in Appendix S2.

The distribution of token counts per patient note exhibited substantial variability across the cohort. Overall, the average number of tokens per patient was 1321 for those who died during hospitalization and 1707 for those who survived. The standard deviations were also large, 909 and 2022 respectively, indicating a wide spread in note lengths. Notably, patients who survived had longer clinical notes on average, suggesting that survival cases may involve more extensive documentation.

When examining classification performance by subgroup, correctly classified death cases had a mean token count of 1376 (SD = 906), while misclassified ones had significantly fewer tokens (mean = 461, SD = 362). An independent samples t-test confirmed the significance of this difference (t = 6.35, P = 3.88 × 10⁻¹⁰), suggesting that longer notes are associated with more accurate death predictions, likely due to the richer clinical detail they provide. In contrast, among patients who survived, correctly classified cases had a mean of 922 tokens (SD = 1537), whereas misclassified survival cases, those incorrectly predicted, had substantially longer notes (mean = 1812, SD = 2057). This difference was also statistically significant (t = −3.76, P = 1.87 × 10⁻⁵), with the negative t-statistic indicating that excessive documentation may contribute to false positives. These findings suggest that while longer notes generally aid prediction in death cases, they may introduce ambiguity in survival cases. To probe this ambiguity, we conducted a content-focused review of both short and long notes across survival cases and compared correctly and incorrectly classified examples. Among short notes of comparable length, correctly predicted survivors emphasized a coherent recovery trajectory (eg, weaning off vasopressors, improved oxygenation, and stable hemodynamics), whereas misclassified survivors contained mixed or competing severity cues (persistent hypoxemia/wheezing, rising creatinine, tachyarrhythmia, marked hyperglycemia) that likely elevated perceived risk. Extending this analysis to long survival notes misclassified as death cases, we observed dense clusters of high-acuity terminology (eg, sepsis, aspiration/pneumonia/atelectasis, massive hemoptysis), intervention-heavy narratives (intubation, paralytics, vasopressors, blood products), and repeated radiology blocks with hedged differentials (“versus,” “could reflect,” “cannot be excluded”). These features amplify perceived risk, particularly when temporal resolution is unclear and improvement cues are sparse, transient, or embedded within high-acuity context; comorbidity priors (eg, metastatic disease) and copy-forward repetition further magnify severity signals, while negated or qualified improvements are linguistically weaker and appear underweighted by the model. Collectively, these observations suggest that semantic coherence and a resolved clinical trajectory, rather than length per se, are the principal determinants of correct classification; notes that clearly establish either improvement or deterioration are classified accurately regardless of length, whereas those mixing competing severity cues or lacking temporal resolution—particularly when lengthy—are prone to misclassification.

For the conformal prediction framework, an alpha value of 0.1 was chosen, targeting 90% coverage. This calibration distribution is used to determine the nonconformity threshold that ensures prediction sets on unseen data will include the true label with at least 90% probability.

Figure 2 presents the histogram and density plots of nonconformity scores derived from GPT-4o responses on the calibration set. Each nonconformity score reflects how uncertain the model was about the correct label, computed as $1-p\left(\mathrm{True}\mathrm{ }\mathrm{class}\right)$, based on the class probability derived from the top 10 log-probabilities across 30 sampled responses per patient. The plot on the left shows the distribution of scores for patients who survived (in-hospital survived class), while the right plot shows the scores for patients who died (in-hospital death class). In the case of patients who died, the majority of nonconformity scores are clustered near zero. This indicates that the model assigned high confidence (ie, high probability) to the correct class label. Conversely, for patients who survived, nonconformity scores are generally higher, with a notable shift toward 1.0. This shows that the model often assigned low probability to the correct survival label, implying higher uncertainty or overconfidence in incorrect predictions.

Figure 2.

Distribution of nonconformity scores by outcome class in the calibration set, with blue indicating survival and orange indicating in-hospital death.

Figure 3 shows the relationship between the length of clinical notes (in tokens, y-axis) and the corresponding nonconformity scores (x-axis) for each patient in the calibration set, stratified by in-hospital death outcome. The distribution of note categories recorded within the first 24 h of hospital admission is as follows: “Radiology” (40.48%) and “Nursing/other” (31.97%) categories dominate the dataset and collectively account for over 70% of all early documentation. This highlights the intensive use of diagnostic imaging and nursing observations in the immediate phase of patient assessment and care. The next most common category of notes are “Nursing” (14.36%) and “Physician” (10.46%). Categories “Nursing” and “Nursing/other” capture vitals, observations and symptom tracking, while physician notes contain diagnostic and treatment plans. The remaining note categories, including “General,” “Respiratory,” “Case Management,” “Consults,” “Nutrition,” “Pharmacy,” “Rehab Services,” and “Social Work,” make up a small fraction of the early documentation. This reflects their limited involvement during the initial 24-h window, which is typically focused on acute medical assessment and stabilization. Consistent with previous findings, the plot reveals that nonconformity scores for patients who died tend to be low across a range of note lengths, indicating that the model often assigns high probability to the correct death prediction in these cases. In contrast, surviving patients exhibit a broader spread of nonconformity scores, with many clustered near 1.0, especially at the extreme ends of the note length spectrum. Notably, a dense cluster of long survival notes (far right) is associated with high nonconformity scores, suggesting that the model becomes overconfident in incorrect death predictions when presented with excessively lengthy documentation. Overall, this figure reinforces the insight that both insufficient and excessively detailed notes contribute to model uncertainty and misclassification, particularly for survival cases, which are aligned with data presented in Table 1.

Figure 3.

Scatter plot of nonconformity scores versus note length, stratified by in-hospital death outcome. Each point represents a patient from the calibration set. The y-axis shows the total number of tokens in the concatenated clinical note documented within the first 24 h of admission. The x-axis represents the nonconformity score calculated for the true class label using conformal prediction.

Table 1.

Note length and LLM-only prediction performance by death status.

	Class
	In-hospital death		In-hospital survived
Performance metrics
Recall	93%		13%
Precision	52%		63%
F1-score	67%		22%
Number of tokens per patient
Mean	1321		1707
SD	909		2022
	Correctly classified	Misclassified	Correctly classified	Misclassified
Mean	1376	461	922	1812
SD	906	362	1537	2057

On a balanced test set, the conformal prediction framework achieved an overall empirical coverage of 90.4%, slightly above the nominal 90% level, indicating conservative calibration. Disaggregated by class, coverage was 99.7% for “in-hospital death” (over-coverage) and 81.1% for “in-hospital survive” (under-coverage). The mean prediction set size was 1.67 labels overall, with “in-hospital survive” instances producing larger sets (mean = 1.80) than “in-hospital death” instances (mean = 1.54), reflecting greater uncertainty for survival cases.

These results reveal a pronounced class-dependent calibration disparity. The underlying classifier assigns consistently higher predictive probabilities—and thus higher P value—to the “in-hospital death” class, causing most positive instances to exceed the global P value threshold and achieve near-perfect coverage (99.7%). Conversely, the model expresses lower confidence in the “in-hospital survive” class, resulting in more frequent exclusion from prediction sets, undercoverage (81.1%), and larger average prediction set sizes.

This imbalance is further evident in the distribution of prediction set sizes. Among all conformal predictions, 32.9% produced singleton sets (representing confident predictions where the model assigned only 1 class), while 67.1% were hedged cases containing multiple possible outcomes, indicating calibrated uncertainty in borderline scenarios.⁵⁸ The class-specific breakdown reinforces this pattern: “in-hospital survive” cases yielded only 20.1% singleton sets (79.9% multiple outcome), whereas “in-hospital death” cases produced 45.8% singleton sets (54.2% multiple outcome), with no empty sets observed in either class. This distribution reveals a clear class-specific stratification of uncertainty, with survival predictions exhibiting systematically higher uncertainty than mortality predictions.

Importantly, this class-conditional calibration imbalance represents a known limitation of the standard split-conformal prediction framework, which enforces marginal coverage over the entire test set but does not guarantee class-specific calibration. As a result, when the learned P value threshold is even slightly too permissive, the method tends to favor overcoverage overall, while still allowing class-level imbalances to persist.

These findings suggest that class-conditional conformal prediction methods may be necessary in contexts where balanced coverage across classes is critical—particularly when the clinically relevant outcome is a minority class—though such approaches reduce effective calibration set size and warrant careful consideration of this tradeoff when class imbalance is moderate or when the outcome currently favored by the model aligns with clinical priorities.

Figure 4 displays the relationship between credibility (conformal P value of the most likely class) and confidence (1 minus the P value of the second most likely class) in GPT predictions on the test set. The strong positive relationship indicates that high statistical support for the top prediction typically coincides with clear separation from competing alternatives. Most predictions are correctly covered (orange points along the upward trend). However, a small cluster of blue points at high credibility reveals a form of overconfidence: the model occasionally assigns high statistical support to incorrect predictions. In an ideally calibrated model, we expect errors to cluster around low credibility (low-confidence) regions, but this pattern indicates the model sometimes exhibits overconfidence on mistakes.

Figure 4.

Credibility vs Confidence in GPT Predictions. Each point is a model prediction; dots are color-coded by conformal coverage: orange = covered (correct), blue = miscoverage (incorrect). Credibility reflects statistical support for the top label, while confidence reflects separation from competing labels. Points follow a strong upward trend: higher credibility generally coincides with higher confidence, and most predictions are correctly covered (orange). Notably, a small cluster of blue points at high credibility indicates overconfident errors: cases where the model assigns high support to an ultimately incorrect label.

Discussion

The GPT performed substantially better in predicting death cases than survival. It correctly identified around 94% of the patients who died in the hospital, indicating a strong sensitivity or recall for the positive class (“in-hospital death”). In contrast, it correctly predicted survival in only around 22% of the cases. Prioritizing sensitivity for in-hospital death can be clinically advantageous in high-stakes settings where the consequence of missed deterioration is severe. A model that preferentially captures in-hospital death cases can serve as an early warning layer to trigger rapid evaluation, closer monitoring, and timely escalation of care. In practice, this bias can help surface a manageable subset of patients for proactive interventions, support dynamic resource allocation, and focus team attention on those at greatest imminent risk. Moreover, this imbalance suggests that the model tends to overpredict death, possibly due to characteristics in the notes that bias the model toward more critical outcomes, or due to limitations in distinguishing subtle signals of recovery from high-risk notes.

The relationship observed between note length and nonconformity scores aligns closely with the patterns reported in Table 1. Patients who died during hospitalization had slightly shorter notes on average compared to those who survived. However, when stratified by prediction outcome, a clearer pattern emerges: for death cases, longer notes were associated with correct classification, whereas for survival cases, longer, heterogeneous documentation more often coincided with misclassification. This is supported by the lower nonconformity scores for most death cases in Figure 2, indicating high model confidence when sufficient clinical detail is present.

The behavior observed in Figure 4 signals that conformal prediction may not effectively compensate for overconfident but incorrect model outputs. While most predictions show a consistent relationship between high confidence and accurate outcomes, this figure reveals a recurring failure mode—overconfident but incorrect predictions—even in the presence of calibration. Relying solely on GPT’s internal probability estimates, even when augmented with conformal prediction, could therefore mislead clinicians if model failures are masked by overconfidence. These findings underscore the need for finer-grained calibration approaches, potentially incorporating class-conditional or adaptive conformal techniques, to more effectively mitigate overconfident errors and improve the reliability of AI-assisted decision-making.

This study fills several important gaps in the existing literature on clinical risk prediction. Prior research has largely focused on structured EHR variables, such as laboratory values, vitals, or demographic data, which, while valuable, capture only a fraction of the contextual and temporal nuance present in narrative documentation. By leveraging GPT-4o to interpret early documented ICU notes, this work investigates the applicability of LLMs for early prediction in critically ill patients, and finds that, without task-specific fine-tuning, their discriminative performance is limited. Furthermore, by integrating conformal prediction, we move beyond the traditional performance-centric focus of prior models to explicitly quantify and calibrate patient-level uncertainty, an area rarely addressed in prior LLM-based clinical prediction studies.

Conclusion and future work

This work demonstrates the use of CP to assess the reliability of prediction capability of GPT based solely on clinical notes documented within the first 24 h of ICU patient admission and integrating conformal prediction to quantify and calibrate the uncertainty of model outputs. While the performance of the underlying model is not outstanding, we show that CP can be used to find model deficiencies at greater granularity regardless of the overall model performance. Through the integration of conformal prediction, we quantified uncertainty in model outputs and achieved empirical coverage exceeding the nominal 90% level. However, deeper analysis of credibility, confidence, and nonconformity scores revealed critical limitations. Specifically, the model exhibited overconfidence in certain incorrect predictions, leading to notable undercoverage in survival cases and highlighting a class-conditional calibration imbalance. These findings underscore the importance of uncertainty-aware modeling in clinical decision support. Future work should explore adaptive or class-conditional conformal prediction frameworks, fine-tune language models using domain-specific data, and evaluate hybrid models that integrate structured EHR features with unstructured text to achieve more balanced and reliable predictions. Additionally, our reliance on the MIMIC-III dataset, though comprehensive, restricts the generalizability of our findings. The dataset’s inherent biases and its derivation from a specific clinical environment may not accurately reflect the varied patient populations found across different geographic or healthcare settings. Despite these limitations, the study presents a significant step forward in our understanding of the capabilities and limitations of advanced language models in the critical domain of healthcare.

Author contributions

Fatemeh Shah-Mohammadi: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Validation, Visualization, Writing–original draft, Writing–review & editing. Alexander Millar: Conceptualization, Data curation, Formal analysis, Methodology, Validation, Visualization, Writing–original draft, Writing–review & editing. Julio Facelli: Conceptualization, Funding acquisition, Methodology, Supervision, Writing–review & editing. Ram Gouripeddi: Conceptualization, Methodology, Supervision, Writing–review &editing.

Supplementary material

Supplementary material is available at JAMIA Open online.

Conflicts of interest

None declared.

Funding

The research reported in this publication was supported in part by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UM1TR004409. This work was supported by the Department of Defense (DoD) SMART Scholarship-for-Service Program to AM. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Department of Defense or the National Institutes of Health.

Data availability

The reader can reproduce our experiments by gaining access to the MIMIC-III dataset and using our public https://doi.org/10.5281/zenodo.15793500.

Ethical approval

No protected health information was collected, and the analytical dataset was fully de-identified.