Abstract
Objectives
Transformer-based models for clinical prediction using longitudinal electronic health record (EHR) data are often developed with large feature sets and long patient histories under the assumption that more data improves performance. However, high-dimensional inputs and long sequences increase computational burden, potentially limiting scalability. We evaluated how feature dimensionality and sequence length affect predictive performance, calibration, risk stratification, and computational efficiency in EHR prediction.
Materials and Methods
Using longitudinal EHR data from adults with mild asthma in an integrated healthcare system, we evaluated input representation design for predicting acute asthma exacerbation. Feature dimensionality was reduced using Integrated Gradients attribution scores, univariate performance-based selection, and clinically guided selection strategies. Sequence length was varied using percentile-based truncation of patient histories. Performance was assessed using discrimination, calibration, high-risk classification, threshold-based event capture, and computational efficiency across regions.
Results
Models using fewer features achieved discrimination comparable to the 80-feature reference model, with area under the receiver operating characteristic curve values ranging from 0.843 to 0.864 vs 0.870 for the full model. Moderate sequence-length truncation reduced training time by more than 70% with minimal loss in discrimination. Although reduced-dimensional models showed attenuation of predicted risk at the upper tail, they identified similar high-risk populations and captured comparable proportions of asthma exacerbation events at clinically relevant thresholds.
Conclusion
Transformer-based prediction models maintained strong performance across reduced feature sets. While dimensionality reduction modestly affected calibration at the highest risk levels, moderate sequence-length reduction substantially reduced computational burden with limited change in overall discrimination. These findings highlight tradeoffs between input complexity, predictive performance, and computational efficiency.
Keywords: deep learning, dimensionality reduction, feature selection, longitudinal modeling, risk stratification, computational efficiency
Introduction
Deep learning models are increasingly used for clinical risk prediction using longitudinal electronic health record (EHR) data.1–3 Transformer-based architectures, in particular, enable modeling of temporally ordered clinical events and have demonstrated strong performance across a range of prediction tasks by capturing complex dependencies within patient histories. As a result, there has been growing interest in applying transformer-based models to large-scale EHR data to support risk stratification and population-level clinical decision-making.4–7
In practice, these models are often developed under the implicit assumption that more data leads to better performance, including the use of large feature sets and long longitudinal histories. However, high-dimensional input representations introduce substantial practical challenges. Large numbers of features increase data engineering complexity and require stable availability of heterogeneous data elements across sites, time periods, and clinical workflows, which may limit model portability and robustness after deployment.8,9 Similarly, long patient histories increase computational cost for both training and inference, particularly for transformer-based models where complexity scales with sequence length.10,11 These factors can hinder scalability, delay model updates, and create barriers to routine use in real-world health systems.
Despite these challenges, there is limited empirical guidance on how input design choices, specifically feature dimensionality and sequence length, affect clinically relevant model behavior. Prior work in traditional machine learning has emphasized dimensionality reduction to improve generalizability and interpretability, whereas deep learning approaches often rely on representation learning without explicit feature selection.12,13 In the context of EHR-based modeling, studies examining dimensionality reduction, sequence compression, or truncation strategies have primarily emphasized optimization-oriented metrics such as training loss and discrimination, whereas comparatively less attention has been devoted to calibration, clinical utility, and threshold-based identification of high-risk patients.14–16 As a result, it remains unclear how much input complexity is necessary to achieve robust and clinically useful performance.
In this study, we address this gap by systematically evaluating feature dimensionality and sequence length as key design parameters in transformer-based clinical risk prediction using longitudinal EHR data. Using acute asthma exacerbation (AAE) in adults with mild asthma as a representative use case, we compare multiple feature selection strategies and sequence-length configurations under controlled conditions while holding model architecture constant. We assess the impact of these design choices on discrimination, calibration, risk stratification, threshold-based event capture, and computational efficiency. By providing a comprehensive evaluation of input representation tradeoffs, this work aims to inform the development of scalable and deployable deep learning models for clinical prediction in real-world healthcare settings.
Methods
Study overview
We conducted a retrospective methodological evaluation to examine how input dimensionality influences predictive performance and computational efficiency in deep learning–based clinical risk prediction models. Acute asthma exacerbation in patients with mild asthma was used as a representative use case within a previously established EHR cohort of adults with mild asthma.17,18
The study design held the study population, outcome definition, temporal encoding of EHR data, and transformer architecture constant while systematically varying feature dimensionality, feature-selection strategy, and maximum sequence length (Figure 1). Experiments were conducted under controlled conditions to isolate the effects of these input representation design choices on discrimination, calibration, risk stratification, threshold-based event capture, and computational efficiency.
Figure 1.
Study design and experiment framework. Study design overview illustrating the evaluation of input dimensionality in transformer-based clinical risk prediction using longitudinal EHR data. The study population and data framework, longitudinal sequence representation, model architecture, and training procedures were held constant across all experiments. Two experimental axes were evaluated. In feature dimensionality experiments, feature selection strategy (IG, univariate AUC-based, and clinically guided) and feature set size were varied under a fixed maximum sequence length, with model performance assessed using discrimination, calibration, risk stratification, threshold-based event capture, and computational efficiency. In sequence-length experiments, the maximum input length was varied using percentile-based truncation under a fixed 20-feature IG–selected model, with evaluation focused on discrimination and computational efficiency.
Data source, study population, and outcome
The analytic dataset was derived from structured EHR data from Kaiser Permanente Southern California, including diagnoses, medications, laboratory results, and healthcare encounters.
Cases were defined as adults aged 18–85 years who experienced an AAE during 2013–2019, with the AAE date serving as the index date. Eligible cases required at least one asthma diagnosis and ≥12 months of continuous enrollment before the index date and met criteria for mild asthma, defined by low-intensity controller therapy, limited short-acting beta-agonist use, and absence of markers of moderate-to-severe disease, including asthma-related hospitalization and selected comorbid conditions in the past 12 months. Patients could contribute multiple eligible case events during the study period.
Controls were selected from eligible patients without AAE during the study period and assigned case-derived index dates to align predictor assessment windows. Each control patient contributed only one observation to the analytic dataset.
Outcome definition
Acute asthma exacerbation was defined using structured EHR criteria including asthma-related hospitalization, emergency department or observation encounters, or systemic corticosteroid treatment associated with asthma-related diagnoses.
Baseline transformer architecture and longitudinal input encoding
Predictor variables were ascertained from the 24 months preceding the index date.
Structured EHR data were transformed into temporally ordered longitudinal sequences representing clinical events occurring during the observation window (Figure S1). Each clinical concept (e.g., diagnosis, medication, laboratory result, or healthcare encounter) was represented as an individual token, with multiple tokens permitted on the same calendar day. Demographic characteristics were encoded as static patient-level variables. Clinical events were arranged chronologically to form the longitudinal input sequence for transformer-based modeling.
The transformer-based architecture and training framework were held constant across all experiments. Model inputs consisted of embedded longitudinal token sequences with positional encoding to preserve temporal order.
Training procedures and hyperparameters were held constant across experiments to isolate the effects of feature dimensionality and sequence length.
Dimensionality reduction strategies and feature subsets
To evaluate the effect of dimensionality reduction, we applied three complementary feature-selection strategies to the 80-feature baseline input space (Table 1). For all experiments, reduced-dimensional models were trained using the same transformer architecture and training settings as the full-feature reference model, with only the input feature set modified. For the two data-driven strategies, feature rankings were derived once from the full dataset to define fixed feature subsets that were applied consistently across all cross-validation experiments, thereby enabling direct comparison of identical input representations across model configurations.
Table 1.
Selection strategies evaluated for dimensionality reduction in transformer-based risk prediction.a
| Method | Type | Feature input | Selection basis | Feature subsets tested | Description |
|---|---|---|---|---|---|
| Clinically guided | Expert-derived | 80 | Clinical relevance and guidelines | 20 | Benchmark for interpretability and feasibility |
| IG | Model-based (deep learning) | 80 | Mean absolute attribution score | 10, 20, 40 | Captures non-linear feature importance from the trained model |
| AUC-based selection | Univariate model-based | 80 | Individual feature discrimination (AUC) | 10, 20, 40 | Ranks features by their standalone AUC value |
Feature counts shown under “Feature Input” reflect the full baseline structured feature space available prior to selection. Reduced feature subsets were derived exclusively from the training data and then fixed and applied identically for all subsequent model training and evaluation. Clinically guided features were selected a priori based on clinical expertise, guideline-informed reasoning, and prior asthma risk prediction studies, and did not rely on data-driven ranking. All feature selection experiments were conducted under a fixed maximum sequence length unless otherwise specified.
Integrated Gradients (IG)–based feature selection quantified each feature’s contribution to model output using attribution scores derived from the trained reference transformer model. Integrated Gradients is a gradient-based attribution method that estimates feature importance by integrating gradients from a baseline input to the observed input and has been widely used to interpret deep neural network predictions.19,20 Attribution values were computed across the training set and summarized using the mean absolute IG score for each feature.19 Features were then ranked by importance, and reduced feature subsets were defined using the top 10, top 20, and top 40 ranked features. Features included in each IG-derived subset are listed in Table S1.
Univariate area under the curve (AUC)-based feature selection ranked features according to their individual discriminatory ability for predicting AAE. Each feature was evaluated independently using its area under the receiver operating characteristic curve (AUROC) within the training data. Features were subsequently ranked by AUROC, and reduced feature subsets were defined using the top 10, top 20, and top 40 ranked features. Features included in each AUC-derived subset are listed in Table S1.
Clinically guided feature selection used a fixed subset of 20 features curated a priori based on clinical expertise and guideline-informed reasoning (Table S1).13 This subset was intended to reflect clinically interpretable variables commonly available in real-world healthcare settings. Feature selection was informed by prior evidence on risk factors for AAE, including findings from our previously published work.18
Sequence-length and runtime design
To evaluate the impact of maximum sequence length on predictive performance and computational efficiency, we systematically varied the maximum sequence length (max_len) applied to longitudinal input sequences while holding all other model components constant.
A maximum sequence length of 512 tokens was selected as the reference configuration based on the empirical distribution of sequence lengths (Figure 2). This threshold retained most longitudinal histories while limiting excessive computational burden associated with very long sequences. Sequences exceeding this length were truncated to retain the most recent tokens, whereas shorter sequences were padded.
Figure 2.
Effect of maximum sequence length on training time and discrimination. Mean training time (left y-axis) and AUROC (right y-axis) are shown as a function of maximum sequence length, evaluated using the IG-selected 20-feature model. Shaded regions indicate variability across cross-validation folds. Reducing maximum sequence length was associated with substantial decreases in training time, while AUROC remained largely stable across moderate sequence-length settings, with more pronounced performance degradation observed only at the most restrictive settings.
Alternative maximum sequence length settings were defined using percentile-based cutoffs derived from the empirical sequence-length distribution, including the 98th, 95th, 90th, 85th, 80th, 75th, 70th, 65th, 60th, 55th, and 50th percentiles. For each percentile threshold, max_len was defined such that the corresponding proportion of patient sequences could be retained without truncation, while longer sequences were truncated to preserve the most recent longitudinal information.
Because sequence lengths demonstrated a pronounced right-skewed distribution, percentile-based truncation enabled systematic evaluation of tradeoffs between retained longitudinal information and computational efficiency. Evaluation was limited to the 50th percentile to avoid excessive truncation of patient histories.
All sequence-length experiments were conducted using the IG-selected 20-feature model. Feature-selection strategy, feature count, transformer architecture, and training procedures were held constant to isolate the independent effect of sequence length. Batch size was fixed at 32 across experiments.
Model training and evaluation
Cross-site validation design
Model development and evaluation were conducted using six-fold cross-validation defined by geographically distinct Medical Service Areas (MSAs). In each iteration, data from five MSAs were used for training and the remaining MSA was held out for evaluation. Performance metrics were averaged across folds. This design approximates an internal–external validation framework by evaluating model performance across geographic and operational variation.21
Discrimination performance
Model discrimination was assessed using AUROC and the area under the precision–recall curve (AUPRC),14 calculated on held-out evaluation data within each cross-validation fold.
Calibration and risk stratification across feature set
Calibration and risk stratification analyses evaluated how feature dimensionality influenced predicted risk distributions, patient-level risk assignment, and high-risk identification. Shared risk bins derived from the full-feature reference model were applied unchanged across all reduced-feature models. The mean predicted risk and observed event rates were calculated within each bin to assess calibration across the risk distribution (Figure 3A).
Figure 3.
Distribution of observed sequence lengths and percentile-based maximum sequence length settings. Histogram showing the empirical distribution of observed sequence lengths across the study cohort, where sequence length is defined as the number of structured EHR input elements included in a longitudinal observation. Vertical dashed lines indicate percentile-based maximum sequence length settings evaluated in subsequent analyses, ranging from the 98th percentile (p98) to the 50th percentile (p50). The pronounced right-skew of the distribution indicates that a small fraction of observations have substantially longer sequences than the majority, motivating evaluation of alternative maximum sequence length configurations to improve computational efficiency while preserving predictive performance. The vertical line at 512 indicates the selected maximum sequence length used in model training, chosen to balance sequence coverage and computational efficiency.
To evaluate patient-level risk assignment, predicted risks from reduced-feature models were compared against predictions from the full-feature reference model among patients with predicted risk ≥0.8 under the reference model (Figure 3B).
For operational risk stratification analyses, a fixed predicted-risk threshold derived from the reference model was used to classify approximately 20% of patients as high risk (Figure 3C). Event capture was assessed as the proportion of observed exacerbations occurring among patients classified as high risk (Figure 3D).
Computational efficiency
Computational efficiency was quantified using total training time, inference time per patient record, and throughput, defined as the number of patient records processed per second. Runtime comparisons were conducted under standardized hardware and software conditions.
Sensitivity analysis
Sensitivity analyses were conducted using batch sizes of 32, 64, and 128 while holding all other model parameters constant. Discrimination and runtime metrics were evaluated across cross-validation folds.
Sensitivity of risk stratification to selection thresholds was additionally evaluated using cumulative event capture curves. Patients were ranked by predicted risk, and the cumulative proportion of observed events captured was calculated as progressively larger fractions of the population were selected (Figure S2). This analysis complements fixed-threshold evaluation across broader population strata.
Software
All analyses were performed in Python. Deep learning models were implemented using PyTorch and the Hugging Face Transformers library. Data preprocessing and evaluation were conducted using NumPy, pandas, and scikit-learn.
Ethics approval
This study was approved by the Kaiser Permanente Southern California Institutional Review Board with a waiver of informed consent for retrospective analysis.
Results
Patient characteristics
A total of 395 941 observations were included in the analytic dataset, comprising 86 424 cases with AAE and 309 517 controls without. These observations correspond to eligible index events, and individual patients could contribute multiple events. The mean (±SD) age was 44.7 (±17.9) years, and 63.9% were female.
Baseline characteristics stratified by case status are summarized in Table 2. Compared with controls, cases were older (49.2 vs 43.4 years) and more likely to be female (72.4% vs 61.6%). Indicators of asthma-related disease activity and treatment intensity were consistently more prevalent among cases, including higher use of short-acting β-agonists (87.7% vs 72.2%), asthma controller medications (66.3% vs 38.9%), combination ICS/LABA therapy (22.6% vs 9.6%), and systemic corticosteroids (72.7% vs 38.5%).
Table 2.
Patient characteristics.
| Characteristics | Overall (N = 395 941) | Cases (N = 86 424) | Controls (N = 309 517) |
|---|---|---|---|
| Demographics | |||
| Age, mean (SD) | 44.7 (17.9) | 49.2 (16.7) | 43.4 (18.0) |
| Female, n (%) | 253 042 (63.9) | 62 535 (72.4) | 190 507 (61.6) |
| Asthma-related characteristics | |||
| Short-acting β-agonist use, n (%) | 299 337 (75.6) | 75 770 (87.7) | 223 567 (72.2) |
| Asthma controller use, n (%) | 177 811 (44.9) | 57 309 (66.3) | 120 502 (38.9) |
| ICS/LABA use, n (%) | 49 274 (12.4) | 19 505 (22.6) | 29 769 (9.6) |
| Systemic corticosteroid use, n (%) | 181 832 (45.9) | 62 795 (72.7) | 119 037 (38.5) |
| Comorbidities/phenotype | |||
| Allergic rhinitis, n (%) | 88 674 (22.4) | 24 133 (27.9) | 64 541 (20.9) |
| Chronic sinusitis, n (%) | 69 055 (17.4) | 22 905 (26.5) | 46 150 (14.9) |
| Healthcare utilization | |||
| Asthma ED or UC visit, n (%) | 93 856 (23.7) | 31 355 (36.3) | 62 501 (20.2) |
| Laboratory | |||
| Eosinophil counts (elevated), n (%) | 134 474 (33.9) | 32 825 (38.0) | 101 649 (32.8) |
Cases also had a higher prevalence of comorbid atopic conditions, including allergic rhinitis (27.9% vs 20.9%) and chronic sinusitis (26.5% vs 14.9%), as well as greater prior healthcare utilization, reflected by a higher proportion with asthma-related emergency department or urgent care visits (36.3% vs 20.2%). Elevated eosinophil counts were more common among cases than controls (38.0% vs 32.8%), consistent with greater inflammatory disease burden.
Selected feature sets across dimensionality-reduction strategies
Feature-selection strategies yielded overlapping but non-identical subsets of variables, with attribution- and performance-based methods selecting features based on model-derived importance metrics, while the clinically guided subset emphasized interpretability and implementation feasibility (Table S1). These selected feature sets formed the basis for all reduced-dimensional model evaluations.
Discrimination performance under fixed-length sequence design
Under the fixed-length sequence design, reduced-feature models demonstrated discrimination performance comparable to the full-feature reference model despite substantial reductions in input dimensionality (Table 3). Performance was evaluated on held-out MSAs in each fold and averaged across all six folds. The reference model achieved a mean AUROC (±SD) of 0.870 (±0.007) and an AUPRC of 0.718 (±0.034), with only modest differences across reduced-feature configurations.
Table 3.
Discrimination performance and computational efficiency under fixed-length sequence design.a
| Method | Number of features | AUROC c | AUPRC c | Total training time (h) b | Inference time (ms/record) |
|---|---|---|---|---|---|
| All features (reference) | 80 | 0.870 ± 0.007 | 0.718 ± 0.034 | 4.97 ± 0.26 | 292.73 |
| IG | 10 | 0.843 ± 0.006 | 0.664 ± 0.035 | 4.97 ± 0.25 | 292.65 |
| IG | 20 | 0.855 ± 0.005 | 0.687 ± 0.031 | 4.96 ± 0.23 | 291.99 |
| IG | 40 | 0.864 ± 0.006 | 0.698 ± 0.028 | 5.02 ± 0.23 | 295.39 |
| AUC | 10 | 0.829 ± 0.004 | 0.632 ± 0.034 | 4.96 ± 0.27 | 270.40 |
| AUC | 20 | 0.839 ± 0.005 | 0.670 ± 0.020 | 4.96 ± 0.26 | 291.99 |
| AUC | 40 | 0.846 ± 0.006 | 0.668 ± 0.033 | 5.02 ± 0.31 | 296.20 |
| Clinical | 20 | 0.836 ± 0.005 | 0.646 ± 0.034 | 4.90 ± 0.29 | 289.16 |
All results were obtained under a fixed-length sequence design with a maximum sequence length of 512, defined as the maximum number of structured EHR input elements included in a patient’s longitudinal history after chronological ordering. Batch size was fixed at 32 to ensure fair comparison across models. Values are reported as mean (SD) across six cross-validation folds. The reference model uses the complete 80-feature input space without feature selection.
Total training time reflects end-to-end model training time on a single GPU-equivalent environment. Inference time represents the average latency required to generate a single prediction during inference.
AUROC, area under the receiver operating characteristic curve; AUPRC, area under the precision–recall curve.
Among IG-selected models, the AUROC values ranged from 0.843 to 0.864 across 10-, 20-, and 40-feature configurations, with corresponding AUPRC values between 0.664 and 0.698 (Table 3). Discrimination differences remained modest despite substantial reductions in feature count.
Models using univariate AUROC-based selection showed similar patterns, although discrimination was consistently slightly lower than that of IG-based models at comparable feature counts.
The clinically guided 20-feature model achieved an AUROC of 0.836 (±0.005) and an AUPRC of 0.646 (±0.034), retaining most of the discriminatory performance of higher-dimensional models despite its more parsimonious feature set.
The total training time and inference latency were similar across models under the fixed-length sequence design, indicating that reductions in feature dimensionality did not materially affect computational performance in this setting (Table 3).
Impact of maximum sequence-length truncation on runtime and discrimination
Sequence-length truncation was associated with substantial reductions in computational cost while preserving most discriminatory performance across a wide range of truncation thresholds (Table 4). Using the IG-selected 20-feature model as a representative configuration, setting the maximum sequence length at the 98th percentile of the empirical sequence-length distribution reduced total training time by approximately 38% relative to the fixed-length reference configuration, with minimal change in AUROC.
Table 4.
Impact of maximum sequence length on discrimination and computational efficiency.a
| Method | Number of features | Maximum sequence length (max_len) | AUROC b | Total training time (h) c | Throughput (records/s) | Δ Training time (%) | Δ AUROC |
|---|---|---|---|---|---|---|---|
| All features (reference) | 80 | 512 | 0.870 ± 0.007 | 4.97 ± 0.26 | 3.68 ± 1.16 | NA | NA |
| IG | 20 | 512 | 0.855 ± 0.005 | 4.96 ± 0.23 | 3.75 ± 1.16 | −0.1 | –0.015 |
| IG | 20 | p98 | 0.854 ± 0.005 | 3.08 ± 0.17 | 6.05 ± 1.93 | −38.0 | – |
| IG | 20 | p95 | 0.854 ± 0.005 | 2.23 ± 0.12 | 8.36 ± 2.65 | −55.2 | – |
| IG | 20 | p90 | 0.852 ± 0.005 | 1.70 ± 0.09 | 10.98 ± 3.45 | −65.9 | – |
| IG | 20 | p85 | 0.850 ± 0.005 | 1.47 ± 0.08 | 12.63 ± 3.96 | −70.3 | – |
| IG | 20 | p80 | 0.848 ± 0.005 | 1.32 ± 0.07 | 14.08 ± 4.45 | −73.4 | – |
| IG | 20 | p75 | 0.845 ± 0.005 | 1.13 ± 0.06 | 16.43 ± 5.15 | −77.2 | – |
| IG | 20 | p70 | 0.842 ± 0.005 | 1.01 ± 0.06 | 18.41 ± 5.84 | −79.6 | – |
| IG | 20 | p65 | 0.839 ± 0.005 | 0.94 ± 0.06 | 19.78 ± 6.37 | −81.0 | – |
| IG | 20 | p60 | 0.835 ± 0.005 | 0.84 ± 0.04 | 22.30 ± 6.99 | −83.2 | – |
| IG | 20 | p55 | 0.830 ± 0.005 | 0.77 ± 0.04 | 24.07 ± 7.53 | −84.4 | – |
| IG | 20 | p50 | 0.826 ± 0.005 | 0.73 ± 0.04 | 25.49 ± 8.03 | −85.3 | – |
Results are shown for the IG–selected model using 20 features; feature-selection method and feature count were held constant while maximum sequence length (max_len) was varied. All models were trained using a fixed batch size of 32. Values are reported as mean (SD) across six cross-validation folds. Maximum sequence length specifies the allowed number of structured EHR input elements per patient after chronological ordering. A value of 512 indicates the reference configuration, while values of pXX denote settings derived from the XXth percentile of the empirical sequence-length distribution.
AUROC, area under the receiver operating characteristic curve.
Total training time reflects end-to-end model training time (in hours) on a single GPU-equivalent environment. Throughput represents the average number of patient records processed per second during training. Δ Training time (%) and Δ AUROC indicate differences relative to the reference configuration (max_len = 512).
Progressively smaller maximum sequence lengths yielded larger gains in computational efficiency. The maximum sequence lengths corresponding to the 80th and 75th percentiles reduced training time by more than 70% compared with the reference configuration, while AUROC declines remained modest across cross-validation folds. In contrast, maximum sequence lengths below the 65th percentile were associated with more pronounced degradation in discrimination, suggesting diminishing returns when the allowed sequence length becomes overly restrictive (Table 4).
These relationships are illustrated in Figure 2, where the training time decreased monotonically as the maximum sequence length was reduced, whereas the AUROC remained largely stable across a wide range of truncation thresholds, with only modest declines observed at the most restrictive settings.
Figure 3 shows the empirical distribution of sequence lengths, which was highly right-skewed, with most sequences relatively short and a small proportion substantially longer. The median sequence length was approximately 67, with the 80th, 90th, 95th, and 98th percentiles corresponding to sequence lengths of approximately 133, 186, 243, and 328, respectively.
Calibration and risk re-stratification across feature sets
Calibration and risk stratification analyses evaluated the effects of feature dimensionality on predicted risk distributions, patient-level risk assignment, and high-risk identification (Figure 4).
Figure 4.
Effect of feature dimensionality on calibration, risk stratification, and event capture. Multi-panel figure illustrating how dimensionality reduction influences calibration, observation-level risk assignment, high-risk identification, and event capture across models. (A) Calibration across the full risk distribution (upper left). Calibration curves comparing predicted and observed risk for the full-feature reference model and reduced-feature models using shared risk bins. Risk bin boundaries were defined as quintiles (20% intervals) of predicted risk from the full-feature reference model and then applied unchanged to all reduced-feature models. In higher-risk bins, reduced-feature models exhibited lower predicted and observed risks than the reference model, consistent with attenuation of high-risk stratification with increasing dimensionality reduction. (B) Observation-level comparison of predicted risk relative to the full-feature reference model (upper right). Scatter plot comparing predicted risks from reduced-feature models against predictions from the full-feature reference model. Each point represents an individual observation, and the diagonal line indicates equality between models. Systematic deviation below the identity line at higher reference-model risk values indicates a narrower range of predicted risks assigned by reduced-feature models at the upper end of the risk spectrum. (C) Number of observations exceeding a reference-defined high-risk threshold (lower left). A fixed predicted-risk threshold was derived from the full-feature reference model to classify 20% of observations as high risk. Applying this same threshold to reduced-feature models, the number and percentage of observations classified as high risk are shown for each model configuration. The number and percentage of observations classified as high risk were similar across models, with only modest differences between the full-feature reference model and reduced-feature configurations. (D) Event capture at a fixed high-risk threshold (lower right). Number and percentage of observed asthma exacerbation events captured among observations classified as high risk using the same fixed threshold applied in Panel C. Event capture was similar across models, with only modest differences between the full-feature reference model and reduced-feature configurations. Sensitivity analyses examining broader risk-selection thresholds are shown in Figure S2.
Using shared risk bins defined by quintiles of predicted risk from the full-feature reference model, calibration plots demonstrated systematic differences between the reference and reduced-feature models in higher-risk strata (Figure 4A). Across reduced-feature configurations, the mean predicted risk and observed event rates were lower in the highest-risk quintiles, indicating attenuation of risk estimates with increasing dimensionality reduction. Differences between models were minimal in lower-risk groups.
Patient-level comparisons further illustrated differences in risk assignment (Figure 4B). This analysis was restricted to patients with predicted risk ≥ 0.8 in the reference model and is shown for three representative 20-feature configurations (IG-20, AUC-20, and Clinical-20), based on a random sample of 1000 patients per model. Across all reduced-feature models, predicted risks were systematically lower than those of the reference model, as indicated by deviation below the identity line. The degree of attenuation varied across feature-selection strategies, with IG-20 showing closer alignment and AUC-20 demonstrating greater downward deviation.
To assess the operational implications of these differences, a fixed high-risk threshold defined by the reference model (top 20% of patients) was applied across all models (Figure 4C). The proportion of patients classified as high risk remained close to 20% across configurations. For example, IG-10 and IG-20 identified slightly fewer high-risk patients (19.2% and 19.4%, respectively), whereas IG-40 identified a proportion similar to the reference model.
Using the same threshold, the proportion of observed asthma exacerbation events captured among high-risk patients was also similar across models (Figure 4D). IG-40 and IG-20 captured proportions of events most comparable to the reference model, with only modest differences observed across reduced-feature configurations.
Sensitivity analysis
In sensitivity analyses, discrimination performance was highly consistent across batch-size configurations (Table S2A). The mean AUROC values increased modestly with larger batch sizes, from 0.8479 at batch size 32 to 0.8495 at batch size 128, with differences ≤0.002 across folds.
Computational efficiency improved with increasing batch size. The average total training time decreased from 1.32 hours at batch size 32 to 1.16 hours at batch size 128, corresponding to an approximately 12% reduction in runtime (Table S2B).
In a separate sensitivity analysis, the cumulative event capture was evaluated across a range of risk-selection thresholds (Figure S2). Differences in event concentration across models were minimal at the operational threshold used in the main analysis (Figure 4D) and became more apparent only when broader fractions of patients (approximately 25%–35%) were considered. Even under this worst-case ranking-based scenario, differences in cumulative event capture across models remained modest.
Discussion
In this study, we provide a systematic evaluation of input representation design in transformer-based clinical risk prediction using longitudinal EHR data. Across multiple feature-selection strategies and sequence-length configurations, we demonstrate that substantially more parsimonious input representations can preserve most discriminatory performance while markedly improving computational efficiency. Importantly, our findings show that reductions in feature dimensionality and sequence length alter predicted risk distributions, particularly at the upper tail, without materially affecting clinically relevant risk stratification or event capture at operational decision thresholds. These results highlight a critical distinction between global performance metrics and deployment-relevant behavior. By quantifying tradeoffs between input complexity, computational burden, and clinically meaningful performance, this study provides practical guidance for scalable and deployable transformer-based models in real-world healthcare settings.
Longitudinal EHR-based prediction models are frequently constructed using large numbers of heterogeneous features and extended temporal histories in an effort to maximize predictive signal.2,4,7 However, this approach introduces important practical challenges that are often underemphasized in model development. High-dimensional feature spaces increase data engineering complexity and may reduce portability across sites and workflows, while long input sequences substantially increase computational cost for transformer-based architectures.8,9,22 Our findings suggest that much of this input complexity may be unnecessary for achieving strong and clinically meaningful predictive performance. Reduced feature sets and shorter sequences preserved discrimination and threshold-based risk stratification while substantially improving computational efficiency, supporting more parsimonious input design as a viable strategy for real-world implementation.
Prior studies examining dimensionality reduction in deep learning models for longitudinal EHR data have largely emphasized optimization-oriented metrics such as training loss, accuracy, and discrimination.3,10,14,15 In contrast, our study evaluated calibration, patient-level risk assignment, threshold-based risk stratification, and event capture under controlled comparisons of both feature dimensionality and sequence length. This integrated framework provides practical insight into how input representation choices influence both predictive performance and deployment-relevant behavior in transformer-based EHR models.
Figure 4 highlights an important distinction between distribution-level calibration and threshold-based clinical performance.14,16 Reduced-feature models exhibited attenuation of predicted risk and modest calibration differences at the upper end of the risk spectrum, particularly within the highest-risk group (Figure 4A and B). Despite these differences, performance at clinically relevant decision thresholds remained largely unchanged. When a fixed threshold identifying the top 20% of patients as high risk was applied, reduced-feature models identified a similar proportion of high-risk patients and captured a comparable fraction of observed asthma exacerbation events relative to the full-feature reference model (Figure 4C and D). Across feature-selection strategies, IG-based models (IG-40 and IG-20) showed performance most closely aligned with the reference model, whereas more aggressive reduction strategies demonstrated slightly greater attenuation. These findings suggest that dimensionality reduction introduces modest attenuation in calibration at the extreme upper tail of predicted risk, but with limited practical impact on threshold-based patient identification and event capture.
Although this study was not designed to evaluate clinical interventions, the observed differences in risk stratification across model configurations have important implications for how prediction models are operationalized in clinical settings. In practice, such models are typically deployed using fixed decision thresholds to identify patients for enhanced monitoring or preventive action.9,16
Reduced-feature models, particularly when combined with moderate maximum sequence length designs, may offer a favorable balance between computational efficiency, data availability, and clinical performance in settings with constrained resources or frequent retraining requirements. Conversely, higher-dimensional models with longer effective input histories may better differentiate patients at the extreme upper end of the risk distribution, albeit with greater computational and operational complexity. These considerations are particularly relevant for health systems seeking to operationalize deep learning models under real-world constraints of data availability, computational resources, and model maintenance.8,9
Longitudinal EHR data are characterized by highly skewed distributions of observed sequence length, with a small proportion of patients contributing disproportionately long histories.2 By empirically characterizing this distribution and evaluating alternative maximum sequence length settings, we demonstrate that reducing the allowed input length yields substantial gains in computational efficiency. Moderate maximum sequence length settings reduced training time by more than half while preserving discrimination, whereas more restrictive maximum sequence lengths resulted in diminishing returns and larger performance degradation. Importantly, sequence-length selection was guided by empirical data characteristics rather than arbitrary architectural constraints, enhancing the practical relevance of the modeling framework.
The observed effects of dimensionality reduction and sequence-length design are likely dependent on the prediction task and the underlying data. Reduced-feature models may have less capacity to distinguish patients at the extremes of risk when variables encoding healthcare utilization intensity or complex nonlinear relationships are excluded. Similarly, the impact of sequence-length truncation will depend on the temporal structure of the prediction task and the distribution of sequence lengths of the underlying data. Accordingly, model design choices should be evaluated within the context of the specific clinical application.
Sequence-length truncation may have important implications for preserving longitudinal clinical information. Because sequence truncation removed the earlier portion of longer longitudinal EHR sequences, historical clinical information may have been excluded for patients with more extensive clinical histories. Potentially affected information could include historical exacerbations, prior medication exposures, treatment changes, or patterns of healthcare utilization that may provide important context regarding long-term disease severity. Consequently, patients with longer or more complex longitudinal histories may be more susceptible to information loss than patients with shorter histories. Although overall model discrimination remained stable across moderate sequence-length reductions, these aggregate performance metrics do not exclude the possibility that truncation could have a greater impact within specific patient subgroups. Accordingly, future implementation studies should evaluate model performance beyond population-level discrimination metrics, as aggregate performance measures may mask differences among patients with longer or more complex longitudinal histories. Subgroup validation stratified by historical sequence length or healthcare utilization may help determine whether sequence truncation disproportionately affects these patients before clinical deployment.
This study has several limitations that warrant consideration. First, we evaluated a limited set of predefined feature-selection strategies, including IG, univariate AUROC ranking, and a clinically guided feature subset. Other approaches, such as adaptive or dynamically learned feature selection methods, may further optimize the tradeoff between computational efficiency and predictive performance. Second, all analyses were conducted within a single integrated healthcare system. Although this setting enables consistent longitudinal data capture and outcome ascertainment, generalizability to other healthcare systems may be limited. Third, computational evaluations were performed under controlled experimental conditions with fixed model architectures and training procedures. Real-world deployment may introduce additional constraints related to data latency, missingness, evolving feature availability, and retraining frequency that were not explicitly modeled in this study. In addition, we did not characterize the specific types of longitudinal clinical information removed by truncation or evaluate whether truncation differentially affected patients with longer or more complex disease histories.
In summary, transformer-based clinical risk prediction models can achieve strong performance using substantially more parsimonious input representations than is commonly assumed. Although reductions in feature dimensionality and sequence length introduce modest changes in calibration and risk distribution, these effects have limited impact on clinically relevant risk stratification and event capture at operational decision thresholds. These findings suggest that input complexity can be strategically evaluated and reduced to improve computational efficiency while maintaining strong overall predictive performance.
Supplementary Material
Acknowledgments
The authors thank the patients of Kaiser Permanente for helping to improve care through the use of information collected through our electronic health record systems. The authors also thank Ms. Sole Cardoso for her assistance with manuscript formatting and submission.
Contributor Information
Wansu Chen, Department of Research and Evaluation, Kaiser Permanente Southern California, Pasadena, CA 91101, United States.
Botao Zhou, Department of Research and Evaluation, Kaiser Permanente Southern California, Pasadena, CA 91101, United States.
Robert S Zeiger, Department of Allergy, Kaiser Permanente Southern California, San Diego, CA 92111, United States; Department of Clinical Science, Kaiser Permanente Bernard J. Tyson School of Medicine, Pasadena, CA 91101, United States.
William Crawford, Department of Allergy, Kaiser Permanente Southern California, Harbor City, CA 90710, United States.
Michael Schatz, Department of Allergy, Kaiser Permanente Southern California, San Diego, CA 92111, United States.
Eric Puttock, Department of Research and Evaluation, Kaiser Permanente Southern California, Pasadena, CA 91101, United States.
Stanley Xu, Department of Research and Evaluation, Kaiser Permanente Southern California, Pasadena, CA 91101, United States.
Matthew T Slaughter, Kaiser Permanente Center for Health Research, Portland, OR 97227, United States.
Fagen Xie, Department of Research and Evaluation, Kaiser Permanente Southern California, Pasadena, CA 91101, United States.
Author contributions
Wansu Chen (Conceptualization, Funding acquisition, Writing—original draft, Writing—review & editing), Botao Zhou (Formal analysis), Robert S. Zeiger (Investigation), William Crawford (Investigation), Michael Schatz (Investigation), Eric J. Puttock (Investigation), Stanley Xu (Investigation), Matthew T. Slaughter (Investigation), and Fagen Xie (Data curation, Investigation)
Supplementary material
Supplementary material is available at JAMIA Open online.
Funding
National Heart, Lung, and Blood Institute, National Institutes of Health (R01 HL163049).
Conflicts of interest
R.S.Z. reports consulting fees from AstraZeneca; grant support from Sanofi; and royalties from UpToDate. Michael Schatz reports grant support from Sanofi and royalties from UpToDate. The rest of the authors declare that they have no relevant conflicts of interest.
Data availability
Anonymized data that support the findings of this study may be made available from the investigative team in the following conditions: (1) agreement to collaborate with the study team on all publications, (2) provision of external funding for administrative and investigator time necessary for this collaboration, (3) demonstration that the external investigative team is qualified and has documented evidence of training for human subjects protections, and (4) agreement to abide by the terms outlined in data use agreements between institutions.
References
- 1. Choi E, Bahadori MT, Schuetz A, et al. Doctor AI: predicting clinical events via recurrent neural networks. JMLR Workshop Conf Proc. 2016;56:301-318. [PMC free article] [PubMed] [Google Scholar]
- 2. Rajkomar A, Oren E, Chen K, et al. Scalable and accurate deep learning with electronic health records. NPJ Digit Med. 2018;1:18. 10.1038/s41746-018-0029-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Shickel B, Tighe PJ, Bihorac A, et al. Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE J Biomed Health Inform. 2018;22:1589-1604. 10.1109/JBHI.2017.2767063 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Li Y, Rao S, Solares JRA, et al. BEHRT: transformer for electronic health records. Sci Rep. 2020;10:7155. 10.1038/s41598-020-62922-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Yang Z, Mitra A, Liu W, et al. TransformEHR: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records. Nat Commun. 2023;14:7857. 10.1038/s41467-023-43715-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Xian S, Grabowska ME, Kullo IJ, et al. Transformer patient embedding using electronic health records enables patient stratification and progression analysis. NPJ Digit Med. 2025;8:521. 10.1038/s41746-025-01872-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Rasmy L, Xiang Y, Xie Z, et al. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ Digit Med. 2021;4:86. 10.1038/s41746-021-00455-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Kelly CJ, Karthikesalingam A, Suleyman M, et al. Key challenges for delivering clinical impact with artificial intelligence. BMC Med. 2019;17:195. 10.1186/s12916-019-1426-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Wiens J, Saria S, Sendak M, et al. Do no harm: a roadmap for responsible machine learning for health care. Nat Med. 2019;25:1337-1340. 10.1038/s41591-019-0548-6 [DOI] [PubMed] [Google Scholar]
- 10. Nerella S, Bandyopadhyay S, Zhang J, et al. Transformers and large language models in healthcare: a review. Artif Intell Med. 2024;154:102900. 10.1016/j.artmed.2024.102900 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. Adv Neural Inf Process Syst. 2017;30:5998-6008. [Google Scholar]
- 12. Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed. Springer; 2009. [Google Scholar]
- 13. Guyon I, Elisseeff A. An introduction to variable and feature selection. J Mach Learn Res. 2003;3:1157-1182. [Google Scholar]
- 14. Van Calster B, McLernon DJ, van Smeden M.; Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, et al. Calibration: the Achilles heel of predictive analytics. BMC Med. 2019;17:230. 10.1186/s12916-019-1466-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21:128-138. 10.1097/EDE.0b013e3181c30fb2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26:565-574. 10.1177/0272989X06295361 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Xie F, Puttock EJ, Slaughter MT, et al. Cohort vs case-control design for transformer-based prediction of asthma exacerbations in mild asthma. NPJ Digit Med. 2026;9:481. 10.1038/s41746-026-02624-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Chen W, Puttock EJ, Schatz M, et al. Risk factors for acute asthma exacerbations in adults with mild asthma. J Allergy Clin Immunol Pract. 2024;12:2705-2716 e6. 10.1016/j.jaip.2024.05.034 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Sundararajan M, Taly A, Yan Q. Axiomatic attribution for deep networks. Proc Mach Learn Res. 2017;70:3319-3328. [Google Scholar]
- 20. Ancona M, Ceolini E, Oztireli AC, Gross M. Towards better understanding of gradient-based attribution methods for deep neural networks. In: International Conference on Learning Representations (ICLR). 2018.
- 21. Steyerberg EW, Harrell FE. Jr., Prediction models need appropriate internal, internal-external, and external validation. J Clin Epidemiol. 2016;69:245-247. 10.1016/j.jclinepi.2015.04.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25:44-56. 10.1038/s41591-018-0300-7 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
Anonymized data that support the findings of this study may be made available from the investigative team in the following conditions: (1) agreement to collaborate with the study team on all publications, (2) provision of external funding for administrative and investigator time necessary for this collaboration, (3) demonstration that the external investigative team is qualified and has documented evidence of training for human subjects protections, and (4) agreement to abide by the terms outlined in data use agreements between institutions.