Dynamic Updating Strategies to Assess Hospital Performance of Surgical Aortic Valve Replacement

Abstract

Background:

Prediction models determining expected outcomes are infrequently updated (i.e, static), which may reduce accuracy and misclassify hospital performance over time. Dynamic models incorporate changes over time and may improve accuracy and fairness in hospital comparisons. This study evaluated whether dynamic updating, compared to a static model, altered hospital rankings and outlier detection among surgical aortic valve replacement (SAVR) patients.

Methods:

This retrospective cohort study assessed performance across 53 hospitals using claims data from the Pennsylvania Health Care Cost Containment Council. A multivariable logistic regression model using clinical and demographic variables was developed on 1999–2006 data to predict 30-day postoperative mortality, then applied to testing data from 2007–2018 to compare four strategies. (1) A static model with fixed parameters, (2) an annual correction factor (CF) based on The Society of Thoracic Surgeons methodology, (3) calibration regression (CR) for annual recalibration, and (4) dynamic logistic state-space modeling (DLSSM) to continuously update model coefficients. Performance was evaluated using observed-to-expected (O/E) ratios and Z-scores. Lower values indicate better-than-expected outcomes.

Results:

The training sample included 14,070 patients (mean age 66.6; 43.1% female); the testing sample included 29,127 patients (mean age 67.4; 39.1% female). The static model had the widest Z-score variability (range −6.97 to 1.38), compared to CR (−3.04 to 2.85), CF (−2.87 to 3.24), and DLSSM (−2.57 to 3.03). The static model labeled 15 hospitals as significantly better-than-expected; only 3 (20.0%) maintained this classification with CF and DLSSM, and 5 (33.3%) with CR. No hospitals were classified as significantly worse-than-expected under the static model, whereas CR identified 6, and both DLSSM and CF identified 7.

Conclusion:

Static models may misclassify hospital performance and rankings. Dynamic strategies influence outlier detection and change hospital rankings over time. Regular model updates may better reflect current performance, supporting fairer hospital comparisons.

Introduction

Observed-to-expected (O/E) mortality ratios are commonly used when evaluating hospital performance and provide insights into whether a hospital’s observed mortality meaningfully differs from its expected rates. Their reliability depends on the accuracy of the expected mortality estimates derived from clinical prediction models. If shifts in patient characteristics or advances in care are not accounted for, O/E ratios may fail to reflect true differences in hospital performance.

Prediction models are often infrequently updated (static), raising concerns about whether they adequately capture evolving patient populations and treatment practices as they continue to be used in various hospital performance reporting frameworks. Statewide cardiac surgery reports for coronary artery bypass graft surgery (CABG) can lag behind current hospital performance, with reported O/E ratios based on data that are often several years old due to lagged reporting and/or infrequently updated models.¹ In New York, models are refitted every few years using multi-year datasets (e.g., 2016–2018, 2017–2019), but risk estimates remain fixed within each reporting period.^2,3 Consequently, shifts in patient characteristics, treatment practices, or surgical outcomes during a reporting cycle are not incorporated into the model until the next refitting cycle.

Other states, like California, take a different approach by reporting hospital performance ratings for CABG annually since 2005 and recently publishing public ratings for transcatheter aortic valve replacements (TAVR). For both procedures, risk-adjusted hospital mortality rates recalibrate the O/E ratio based on the yearly procedure-specific statewide mortality rate.^4–6 However, the underlying model parameters remain constant during each reporting period, meaning shifts in patient risk factors are not addressed until a new model is developed. In either approach, the question remains whether the modeling strategies used to estimate O/E ratios adequately account for evolving patient demographics and clinical practices or if alternative methods, such as dynamic updating, provide a more adaptive framework for hospital performance comparisons.

Patient risk profiles for surgical aortic valve replacement (SAVR) have changed substantially, particularly with the introduction of TAVR to treat aortic stenosis. This shift has contributed to improved overall outcomes among patients undergoing SAVR, leading to concerns that existing risk models, including the Society of Thoracic Surgeons Predicted Risk of Mortality (STS-PROM) and other commonly used scoring systems, overestimate mortality, particularly in low-risk patients.^7,8 This may lead to biased O/E ratios. Some studies report two- to three-fold overprediction, highlighting the need for more timely and accurate models to support benchmarking and quality improvement efforts.⁸

As such, concerns have emerged about the sustained predictive accuracy between the infrequent (static) model updates and their impacts on patients and providers.^9–12 Inaccurate risk assessments can have lasting impacts by altering a hospital’s reputation, referrals, and financial stability.¹³ As demonstrated throughout the history of publicly reported scorecards for CABG in various states across the United States, it can be difficult for hospitals/providers to recover from misleading O/E ratios and perceived poor quality of care.^13–16

While substantial progress has been made in refining risk adjustment methodologies for improved O/E ratios and fairer assessments of hospital performance, the potential of dynamic prediction models (DPMs) to enhance hospital performance assessments remains underexplored. DPMs may be a tool to help facilitate more equitable and accurate comparisons when ranking hospitals as they adjust the parameters of existing models, allowing the model to evolve with the changing patient profiles and outcome risk over time, without having to wait to develop an entirely new model. Previous research has emphasized the importance of risk adjustment for improved O/E ratios¹⁷ or model updating^18,19, with little attention to the connection between the two concepts. Prior research has demonstrated improved predictive performance of DPMs compared to static models, but it remains unknown how DPMs impact hospital performance and quality of care evaluations.^12,20,21

This study compares hospital performance rankings and outlier detection between a static model and dynamic updating for patients who underwent SAVR. We hypothesize that the DPMs, by enhancing prediction accuracy, will yield different expected mortality rates compared to the static model, consequently altering O/E ratios and impacting hospital rankings. Additionally, we expect the DPMs to improve outlier detection compared to the static model. Rather than aiming to recommend specific updating methods for clinical adoption, this work evaluates the theoretical and empiric benefits of dynamic updating and its potential effects on hospital quality assessments, providing insights that could inform future research.

Methods

Data

The data used in this retrospective cohort study were obtained from the Pennsylvania Health Care Cost Containment Council (PHC4) under a data use agreement restricting public sharing. These data cannot be made available to other researchers unless explicitly approved by the researchers and PHC4. However, aggregated results and analytic methods are available upon reasonable request.

PHC4 collects claims data from all non-federal hospitals across the state.²² After excluding non-residents to ensure complete 30-day postoperative mortality data, the initial dataset included 104,468 aortic valve replacement (AVR) procedures between January 1, 1999, and December 31, 2018. These procedures were conducted at hospitals with varying procedural volumes and patient risk profiles, reflecting a diverse patient population across institutions.

Patients with concomitant surgeries such as CABG (N=43,416, 41.6%), prior AVRs (N=1,819, 1.74%), or TAVR (N=13,294, 12.72%) were excluded. Cases with missing admission status (N=60, 0.14%) or mortality data (N=29, 0.06%) were also excluded. After applying these criteria, 44,546 patients ≥30 years old undergoing their first isolated SAVR were included. Procedures performed between January 1, 1999, and December 31, 2006, were assigned to the training set, and those between January 1, 2007, and December 31, 2018, were assigned to the testing set.

The training period intentionally includes years when SAVR was the primary treatment option for aortic stenosis, while the testing period reflects the evolving case mix as TAVR became available for high-risk (2011) and intermediate-risk (2016) patients.²³ This split ensured the representation of a broad range of patient characteristics in both periods and the changing risk of SAVR over time, while also providing a sufficient sample size (approximately 50 events per year) for model development and testing.

Study Population

Hospitals were the primary unit of analysis in the test data and were included if they performed SAVRs annually and had a combined surgical volume of at least 30 procedures over the 12-year testing period, aligning with PHC4 methodology.^24–27 This cumulative threshold was selected to help the validity and precision of our estimates, as hospitals with low procedural volumes tend to exhibit increased variability and more imprecise estimates. Very low-volume hospitals may observe greater variability in their O/E ratios as a small number of adverse outcomes can significantly impact the stability of the O/E ratio and skew the results, particularly if such hospitals treat even a small number of high-risk patients.^28,29

Statistical Analysis

Descriptive characteristics of the participants were calculated, including count (percentage) and mean (standard deviation) for categorical and continuous variables, respectively. To evaluate differences in the distribution of baseline characteristics between the training and test data, the standardized mean difference (SMD) was calculated for each variable. Values ≥0.1 were considered meaningful differences.³⁰

Model Development

Details of the baseline model have been previously reported.¹² Briefly, it was developed using least absolute shrinkage and selection operator (LASSO) logistic regression using the glmnet package in R.³¹ Model selection was performed using 10-fold cross-validation, where the penalty parameter was selected by minimizing deviance. It was developed with 14,070 SAVR procedures (557 deaths) from 1999-2006. The final model included eight predictors: age, acute myocardial infarction (primary diagnosis), aortic aneurysm/dissection, chronic kidney disease stage 5, chronic liver disease or cirrhosis, endocarditis, heart failure, and admission type (scheduled vs emergency procedure). Multicollinearity between predictors was assessed using the variance inflation factor (VIF), and all predictors had acceptable VIF values (<5).³² All variables were considered binary based on the presence or absence within the claims data, except for age, which was modeled linearly.

Modeling Strategies

Hospital performance was assessed using O/E ratios and Z-scores across four modeling strategies: (1) the static model, where the baseline model remained fixed throughout the study period, meaning the relationship between the predictors and outcome and the intercept (event rate) remains the same. (2) Applying an annual correction factor (CF), recalibrating the static intercept based on the Society of Thoracic Surgeons (STS) methodology, described below. (3) Calibration regression (CR) to annually rescale the predicted probabilities through a unified update of the intercept and slope of the model. (4) Dynamic logistic state space modeling (DLSSM) to continuously update model coefficients as needed, accounting for both time-varying and time-invariant parameters when updating the model. The latter two methods have been previously described in detail, but a brief overview is provided in the Appendix to highlight their key differences and how they were applied in this study.^12,33,34

Calibration factor

STS applies an annual correction factor (CF) to their predicted probabilities so that, on average, the observed to expected outcomes are equal within the Adult Cardiac Surgery Database (ACSD) for any given year. The CF is determined by dividing the sum of the expected deaths by the total number of deaths across all hospitals for that year. When the CF>1, it indicates the model overestimated the risk of mortality (observed < expected). A CF<1, suggests the model underestimated the predicted risk of mortality (observed > expected).³⁵

The CF is applied to the predicted probabilities generated by the static model so that the expected overall mortality aligns with the observed mortality, without altering the baseline (static) models’ coefficients. For example, in 2007, the CF was the sum of the predicted probabilities of the static model divided by the total number of observed deaths across all institutions that year. Subsequently, the 2007 recalibrated expected mortality per hospital was determined by dividing the original static model expected values by the 2007 CF. This recalibrated expected count was then used to obtain the STS-based recalibrated O/E in 2007.

This process repeats annually through 2018, using the same static baseline model to estimate expected values to generate the newly recalibrated expected values across the testing period. Finally, the sum of all the recalibrated expected mortality estimates per hospital across the years was used to determine the overall STS approach O/E ratio.

The rationale for selecting and comparing these updating strategies was to evaluate their effectiveness against conventional methods, such as the static model and the STS correction factor. A comparative analysis of updating strategies, including non-updating, recalibrating the intercept, recalibrating the intercept and slope, the closed testing procedure, and model revision, was conducted to predict one-year post-lung transplant survival using the Lung Allocation Score. Results from that study found that CR required minimal data, led to more consistent improvements, and exhibited less variability over time compared to the other strategies.²⁰

Expanding upon that work more specifically, our previous research demonstrated that CR and DLSSM improved prediction accuracy over time, as reflected by better calibration and discrimination than static models while balancing computational efficiency and complexity in this patient population.¹² The methods in our study allow for more responsive risk adjustment without requiring full model refitting or estimating a large number of parameters, which can introduce overfitting concerns. Additionally, alternative updating methods proposed in the literature, such as model revision or Bayesian approaches, add logistical and computational complexity, which may make them challenging to implement in clinical settings.^18,19,36 These methods may introduce too much variability for consistent yearly comparisons of hospital performance as they typically require larger datasets for stable estimation. Given that the goal of this study is to evaluate how updating strategies influence hospital rankings and outlier detection in a manner that aligns with real-world performance evaluation practices, we prioritized approaches that balance adaptability with stability, ensuring that observed changes in rankings are more likely to reflect meaningful shifts in hospital quality rather than random variation.

Hospitals typically assess and rank their performance yearly or over extended periods of calendar time rather than after every new patient. As such, these approaches and continuous model updates after each new patient were not considered as they fall outside the scope of this study, which aims to provide stable hospital performance comparisons. Frequent updates could cause small changes in patient outcomes, such as a few unexpected events, to disproportionately impact rankings. This could result in large, unstable shifts that are less reflective of consistent overall hospital performance. To ensure more reliable and interpretable comparisons, we opted for annual updates through the abovementioned four methods to average out these short-term fluctuations and provide a more stable basis for evaluating hospital performance.

O/E ratios

The O/E ratio was calculated for each hospital and to help mitigate potential biases attributed to a low volume hospital for any given year, the overall O/E ratios across the entire testing period from 2007-2018 were compared.

The overall observed mortality per institution was the sum of all deaths observed within each hospital throughout the 12 years. The overall expected mortality for an institution was estimated by adding up the individual predicted probabilities for each patient (based on each modeling strategy). A more detailed explanation of the calculations to determine the expected values can be found in the online supplemental material.

Identifying outliers

The Z-score represents the number of standard deviations by which a hospital’s performance deviates from the null value and more adequately accounts for data variability compared to the O/E ratio alone. Each hospital’s overall Z-score was estimated as (O/E −1) divided by the standard error (see supplemental material).³⁷ Hospitals with Z-scores outside the 95% confidence interval boundaries of ±1.96 were considered statistical outliers, indicative of suboptimal and superior performance, respectively.

Rankings

Hospitals were ranked from “best” (denoting lower O/E ratios or Z-scores) to “worst” (indicating higher O/E ratios and Z-scores) within each of the four strategies. The magnitude and the frequency of the change in ranks across the strategies were quantified.

R version 4.1.2 (R Foundation for Statistical Computing) was used for all data analyses and graphical representations. The study was reviewed by the University of Florida’s Internal Review Board and deemed exempt as nonhuman subject research (entry ID 17591; February 2, 2023).

Results

Study Population

The initial test data included 30,476 patients from January 1, 2007 to December 31, 2018. To ensure that the same hospitals were consistently compared year after year in the test data, only hospitals that performed SAVRs annually throughout the testing period were considered. Out of the 67 hospitals initially considered, 14 (20.9%) were excluded for not meeting the criterion of performing SAVRs every year. This exclusion removed only 1,349 patients (4.43%) from the test set, resulting in a final test set of 29,127 patients with 765 deaths across 53 hospitals. No hospitals were excluded due to low surgical volume (<30).

The surgical volume across the 12 years in the test data varied widely across the hospitals from, 50 to 4,107 SAVRs performed, with a mean of 549.6 (SD=688.6). The mortality risk per hospital also ranged broadly from 0%-6.88%, with a mean of 2.72% (SD=1.35%). The median mortality risk was 2.61% with an interquartile range of 1.80% (Q1=1.79%, Q3=3.59%). The surgical volume and mortality risk per hospital included in our analysis are shown in Supplemental S.Figure.1.

Table 1 provides an overview of the study population, summarizing the cumulative distribution of both model predictors and additional characteristics across the training and testing periods. When examining the yearly distribution of predictors included in the model, the prevalence of aortic aneurysm/dissection and heart failure increased over the study period, while the number of emergency procedures decreased, and other predictors remained relatively stable (Supplemental S.Figure.2).

Table 1.

Characteristics of the study population stratified by training and testing sets.

	Participants N (%)
	Training	Testing	SMD*
	1999-2006 N=14,070	2007-2018 N=29,127
Mean Age Years (SD)	66.6 (13.3)	67.4 (12.3)	0.06
Acute Myocardial Infarction (Primary Diagnosis)	169 (1.2)	177 (0.6)	0.06
Admission Type (emergency)	4,826 (34.3)	7,406 (25.4)	0.20
Alzheimer/Dementia	24 (0.2)	159 (0.5)	0.06
Anemia	2,615 (18.6)	14,205 (48.8)	0.67
Aortic Aneurysm and/or Dissection	2,150 (15.3)	6,178 (21.2)	0.15
Aortic Root Surgery (Concomitant)	1,715 (12.2)	4,957 (17.0)	0.14
Asthma	612 (4.3)	1,662 (5.7)	0.06
Cardiomyopathy	864 (6.1)	2,931 (10.1)	0.14
Cachexia	12 (0.1)	46 (0.2)	0.02
Chronic Pericardial Disease	165 (1.2)	340 (1.2)	<0.01
Chronic Kidney Disease Stage 1-4	254 (1.8)	3,163 (10.9)	0.38
Chronic Kidney Disease Stage 5+	468 (3.3)	498 (1.7)	0.10
Chronic Liver Disease/Cirrhosis	169 (1.2)	648 (2.2)	0.08
Chronic Obstructive Pulmonary Disease	2,107 (15.0)	3,761 (12.9)	0.06
Other Chronic Lung Diseases	145 (1.0)	377 (1.3)	0.03
Coronary Artery Disease	4,017 (28.6)	9,753 (33.5)	0.11
Endocarditis	465 (3.3)	1,023 (3.5)	0.01
Excision of Other Lesion/Heart Tissue Same Day	325 (2.3)	1,681 (5.8)	0.18
Heart Failure	4,834 (34.4)	9,925 (34.1)	0.01
History of Chronic Steroid Use	13 (0.1)	192 (0.7)	0.09
Lupus	65 (0.5)	135 (0.5)	<0.01
Oxygen Dependence Therapy	26 (0.2)	496 (1.7)	0.16
Obstructive Sleep Apnea	51 (0.4)	2,766 (9.5)	0.43
Parkinson’s	46 (0.3)	132 (0.5)	0.02
Peripheral Arterial Disease	2,160 (15.4)	6,265 (21.5)	0.16
PTCA/Stent	375 (2.7)	1,301 (4.5)	0.10
Pulmonary Hypertension	405 (2.9)	799 (2.7)	0.01
Rheumatoid Arthritis	124 (0.9)	323 (1.1)	0.02
Sex (Female)	6,067 (43.1)	11,396 (39.1)	0.08

^*SMD-standardized mean difference. Values ≥ 0.1 were considered meaningful differences between the training and testing cohorts and are bolded. SAVR-surgical aortic valve replacement, PTCA-percutaneous transluminal coronary angioplasty.

Identification of Outliers

All DPMs demonstrated improved overall calibration over the static model, with calibration metrics indicating better agreement between predicted and observed mortality, supporting the use of these models for deriving O/E estimates in this analysis. This was reflected in lower overall Hosmer-Lemeshow statistics calculated for the entire testing period (2007-2018) with the static model = 151.18, CF = 4.01, CR = 8.39, and DLSSM = 9.21. Overall discrimination improved slightly across updating methods, with C-statistics for the static model = 0.687, CF = 0.692, CR = 0.688, and DLSSM = 0.698. The overall mean absolute error was also lower across all updating methods (static = 0.063, CF and CR = 0.052, DLSSM = 0.050), reflecting small but consistent improvements in predictive performance (supplemental Table S1).

The implication of a miscalibrated model is evident when it comes to identifying outliers. The static model indicated 15 hospitals as significantly better-than-expected (Z ≤−1.96). However, after updating, 10 hospitals (66.7%) with CR and 12 hospitals (80.0%) with DLSSM and CF were designated as expected (i.e., no longer significantly better-than-expected). Likewise, the static model did not identify any hospitals as significantly worse-than-expected (Z ≥1.96), but CR identified 6, and DLSSM and CF both identified 7 (Figure 1).

Figure 1: Comparison of Hospital Classification Under Static vs Updated Models.

Each point represents a hospital. The x-axis shows Z-scores under the static model; the y-axis shows scores under the corresponding updating approach. Z-scores reflect deviation between observed and expected mortality (O/E); negative values indicate better-than-expected performance. Red lines mark statistical significance thresholds at −1.96 and 1.96; hospitals outside this range are outliers. The dashed diagonal line marks where hospitals would fall if classified identically by both models. Colors and shapes highlight classification changes after model updating: green squares remained significantly better, blue circles lost significance, orange plus signs became significantly worse, and gray triangles were classified as expected in both models. Panel A compares the correction factor (CF) and static models; Panel B, calibration regression (CR) vs. static; Panel C, dynamic logistic state-space model (DLSSM) vs. static.

Hospital Rankings

When examining the variation in rankings based on the Z-score, nearly all hospitals experienced rank changes when compared to the static model across the three updating strategies. Out of the 53 hospitals, 47 (88.7%) shifted in rank with CF, 48 (90.6%) with CR, and 50 (94.3%) with DLSSM (Figure 2). Examining the magnitude of the shifts demonstrated that about half of the hospitals moved within +/−3 spots within each updating strategy (Figure 2). Though most facilities only varied slightly in their positioning, the updating strategies identified several hospitals with a more substantial repositioning in ranks. One hospital declined as many as 29 spots under CR and DLSSM (Figure 2, middle and bottom panel) and 26 spots with CF (Figure 2, top panel). Conversely, another hospital benefited from more marked improvements, ascending 14 spots with CR and DLSSM and 13 with CF. A more detailed visualization of individual hospital-level rank changes across updating strategies is provided in the supplementary material (Figure S.3).

Figure 2: Distribution of hospital-level changes in performance rankings based on Z-scores.

Rank differences were calculated as updated model minus static model rank. Negative values indicate a decline in ranking after model updating; positive values indicate improvement. The y-axis represents the number of hospitals per rank difference. Panels: Top- correction factor (CF); middle- calibration regression (CR); Bottom- dynamic logistic state-space model (DLSSM).

Discussion

This study evaluated how dynamically updating prediction models for 30-day postoperative mortality among patients undergoing SAVR impacts hospital rankings compared to a standard static model. The results from this analysis indicate that DPMs classified hospitals’ performance differently compared to the static model, with varying classifications of hospitals with significantly better or worse than expected performance over time. The static method identified more hospitals with significantly better-than-expected performance and fewer hospitals with significantly worse-than-expected performance than CR, CF, or DLSSM. Furthermore, the dynamic models contribute to shifts in the rankings of hospital performance, impacting which hospitals would be identified as top or bottom performers.

While the updating strategies were comparable regarding their predictive performance and impact on rankings and outlier identification, the advantage of CR or DLSSM over CF is their ability to update both the model intercept and slope. By annually updating the prediction model itself, (which CF does not), CR and DLSSM may enhance decision-making through improved predictive accuracy.^12,20 Previous research has documented a national decline in SAVR mortality over time and within this study population.^12,38 Given the documented decline in the accuracy of static SAVR models,^7,11,39,40 CF, which rescales the O/E ratios from a static model, may be limited in its ability to reflect the temporal changes. Adapting to temporal changes allows for more accurate prediction of patient outcomes, which is crucial for patient safety and quality of care evaluations.^13,41–43

Differences in the O/E ratios and shifts in rankings between the static and DPMs may be attributed to several factors, including hospital-level characteristics, such as surgeons and hospital staff; the severity of patient comorbidities over time; differences in referral patterns; and the rate of uptake of TAVR (which began in 2011). There may also be differences in postoperative care or other structures and processes within the healthcare framework that impact the observed mortality counts that are difficult to account for within standard prediction models.

Implications

An analysis of publicly available New York and California CABG reports found that O/E ratios and hospital rankings shifted during the three-year lag between data collection and publication, sometimes reversing classifications of high- and low-performing hospitals. Among hospitals initially classified as having less than half the expected mortality, only 29% retained that classification by the time the report was released, while 22% were reclassified as having more than twice the expected mortality. Similarly, of hospitals initially identified as having more than twice the expected mortality, only 27% remained in that category, whereas 23% were later classified as having less than half the expected mortality.¹ While that study examined the impact of delayed reporting, it underscored a broader challenge in hospital performance assessment- ensuring that risk-adjusted comparisons remain reflective of evolving patient risk and clinical practices. DPMs may offer an alternative approach by incorporating ongoing updates to model parameters rather than relying on periodic refitting, potentially improving hospital comparisons’ timeliness and stability.

Furthermore, inaccurate risk assessments and misleading O/E ratios can influence patients to seek or defer care from providers falsely misclassified with better or worse than expected performance, respectively. While several studies have demonstrated the mixed or limited influence of public reporting systems on cardiologists’ referral patterns, public perception still impacts where patients seek care.^14,44–46 Schwartz et al. (2005) reported among a sample of Medicare patients who underwent high-risk scheduled surgical interventions, including valve replacements, and who had seen comparative hospital data, most respondents indicated that provider reputation (both the surgeons and hospitals) was either extremely or very important.⁴⁷

Jawitz et al. (2022) recently compared hospital rankings based on consumer-directed websites with clinical outcomes measured by the STS ACSD based on the O/E ratios. The authors found that consumer-directed rankings were not strongly associated with STS ACSD composite operative mortality and major morbidity scores and that there was no correlation between hospital rankings and mortality rates.⁴⁸ Similarly, Ghandour et al. (2021) reported weak correlations between STS scores, the US News & World Report, and evaluations by Centers for Medicare and Medicaid (CMS) for 30-day postoperative mortality in patients undergoing CABG.⁴⁹

While patient awareness of public reporting systems was limited in their early phases, and few cardiologists discuss the scores with patients, awareness may increase, or the impact of reports on patient preferences may change with the increased demands for transparency in healthcare, growing media attention, and continued internet accessibility.^45,50 There may also be financial implications, as seen with CABG mortality, as CMS may adjust reimbursements based on outcomes such as survival, combined with the influence of scorecards and performance rankings, influencing the market share of an institution.^14,51,52 As discussions surrounding the optimal risk adjustment approaches for determining O/E ratios and which reporting systems patients and providers rely on, DPMs may be a potential tool to improve rankings and decision-making.

Our results highlight the potential limitations of static models in evolving patient populations by demonstrating that hospital rankings and outlier classifications can shift depending on the updating method used. As risk prediction models are often used to assess hospital performance, understanding how different updating methods affect rankings is important for ensuring fair and meaningful comparisons.

Furthermore, DPMs and their potential for improved prediction accuracy may extend into other cardiovascular surgeries and various other procedures and outcomes. For instance, with its diverse range of surgical outcomes, the American College of Surgeons National Surgical Quality Improvement Program could help validate DPMs and expand their applicability.⁵³ By facilitating fairer comparisons of hospital performance, these updated models could inform quality improvement efforts and enhance patient outcomes.⁵⁴

Strengths and Limitations

While the results of this analysis offer insights into the impact of DPMs on performance rankings, expected mortality for any hospital could be influenced by factors not accounted for in our analysis. By using medical claims data, we do not have other morbidity or mortality indicators or access to other SAVR mortality predictors that leading agencies such as STS or PHC4 have within their models (e.g., laboratory values, ejection fraction, renal function). Still, our baseline model included important predictors of postoperative mortality for SAVR patients with moderate discriminatory ability in the training data. There may also be concerns of misclassification or missing data. However, PHC4 conducts quarterly trend analyses and audits their data to identify and reconcile any inconsistencies or unexpected trends for each hospital. This process helps ensure minimal missingness and that coded diagnoses and procedures accurately reflect clinical documentation. ⁵⁵

Beyond individual patient-level factors, hospital-level differences in case mix and institutional practices could also contribute to variability in performance rankings. While hierarchical modeling was not used to account for hospital-specific effects on expected mortality estimates, this study aimed to evaluate how different updating strategies influence hospital rankings and outlier detection over time, rather than to model institutional variability directly. Statewide cardiac surgery reports typically update risk models every few years or apply annual recalibrations uniformly across hospitals, rather than adjusting models separately for each institution. This study follows a similar framework by evaluating hospital performance over a defined period rather than continuously recalibrating at the individual hospital level. In addition, the study analyses were done at the hospital level.

Importantly, although current data limitations constrain the granularity of our findings, this does not detract from the relevance or validity of our results. The dynamic updating methods used in this study can be readily extended and applied to datasets with more granular information beyond claims data, potentially offering more insights into the adaptability and implications of these methods. This study’s updating methods are practical, computationally efficient, and clinically relevant without requiring real-time patient data or extensive model revision. Thus, while the data used in this study may not capture all relevant predictors of postoperative mortality, the methodological framework established here provides a solid foundation for future research with more detailed datasets, allowing for even more accurate and insightful evaluations of hospital performance over time.

Furthermore, the aim was to assess the potential impact of model updating on performance rankings and DPMs’ adaptability to changing risk profiles and outcomes over time, not to integrate the model into practice. Comparing the O/E ratios of each hospital to itself and other institutions across the modeling strategies provides a better understanding of the agreement between the observed and expected mortality and how each hospital responds to the evolving risk profiles of patients undergoing SAVR. Although the true O/E ratio remains unknown, prior research has shown that DPMs improve predictive performance and, therefore, should more accurately represent the O/E values than the static model.¹²

This study also has several strengths. To our knowledge, this is the first study to empirically evaluate how DPM methods, compared to a static model, impact hospital performance rankings in a large, statewide sample of patients who underwent SAVR. The sample included nearly 80% of all hospitals (n=53) in the state, helping to evaluate the impact of model updating on hospitals that may treat patients with varying levels of risk profiles and caseloads. To minimize sampling bias, we only included hospitals with ≥30 procedures throughout our test period and also ranked hospitals through Z-scores, a more stable measure adjusting for sample size and random error. An advantage of evaluating the performance and ranking hospitals by Z-scores is that it considers the variability in expected counts by incorporating the standard error, which will vary based on each modeling strategy’s predictive accuracy and reduce the sensitivity to random errors from the varying sample sizes across the hospitals more so than the O/E ratio alone.

Conclusion

DPM methods have the potential to provide more timely and accurate predictions by incorporating new data and information as they become available. This flexibility in responding to advances in the field can help facilitate fairer comparisons of hospital performance and provide better identification of hospitals making incremental improvements or worsening of outcomes, both important aspects for improving patient outcomes.

Further research is needed to validate these findings in more robust datasets with important laboratory measures and other established prognostic SAVR variables that may be missing in our baseline model and other risk models across different patient populations. Still, the dynamic strategies assessed in this analysis represent important improvements between infrequent model updates. Furthermore, our CR and DLSSM methods have advantages over the STS method because they update the actual prediction models annually, while the STS method historically has updated less frequently.

Non-standard Abbreviations and Acronyms

ACSD

Adult Cardiac Surgery Database

AVR

Aortic Valve Replacement

CABG

Coronary Artery Bypass Graft

CF

Correction Factor

CMS

Centers for Medicare & Medicaid

CR

Calibration Regression

DLSSM

Dynamic Logistic State Space Model

DPM

Dynamic Prediction Model

O/E

Observed to Expected

PHC4

Pennsylvania Health Care Cost Containment Council

SAVR

Surgical Aortic Valve Replacement

SMD

Standardized Mean Difference

STS

Society of Thoracic Surgeons

TAVR

Transcatheter Aortic Valve Replacement