Concordance Indices for Risk Scores With Policy Evaluations

ABSTRACT

Objective

To demonstrate the differences between concordance index (C‐index) methodologies and clarify the appropriate usage for risk score evaluations in health services applications.

Study Setting and Design

We performed a methodological comparison of C‐index metrics and illustrated the consequences of these differences through a study of liver failure patients.

Data Sources and Analytic Sample

We analyzed secondary adult liver transplant registry data from the Organ Procurement and Transplantation Network (OPTN), including all waitlist registrations from 2002 to 2022.

Principal Findings

The recommended concordance metric based on Gerds' weighting was higher for the original model for end‐stage liver disease (MELD) than Harrell's C‐Index, Uno's C‐Index, and naïve binary outcome metrics (0.864 [95% confidence interval (CI): 0.840, 0.888] versus 0.854 [95% CI: 0.844, 0.864], 0.832 [95% CI: 0.819, 0.844], and 0.727 [95% CI: 0.715, 0.740]), and it did not increase after the latest MELD formula update (0.874 [95% CI: 0.859, 0.889] to 0.869 [95% CI: 0.853, 0.885]).

Conclusions

The concordance indices that are often used in health services applications have important deficiencies under policy‐related dependent censoring, and researchers must apply appropriate weighting schemes to avoid bias. The findings uncover new interpretations of past evaluation results that have shaped national liver transplant policies.

Summary.

• What is known about this topic

  • ○
    The concordance index (C‐index) is widely used to evaluate the performance of model‐based risk scores in predicting adverse events.
  • ○
    There are many different versions of the C‐index, each with specific methodological advantages and disadvantages.
  • ○
    In health services applications with dependent censoring, such as national policy development for liver transplantation, the C‐index has been used inconsistently and incorrectly, generating misleading conclusions.
• What this study adds

  • ○
    We uncover the limitations of popular C‐index approaches in the context of health services research and clarify the appropriate methodology under policy‐related dependent censoring.
  • ○
    Using national registry data on liver failure patients, we demonstrate how methodological differences in C‐index calculations have the potential to change conclusions about risk score performance.
  • ○
    We identify the appropriate evaluation metric for risk prediction within a specific timeframe (e.g., 90‐day survival) and illustrate how liver transplant policy development has been impacted by these considerations.

1. Introduction

Statistical models are often used in health services research to identify patients with greater risks of adverse events [1, 2, 3]. If a model performs well at accurately discriminating these risk levels, policymakers may use it to distribute healthcare resources to patients who need them most. For example, the model for end stage liver disease (MELD) is a numeric score that quantifies mortality risk among liver failure patients, and it is used by the United States Organ Procurement and Transplantation Network (OPTN) to allocate deceased donor livers with greater priority to patients with higher MELD scores [4].

In practice, it is essential that predictive risk scores are validated to confirm that they maintain adequate predictive ability. These validations are typically conducted using a statistic known as the concordance index (C‐index), which measures the probability that the score correctly discriminates risk among any two patients [5, 6]. The MELD, which has been closely scrutinized for its national impact, is routinely evaluated based on the C‐index, and changes in this metric over time and across MELD methodologies have prompted policymakers to implement updated versions of the MELD [7, 8, 9, 10].

One statistical issue that arises when risk scores are evaluated under an active policy implementation is dependent censoring. That is, if healthcare policies allow patients with higher risk scores to receive preventive interventions, then these patients may become less susceptible to the adverse event and leave the study before developing the outcome (i.e., become censored) [11, 12]. When policy effects make censoring rates dependent on the risk scores in this way, naïve C‐index evaluations can be severely biased [13]. Intuitively, this is because a valid risk score should be higher for patients that would experience the adverse event sooner than others without any intervention, but dependent censoring makes it such that these same patients often avoid adverse events, and we never observe the counterfactual scenario without policy effects. For example, patients with higher MELD values may avoid mortality by receiving liver transplants with greater priority under national policy, but higher MELD scores are still expected to indicate greater transplant‐free mortality risk.

There are many adaptations of the C‐index that are designed to accommodate various data structures such as dependent censoring, and selecting the appropriate methodology is essential for accurate risk score evaluations [13, 14, 15, 16, 17]. In the context of liver transplantation, several authors have recognized the issue of censoring, but very few have correctly applied the C‐index to adequately account for this issue [8, 10, 18]. In fact, some of the most influential studies that have resulted in new MELD policies are based on misleading C‐index estimates [7, 18]. In this paper, we clarify the differences between C‐index methods and demonstrate the appropriate application under policy‐related dependent censoring. Using the MELD score as a motivating example, we illustrate how the appropriate C‐index overcomes the issue of dependent censoring and changes the risk score evaluation.

2. Methods

2.1. Target Concordance Probability

We first introduce notation to describe the patient outcomes and risk score values. Let $T$ be the time that a patient would experience the adverse event if they could be followed until this event occurred and let $R$ be the risk score under evaluation. Furthermore, let $C$ be the amount of time that a patient can be followed in the study, where $C$ may be less than $T$. In our liver transplantation example, $T$ is the time of transplant‐free death on the waitlist, $R$ is the MELD score, and $C$ is the time at which a patient is removed from the waitlist (often due to the patient receiving a liver transplant).

The C‐index aims to measure a quantity known as the target concordance probability, which describes the rate at which a patient who experiences the adverse event earlier than another patient is correctly assigned a higher risk score. Written in terms of our notation, this probability is:

$$\mathrm{P}\left(R_i>R_j|T_i<T_j\right),$$ (1)

where the subscripts denote the two patients $i$ and $j$. In general, the C‐index estimates this quantity based on the proportion of comparable pairs (i.e., pairs in which the ordering of the adverse event times is known) from the sample that have correctly ordered or “concordant” risk scores. A C‐index value of 0.5 corresponds to a completely random classifier of risk, whereas a C‐index value of 1.0 corresponds to a model that perfectly discriminates risk. In the following sections, we introduce several specific definitions of this general C‐index formula, highlighting the implications of their differences.

2.2. Harrell's C‐Index

Harrell's C‐index was one of the first methods to evaluate risk score performance using time‐to‐event data [5, 6]. The comparable pairs under Harrell's definition include any two patients from the sample where it is known which of these patients first experienced the adverse event. When there is censoring, this implies that both patients must be event‐free and uncensored until the first adverse event in the pair occurs. Despite its popularity, Harrell's C‐index is biased in the presence of censoring, and it estimates the following probability:

$$\begin{array}{c}\mathrm{P}\left(R_i>R_j|T_i<T_j,T_i<C_i,T_i<C_j\right),\end{array}$$ (2)

which does not equal the target concordance probability and depends on the censoring variable C [15].

By requiring the comparable pairs to include patients who remain uncensored until the first of the two event times, Harrell's C‐index cannot form many comparisons with patients who are censored very early. In our liver transplantation example, these patients having early censoring times from receiving a transplant tend to be the ones of primary interest with higher transplant‐free mortality risk. Therefore, Harrell's C‐index is not informative as a measure of the risk score's performance in predicting transplant‐free mortality under dependent censoring.

2.3. Uno's C‐Index

To address the limitations of Harrell's C‐index, Uno et al. [15] proposed an alternative estimator based on inverse probability of censoring weighting (IPCW). In essence, the adverse event times from Harrell's definition that are less likely to be observed due to censoring are given more weight in the C‐index calculation, where the weights are calculated from the marginal distribution of the censoring times. This weighting makes the sample more representative of the population and results in a C‐index estimate that is closer to the target concordance probability. If the censoring is independent of the adverse event times, then Uno's C‐index estimates the quantity:

$$\begin{array}{c}\mathrm{P}\left(R_i>R_j|T_i<T_j,T_i<\tau \right),\end{array}$$ (3)

where $\tau$ is some upper bound to keep the weights stable.

Given that the quantity shown above does not depend on $C$, Uno's C‐index is often thought to remove the influence of censoring on the evaluation. However, as mentioned above, this result only holds if the censoring times are independent of the adverse event times (i.e., $C$ and $T$ are independent), which is almost certainly violated in policy applications where both variables depend on the risk score values. For example, patients with higher MELD scores tend to experience adverse events earlier, and they also become censored earlier by receiving liver transplants with increased priority. Therefore, Uno's C‐index and similarly weighted statistics, such as the one used by Kim et al. [8] in attempts to address the issue of dependent censoring in liver transplantation [8, 19], are still biased and unsuitable for this scenario.

2.4. Gerds' C‐Index

Gerds et al. [13] extended Uno's C‐index to settings where the censoring depends on the risk scores, which matches the censoring mechanism that arises in liver transplantation. Instead of weighting the event times based on the marginal censoring distribution, Gerds' C‐index models the censoring distribution as a function of the risk score values. Thus, higher MELD scores may correspond to different censoring distributions than lower MELD scores, which appropriately reflects the differential impact of the policy on patients with different scores. To the best of our knowledge, Gerds' C‐index has been applied very rarely in the liver transplantation setting, but we argue that it is the only appropriate C‐index estimator for this application.

2.5. Naïve Binary AUC

In some health services applications, patient risks within a specific timeframe are of primary interest. For example, the MELD score is frequently evaluated for its ability to detect 90‐day mortality risks [8, 10]. The target concordance probability at timepoint $t$ is then:

$$\begin{array}{c}\mathrm{P}\left(R_i>R_j|T_i<t,T_j>t\right).\end{array}$$ (4)

In these instances, it has been shown that the C‐index is not a proper evaluation metric and does not accurately estimate the target concordance probability above [20]. Instead, this problem is typically approached by defining a binary outcome variable to indicate whether the patients are still event‐free at the timepoint of interest (e.g., 90 days) and applying evaluation metrics for binary outcomes such as the area under the receiver operating characteristic curve (AUC). However, for patients who are censored before the timepoint of interest, it is unknown whether they would have been event‐free if they had remained in the study.

Godfrey et al. [10] attempted to resolve this issue for the MELD score by assuming that all censored patients would have been alive at 90 days, and they concluded based on this that the AUC had been declining in performance over time [10]. These findings were cited by the OPTN as motivation for updates to MELD score policies [7], despite conflicting studies that did not observe declines in the C‐index [18]. In fact, we will clarify why the decline observed by Godfrey et al. [10] is an artifact of the dependent censoring mechanism and not a true waning of the concordance probability. Furthermore, the authors who first questioned the findings in Godfrey et al. [10] suggested the use of Harrell's or Uno's C‐indices instead [18], which are also not suitable according to the arguments above. In the next section, we introduce the cumulative‐dynamic AUC metric that is most appropriate for assessing mortality risk prediction within a given timeframe [20].

2.6. IPCW Cumulative‐Dynamic AUC

Blanche et al. [20] showed that for assessing risk scores at fixed timepoints, an IPCW cumulative‐dynamic AUC is the most appropriate metric. This statistic defines the “cases” as patients known to have $T\le t$ and the “controls” as those with $T>t$ and $C>t$, and calculates the AUC from these groups. Patients who are censored before the timepoint of interest with $C\le t$ are excluded from the calculation. The impact of dependent censoring on the AUC is accounted for through the same IPCW method proposed by Gerds' et al. [13] for the C‐index.

2.7. Empirical Comparisons

We demonstrate the differences between evaluation metrics through our motivating example of liver transplantation. Using the Standard Transplant and Analysis Research dataset from the OPTN [21], we construct a retrospective cohort of all patients newly registered for the liver transplant waitlist between January 1, 2020 and December 31, 2022, and estimate the weight functions for Uno's and Gerds' C‐indices across timepoints and MELD scores (based on the original MELD formula). Then, we compare various evaluation metrics for both the original MELD score and its updated versions (i.e., MELD‐na and MELD 3.0) [8, 9]. Finally, we construct year‐specific cohorts to show the distributions of the original MELD scores among those who were dead within 90 days (cases), alive and uncensored through 90 days (controls), and censored within 90 days [10, 18]. This study received Institutional Review Board approval.

3. Results

The weight functions in Figure 1 highlight the differences between the C‐index definitions. Harrell's C‐index applies a uniform weight for all observed event times and MELD values. Uno's C‐index applies more weight to later event times that are less likely to be observed due to censoring, but this weight function does not change with the MELD score. Finally, Gerds' C‐index also applies more weight to later event times, but this weighting scheme is amplified for higher MELD scores, where the event times are even less likely to be observed due to MELD‐dependent censoring.

FIGURE 1.

Empirical weight functions used in Harrell's, Uno's, and Gerds' concordance indices for the model for end‐stage liver disease (MELD) score, based on Organ Procurement and Transplant Network data as of January 2023.

Across all MELD versions in Table 1, the estimated Gerds' C‐index is higher than both Harrell's and Uno's C‐indices, and the IPCW cumulative‐dynamic AUC is higher than the naïve binary AUC. We also observe that Harrell's and Uno's C‐indices increase with each update of the MELD score, but the recommended IPCW cumulative‐dynamic AUC is not any higher for MELD 3.0 compared to MELD‐na. While Harrell's C‐index, Uno's C‐index, and the naïve binary AUC were all cited as motivation for the OPTN to adopt newer MELD versions in national policy [7], our analysis would not have supported the conclusion that MELD 3.0 was necessary.

TABLE 1.

Estimated Evaluation Metrics (and 95% Bootstrap Confidence Intervals) and Numbers of Comparable Pairs for the Model for End‐Stage Liver Disease (MELD) Score and Its Updated Versions (MELD‐na and MELD 3.0), Based on Organ Procurement and Transplant Network Data as of January 2023.

	Evaluation metric			Comparable pairs
	MELD	MELD‐na	MELD 3.0
Harrell's concordance index	0.854 (0.844, 0.864)	0.858 (0.848, 0.868)	0.860 (0.850, 0.870)	48,622,352
Uno's concordance index	0.832 (0.819, 0.844)	0.839 (0.827, 0.851)	0.841 (0.829, 0.852)	48,622,352
Gerds' concordance index	0.870 (0.855, 0.885)	0.865 (0.853, 0.877)	0.866 (0.854, 0.878)	48,622,352
Naïve binary AUC (t = 90)	0.727 (0.715, 0.740)	0.729 (0.716, 0.742)	0.733 (0.720, 0.746)	46,039,982
IPCW cumulative‐dynamic AUC (t = 90)	0.864 (0.840, 0.888)	0.874 (0.859, 0.889)	0.869 (0.853, 0.885)	24,061,768

Abbreviations: AUC, area under the receiver operating characteristic curve; IPCW, inverse probability of censoring weighting.

The results in Figure 2 illuminate the impact of dependent censoring on the naïve binary AUC, which includes the patients who were censored before day 90 in the control group. In 2002 (the year that the MELD score was first used under national policy), the distribution of MELD scores among the censored group appeared to be distinct from the distributions in the cases and controls. Over time, the MELD score distribution among the censored group became closer to the distribution among the cases, potentially reflecting the policy's impact of allowing patients with higher MELD scores and mortality risks to receive liver transplants quickly. Thus, treating the censored patients as controls is misleading and gives the false impression that the MELD's performance in distinguishing cases and controls declines over time.

FIGURE 2.

Time trends in the distributions of the model for end‐stage liver disease (MELD) score by survival status, based on Organ Procurement and Transplant Network data as of January 2023. The United Network of Organ Sharing System (UNOS) policy is to form 35 adult patient classes based on MELD scores that take on integer values between 6 and 40.

While Harrell's C‐index has been shown to be stable over time [18], the reason is not because it avoids dichotomizing the risk set at 90 days into cases and controls as argued by Kwong et al. [18], but to a greater extent because it excludes certain comparisons with censored patients where the ordering of the underlying survival times cannot be determined. The IPCW cumulative‐dynamic AUC similarly excludes comparisons with censored patients whose survival status at 90 days is unknown, and it has additional advantages over Harrell's C‐index in this context because it accounts for the dependent censoring and focuses on the timepoint of interest (i.e., 90‐day mortality). Given that the decline in the naïve binary AUC was cited as motivation for revising the MELD score in national liver transplant policy [7], the use of more appropriate concordance metrics would have eliminated concerns of declining MELD performance and potentially influenced these policy decisions.

4. Discussion

In this paper, we have clarified the differences between various metrics for risk score evaluation, highlighting their implications for health policy applications with dependent censoring. Specifically, we draw attention to two key considerations for risk score evaluations in health services research. First, one must consider whether censoring rates depend on the risk score values and apply Gerds et al.'s [13] weighting scheme to obtain accurate risk score evaluations in these cases. Second, if primary interest is in survival prediction until a specific timepoint (e.g., 90‐day survival), then IPCW cumulative‐dynamic AUCs are more informative than the C‐index [20].

This study is limited by the fact that the patient MELD scores could only be calculated at the time of waitlist registration, whereas in clinical practice these scores may be updated at later timepoints. Future work may seek available data on longitudinal MELD measurements and incorporate this information into concordance statistic calculations.

We have illustrated methodological considerations for the C‐index and their consequences through our motivating example of liver transplantation. The MELD score and its corresponding transplant allocation policies have a national impact on patient outcomes, and evaluations of MELD score performance based on the C‐index have shaped policymakers' decisions. We have shown that these influential evaluations are often based on inaccurate methodology that is unsuitable for data with risk‐dependent censoring. With these comparisons, we hope that health services researchers can identify the appropriate evaluation metric in support of efficient healthcare delivery.

Conflicts of Interest

The author declares no conflicts of interest.

Hartman N., “Concordance Indices for Risk Scores With Policy Evaluations,” Health Services Research 60, no. 6 (2025): e14619, 10.1111/1475-6773.14619.

Data Availability Statement

The data that support the findings of this study are available from the Organ Procurement and Transplantation Network. Restrictions apply to the availability of these data, which were used under license for this study. Data are available from https://optn.transplant.hrsa.gov/data/view‐data‐reports/request‐data/ with the permission of the Organ Procurement and Transplantation Network.