Comparison of Seven Liver Allocation Models with Respect to Lives Saved Among Patients on the Liver Transplant Waiting List

Abstract

Background

Patients with End-Stage Liver Disease (ESLD) on the liver transplant waiting list are prioritized for transplant based on the Model for End-Stage Liver Disease (MELD) score. We developed and used an innovative approach to compare MELD to six proposed alternatives with respect to waiting list mortality.

Methods

Our analysis was based on United Network for Organ Sharing data of patients with ESLD on the waiting list between January 2006 and June 2009. We compared six allocation models to MELD. Two models were based on re-weighting the variables used by MELD: an “updated” MELD, and ReFit MELD. Four models also included serum sodium: MESO, MeldNa, UKELD, and ReFit MELDNa.

Results

We estimated that UKELD and the updated MELD would result in significantly fewer lives saved. There were no significant differences between the other models.

Conclusion

Our new approach can supplement standard methods to provide insight into the relative performance of liver allocation models in reducing waiting list mortality. Our analysis suggests that UKELD and the updated MELD score would not be optimal for reducing waiting list mortality in the United States.

INTRODUCTION

In 2002, in an effort to reduce liver transplant waiting list mortality, the Organ Procurement and Transplantation Network (OPTN) adopted the Model for End-Stage Liver Disease (MELD) score to prioritize adult patients with End-Stage Liver Disease (ESLD) on the liver transplant waiting list (1). The MELD score is a modified version of a score developed by Malinchoc et al. (2) based on the survival experience of 231 patients undergoing transjugular intrahepatic portosystemic shunt and was subsequently shown to be a good predictor of 3-month mortality in diverse subpopulations of patients with liver disease (3, 4). However, in developing the new allocation policy based on the MELD score, the United Network for Organ Sharing (UNOS)/OPTN committee recognized that the allocation policy should be a “fluid, constantly changing system whereby new data and experience are constantly reanalyzed and incorporated into the system where appropriate” (1).

Since then, in the spirit of this comment, a number of research teams have sought to develop other models that could better predict waiting list mortality. Huo et al. (5) proposed a model that consisted of the ratio of the MELD score and serum sodium (MESO). Using OPTN data regarding 6769 on the U.S. waiting lists in 2005, Kim et al. (6) developed a model based on MELD, serum sodium and their interaction (MELDNa). Sharma et al. (7) developed a an updated MELD using the same variables as in the MELD score, but reweighting them based on an analysis of data on 38,899 patients on the liver transplant waiting list from 2001 to 2006 (which we will refer to as Updated MELD). Barber et al. (8) used 1103 patients on the transplant waiting lists in the United Kingdom to derive an allocation model based on the MELD variables (serum creatinine, bilirubin, and INR), and serum sodium. This model is currently used in the U.K. to prioritize patients. Most recently Leise et al. (9) developed a two models based on 14,214 patients on the U.S. waiting lists, one with the original three MELD variables (reFit MELD) (9), and one including sodium (reFit MELDNA) (9).

Most evaluations of liver allocation risk models compare their performance on historical waiting list mortality data using a “c-statistic” (10). A c-statistic is calculated as the area under a Receiver Operator Curve (ROC) which is a plot of sensitivity versus (1-specificity) at each possible cut-point of the quantitative predictor (10). C-statistics take values from 0 to 1; values closer to 1 indicate better performance of the predictor (10). Also, given one person with the event (e.g., death) and one person without the event, the c-statistic can be shown to equal the probability that the person with the event has a higher value of the quantitative predictor (11). Although the c-statistic is widely used, its direct clinical implications in this context are difficult to decipher.

In this paper, we utilized an alternative method to use historical data to compare liver allocation models. It is based on the number of lives that would have been saved over a period of time had additional donor livers been available. Using this method we compared the seven models described above with respect to lives saved on the transplant list over a period of time.

METHODS

Our approach is based on the following question: What if 10% more livers were available and were allocated based on a specific risk score? How many lives would be saved? We can estimate that number directly from historical waiting list data by hypothetically allocating 10% more livers and seeing how many of those who received a hypothetical liver in reality died on the waiting list. We performed this data analytic experiment separately for 7 different allocation scores, and compared the scores with respect to number of lives saved. Specific details are given below.

Standard Transplant Analysis and Research (STAR) Datasets

Our approach was designed to apply to historical liver transplant waiting list files provided by UNOS. These files contain one record for every patient registered on the waiting list in the U.S. since 1987. Each record indicates whether the patient was ever removed from the waiting list, and if so, the reason why they were removed. Reasons included death, transplant, condition deteriorated so that transplant was not indicated, and condition improved.

In addition, we have obtained a supplemental file from UNOS with all the patients’ serial laboratory values while they were on the waiting list. This includes repeated measures of serum creatinine, serum sodium, bilirubin, and international normalized ratio (INR). For serum creatinine, the number of measures per person varied from 1 to 102, and the median number of measures per person was 4. The median interval between two measures from the same person was 33 days (range 1–1258, 99th percentile=377). Similar numbers of measures were available for the other laboratory measures. This file also indicates time periods during which the person was “temporarily inactive”, i.e., time periods when the patient was not a candidate for a liver transplant.

As we previously described (12), to implement our approach, we reformatted the waiting list dataset into a person-month dataset that consisted of one record for each month that each person was on the waiting list.

Study Population

Our analysis was based on UNOS data as of August 25, 2009. We included patients with ESLD whose priority for liver transplantation was based on the MELD score and who were listed between January 1, 2006 and June 30, 2009. We excluded those under 18 years of age, patients listed as status 1A and exceptional cases (including those with hepatocellular cancer) whose priority for transplant was not based strictly on MELD score. We also excluded waiting list time during which the values of creatinine, bilirubin, sodium, or INR were unknown. Excluding these periods with unknown values resulted in exclusion of 2% of the waiting list time.

Liver Allocation Models Evaluated

We compared seven different models including MELD (1, 2), MESO (5), MELDNa (6), Updated MELD (7), UKELD (8), ReFit MELD (9), and ReFit MELDNa (9) that were previously described in the literature. Their formulae are shown in Table 1. In addition we compared the performance of each model to what would have happened if length of time on the waiting list had been used as the only factor determining liver allocation.

Table 1.

Liver Allocation Models

Allocation Models	Formula1	Details
MELD (1, 2)	9.57 Loge Cr + 3.78 LogeBil + 11.2 LogeINR +6.43	Values of Cr greater than 4.0 are set to 4.0. Values of Cr, Bil or INR below 1.0 are set to 1.0. Those on dialysis are given a Cr of 4.0. The score is rounded to the nearest integer and capped at 40.
MESO Index (5)	MELD/Na (mEq/l) × 10	MELD is calculated as above, but it is not rounded or capped at 40.
MELDNa (6)	MELD − Na −[0.025(MELD)(140-Na)] + 140	MELD is calculated as in the original formula (rounded, capped at 40). Values of Na below 125 are set to 125 and values of Na above 140 are set to 140. After calculation, the score is rounded to the nearest integer.
Updated MELD (7)	1.266 Loge(Cr+1) + 0.939 Loge(Bil+1) + 1.658 Loge(INR+1)	Values of Cr, Bil and INR are not changed, irrespective of their values, or whether the patient is on dialysis.
UKELD (8)	1.485 Loge (Cr) + 3.13 Loge(Bil) + 5.395 Loge(INR) −81.565 Loge(Na) + 435	Values of Cr (μmol/l) greater than 400 are set to 400. Values of Cr, Bil or INR below 1.0 are set to 1.0. Na is truncated to be between 112 and 150. The score is rounded to the nearest integer. For those on dialysis, Cr is given the value found when the patients were not on dialysis2.
ReFit MELD (9)	8.485 Loge(Cr) + 4.082 Loge(Bil) + 10.671 Loge(INR) + 7.432	Values of Cr greater than 3.0 are set to 3.0. Values of Cr below 0.8 are set to 0.8. Values of Bil or INR below 1.0 are set to 1.0. Values of INR above 3.0 are set to 3.0. Those on dialysis are given a Cr of 3.0. The score is rounded to the nearest integer.
ReFit MELD NA (9)	6.792 Loge(Cr) + 4.258 Loge(Bil) + 8.29 Loge(INR) + 0.652(140-Na) − 0.194(140-Na)Loge(BiliCC)	Values of Cr, Bil, and INR are defined as for ReFit MELD. Values of sodium below 125 are set to 125 and values of sodium above 140 are set to 140. BiliCC is the same as Bil, however, values over 20 are set to 20. The score is rounded to the nearest integer.

¹In all the formulae except UKELD, Cr refers to serum creatinine (mg/dl), Bil to total bilirubin (mg/dl), INR to prothrombin time-international normalized ratio, and Na to serum sodium (mmol/l). For UKELD, Cr refers to serum creatinine (μmol/L), Bil to total bilirubin (μmol/L),.

²Since we did not know the patient’s non-dialysis creatinine value, those on dialysis were given the maximum value of Cr (400 μmol/L).

Estimating the Number of Lives Saved

An ideal way to compare risk models would be to estimate the number of patients who would die on the waiting list if allocation was based on each model and then compare the results. However, with historical waiting list data, where allocation was already performed (and based on the MELD score), it is difficult to make that comparison because, for patients transplanted based on the MELD score, we do not know what would have happened to them if they had not been given a transplant. To make that comparison, it would be necessary to develop a predictive model to obtain the best guess regarding what would have happened to these patients. Developing such a model requires making numerous modeling assumptions.

However, it is possible to use the historical waiting list data to estimate the number of lives that would have been saved if additional livers had been available without modeling the outcomes for transplanted patients. Our approach was as follows: Assume 25 additional livers were available in the first month of the study period. Hypothetically “allocate” these livers to those active on the waiting list with the highest values of the risk score that month (not including those who actually received a transplant that month). Look ahead in time, to see if any of those who received a hypothetical liver died later on the waiting list during the study period. If a person died, that can be viewed as a “life saved” during the study period by the availability of the additional liver and allocation based on that model. This is repeated for each month in the study period, assuming that in each month 25 additional livers are available, and determining the number of lives saved.

Using this approach, we estimated the number of lives saved for each candidate liver allocation model, and then compared the results. We chose to consider 25 additional livers (rather than a different number) because we thought it was large enough so that conclusions about the relative performance of the scores could be reached, and small enough so that it represented only a 10% increase in the number of livers available. This gave us more confidence that the relative performance of the models on these additional livers would be similar to their relative performance had they been used to allocate the actually available livers. In a secondary analysis, we calculated the number of lives saved based on the assumption that 50 (rather than 25) additional livers were available each month.

One complication is that in the current allocation system in the U.S., livers are not allocated to the person with the highest MELD score in the country. Instead, they are allocated to the person (if any) with the highest MELD score in the Donation Service Area (DSA) where the patient is listed. There are 59 DSA in the U.S. (13). It was not possible for us to allocate the hypothetical livers to patients with the highest score in each DSA because DSA information was not included in the STAR file. To approximate this system of allocation, therefore, we randomly divided the patients into five hypothetical DSA’s for each of the 11 regions, and allocated the hypothetical livers based on those with the highest risk score in each hypothetical DSA.

Another complication is the fact that sometimes, a patient who is hypothetically allocated a liver in our experiment will, in reality receive a transplant in a later month. In these cases, we assumed that the transplant received in a later month would be available for another patient. It was then hypothetically allocated to another patient, along with the 25 additional hypothetical livers in that later month.

Often, due to acute health problems or other reasons, patients on the waiting list are considered to be not suitable for liver transplant at a point in time. These patients are listed as “temporarily inactive” in the hope that their condition would improve, and they will be restored to active status. While they are temporarily inactive, they were not eligible for a transplant. In our hypothetical allocation, we also did not allocate livers to patients while they were temporarily inactive.

If a patient was removed from the waiting list due to deteriorating condition, we treated that as a “death” in calculating the number of lives saved. This is consistent with the approach used by other analyses of waiting list mortality (e.g., Fink et al.) (14).

Statistical Analysis

Statistical analysis was performed using SAS software, Version 9.2 (Cary, NC, USA) (15). To assess the statistical significance of differences in estimated numbers of lives saved, we used McNemar’s test (16).

RESULTS

There were 27,473 patients with ESLD who were on the liver transplant waiting list at any point in time during the study period from January 2006 through June 2009. This corresponded to 370,082 person-months on the waiting list. The characteristics of the patients and results of each month are shown in Table 2. Sixty four percent of the patients were male, and almost half (45%) were between 50 and 59 years of age. On average, there were 253 transplants per month, so the addition of 25 hypothetical donated livers represents only a 10% increase in the number of transplants.

Table 2.

Liver Transplant Waiting List Experience from January 2006 to June 2009

Characteristics of Patients with ESLD on the Liver Transplant Waiting List
Number of Patients	27,473
Sex
Male	17530 (64%)
Female	9953 (36%)
Age
18–39	1,614 (6%)
40–49	4,986 (19%)
50–59	11,513 (45%)
60–69	6,851 (27%)
70–79	855 (3%)
Characteristics of Person-Months
Number of Person-months	370,082
Outcomes each month
Number (%) surviving the month, and staying on the list	353,413 (96%)
Number (%) dying	3,650 (1%)
Number (%) leaving list due to becoming medically unsuitable	1,790 (0.5%)
Number (%) transplanted	10,635 (3%)
Number (%) leaving list due condition improved	594 (0.2%)
Average number of patients on the waiting list per month	8,415
Average deaths (or removal for medical unsuitability) per month	130
Average number of transplants per month	253

Table 3 shows the estimated number of lives saved during the study period by allocating 25 additional livers each month using each allocation model. The number of lives saved basing allocation on waiting list time was significantly lower than basing allocation on any of the seven models (P<0.0001). There were no significant differences in the number of lives saved between MELD, MESO, ReFit MELDNa, MELDNa, and ReFit MELD, although of these, MELD performed the poorest, and ReFit MELD performed the best. UKELD resulted in significantly fewer lives saved than the other scores, and Updated MELD score performed significantly worse than the best-performing score.

Table 3.

Estimated Number of Lives Saved if 25 Additional Livers were Available Each Month and the Additional Livers were Allocated by Each Model(From January 2006 through June 2009)

Allocation Model	Number of Lives Saved	P Values for pairwise comparisons
Time on the Waiting List	144	P < 0.0001 (compared to each of the other scores)
UKELD (8)	788	P=0.022 compared to Updated MELD P<0.001 compared to each of the other scores.
Updated MELD (7)	846	P=0.25 compared to MELD, br1>P=0.67 compared to Refitmeldna P=0.072 compared to MESO P=0.067compared to ReFitMELDNA P=0.054 (compared to MELDNa) P=0.021 compared to ReFit MELD
MELD (1, 2)	867	P> 0.17 compared to each of the scores below on this table
MESO Index (5)	879	P> 0.58 compared to each of the scores on this table below
ReFit MELDNa (9)	882	P> 0.73 compared to each of the scores below on this table
MELDNa (6)	884	P=0.86 compared to ReFit MELD
ReFit MELD (9)	887

In secondary analyses, we assumed 50 additional livers were available each month (Table 4). This analysis resulted in generally similar results, although MELD performed significantly worse than the best performing scores.

Table 4.

Estimated Number of Lives Saved if 50 Additional Livers were Available Each Month and the Additional Livers were Allocated by Each Model (Starting from January 1, 2006)

Allocation Model	Number of Lives Saved	P Values for pairwise comparisons
Time on the Waiting List	307	P < 0.0001 (compared to each other model on this table)
UKELD (8)	1360	P=0.086 compared to Updated-MELD, and P<0.002 compared to each other score below
Updated MELD (7)	1407	P= 0.029 compared to MELD, and p<.0.002 compared to each other scores below
MELD (1, 2)	1449	P=0.11 compared to MESO P=0.095 compared to ReFit MELD P=0.043 compared to ReFit MELDNa P=0.033 compared to MELDNa
MESO Index (5)	1471	P>0.32 for each score below
ReFit MELD (9)	1473	P>0.45 compared to each score below
ReFit MELDNa (9)	1488
MELDNa (6)	1488

For comparison, we also calculated c-statistics to quantify the degree to which each monthly score predicted death in that month. In order from best to worst, the results were MELDNa, (c=0.835), ReFit MELDNa (c=0.834), MESO (c=0.826), ReFit MELD (c=0.824), MELD (c=0.822), Updated MELD (c=0.816), UKELD (c=0.805).

DISCUSSION

Based on the current liver-transplantation policy, donor organs are allocated to patients with the highest risk of death, which is estimated using the MELD scoring system. Although the MELD score is widely used as a liver allocation system, a few other allocation models have recently been proposed in an attempt to improve the prediction of survival and enhance liver allocation among candidates for liver transplantation (5–7).

A widely used approach for comparing liver allocation models is the c-statistic. The c-statistic is a quantification of the performance of a quantitative predictor of a binary outcome, integrating over all possible cut-points of the quantitative predictor. However, in the context of liver transplantation, where relatively few livers are available, the performance of a risk model should be based on high cut-offs; specifically on how well relatively high risk scores predict which patients are at imminent risk of death. Another drawback to the c-statistic is that it is difficult to interpret the epidemiologic implications of a difference in c-statistics.

Another approach to comparing risk models is to use the simulation program “Liver Simulated Allocation Models” (LSAM) created by the U.S. Scientific Registry of Transplant Recipients (17, 18). Given input on a series of waiting list patients and their risk for various outcomes, and a set of allocation policies, this program uses Monte Carlo methods to simulate outcomes (19). The series of waiting list patients and their outcomes can be derived from historical experience. However, because some of the patients who were allocated transplants in reality might not receive a transplant in a specific simulation, some method must be used to provide information regarding what would have happened to that patient had they not received a transplant. This can be done based on modeling or matching with other patients. To our knowledge, there is no published comparison of liver allocation models based with respect to waiting list mortality based on LSAM.

As an alternative, we developed a way of comparing risk models using historical data that does not require modeling assumptions or matching. The result of our approach is interpretable as the number of lives saved had additional livers been available. In the work reported here, we used this approach to evaluate previously proposed risk models. However, our approach might also be useful in the development of new risk models, by providing a metric to optimize in the development of a new model.

There are several caveats to the use of our method. First, it is based on the assumption that if a person who died on the waiting list had actually been given a liver earlier, they would not have died. This assumes that the death was due to the primary liver disease. If, for example, the cause of death while on the waiting list was unrelated to liver disease then this person would have died even if the person had received a transplant. To the extent that waiting-list deaths were not related to the primary liver disease, our method over-estimates the number of lives saved.

A second caveat is that, while our method provides direct insight into how an allocation score performs had additional livers been available, this is not necessarily the same as how it would perform as the primary allocation model for all livers. We chose to compare the models on additional hypothetical livers because to in order to compare them using the available livers we would have had to speculate regarding what would have happened to individuals who actually received a transplant, had they not received a transplant. We believe the performance of a model on a set of additional livers that constitutes only a 10% increase in the number of livers (i.e., 25 per month) is a good indication of how it would perform as the primary allocation model.

A third caveat is that our method does not take into consideration the effect that allocation of additional livers based on a specific risk score will have on the structure of the waiting list. The median score will tend to decline, and this effect will be cumulative over time, resulting in reduced mortality. If this effect varies by allocation score, it could affect the relative number of lives saved by the different scores over an extended period of time.

Our approach compares models strictly with respect to waiting list mortality, without consideration of post-transplant survival probabilities of different types of patients. This policy is in accordance with the mandate of the Department of Health and Human Services as stated in the “final rule” (20). In contrast, it might be argued that post-transplant mortality risk should be considered in prioritizing patients for liver transplantation (21, 22). This would reduce the probability that scarce livers would be provided to patients who would die soon after transplantation, and maximize the utility of the available livers. In fact, one of the reasons cited for capping MELD at 40 was to reduce the likelihood of futile transplants (1). Recently, Schaubel et al. (23) proposed an interesting allocation scheme that takes into account both waiting-list mortality and post-transplant mortality, in an effort to maximize lifetime gained through liver transplantation.

Even if the policy changes and post-transplant mortality is taken into consideration in the future liver allocation policy, the approach that we used to compare models with respect to waiting list mortality would be useful. This is because a transplant-benefit score such as the one proposed by Schaubel et al. (23) is based, in part, on a model to predict waiting-list mortality

In summary, we used a novel approach to quantify the performance of a liver transplant priority models. While our approach is based on some strong assumptions, the assumptions required are different than those required by programs such as LSAM, or statistics such as the c-statistic. Thus, our approach can supplement the other approaches and provide additional insight into the relative performance of allocation models. Based on our approach, some recently proposed models were found to perform somewhat better than the MELD score, and significantly better than the UKELD score.