A modular simulation framework for organ allocation

Abstract

BACKGROUND:

We describe and validate a new simulation framework addressing important limitations of the Simulated Allocation Models (SAMs) long used to project population effects of transplant policy changes.

METHODS:

We developed the Computational Open-source Model for Evaluating Transplantation (COMET), an agent-based model simulating interactions of individual donors and candidates over time to project population outcomes. COMET functionality is organized into interacting modules. Donors and candidates are synthetically generated using data-driven probability models which are adaptable to account for ongoing or hypothetical donor/candidate population trends and evolving disease management. To validate the first implementation of COMET, COMET-Lung, we attempted to reproduce lung transplant outcomes for U.S. adults from 2018–2019 and in the 6 months following adoption of the Composite Allocation Score (CAS) for lung transplant.

RESULTS:

Simulated (median [Interquartile Range, IQR]) vs observed outcomes for 2018–2019 were: 0.162 [0.157, 0.167] vs 0.170 waitlist deaths per waitlist year; 1.25 [1.23, 1.28] vs 1.26 transplants per waitlist year; 0.115 [0.112, 0.118] vs 0.113 post-transplant deaths per patient year; 202 [102, 377] vs 165 nautical miles travel distance. The model accurately predicted the observed precipitous decrease in transplants received by type O lung candidates in the six months following CAS implementation.

CONCLUSIONS:

COMET-Lung closely reproduced most observed outcomes. The use of synthetic populations in the COMET framework paves the way for examining possible transplant policy and clinical practice changes in populations reflecting realistic future states. Its flexible, modular nature can accelerate development of features to address specific research or policy questions across multiple organs.

Background

Computer simulations of hypothetical organ allocation strategies have informed all contemporary U.S. allocation policies in the last two decades, including the Lung Allocation Score (LAS) and its revisions, and now the Composite Allocation Score (CAS) system.^1–5 The importance of simulation in predicting population outcomes and guiding policy and clinical practice will likely increase as transplantation practices rapidly evolve. Ongoing donor shortages^6–8 have spurred innovations in ex vivo organ perfusion^7,9,10 and have expanded the boundaries of what donor characteristics are considered high-risk.^7,11,12 Patterns of demand will likely continue to evolve with advances in disease management resulting in older and more complex candidates with more comorbidities undergoing transplant.^13–21

To date, the primary simulation tools used have been the Simulated Allocation Models (SAMs) developed by the Scientific Registry of Transplant Recipients (SRTR): the LSAM (liver),²² KPSAM (kidney/pancreas),²³ and the TSAM (thorax).^24,25 An emerging SAM, the Organ Allocation Simulation Model (OASIM) from the SRTR, builds upon the legacy SAMs to better enable simulation under the CAS system.²⁶ Changes in the supply of organs and in the risk trajectories of transplant candidates pose a challenge to the continued use of the current modeling approach, however. All SAMs rely on direct sampling of SRTR data to simulate candidate and donor populations and their characteristics.^22–24 Direct reliance on this historical data limits the policymaking and research communities’ abilities to explore the potential effects of shifting donor or candidate populations in a realistic clinical and policy future state. The requirement to obtain access to sensitive SRTR data may also practically limit the number of researchers, developers, or other stakeholders working to improve transplant policy or transplant simulation. Here, we describe a new simulation framework, the Computational Open-source Model for Evaluating Transplantation (COMET), which addresses these limitations and which is designed to maximize adaptability for answering new research and policy questions across multiple organs. We provide details on the development and validation of COMET-Lung (the initial iteration of COMET) and demonstrate how COMET-Lung may be used to examine a transplant policy change.

Methods

The COMET framework consists of interconnected modules coded in the R language (R Foundation for Statistical Computing, Vienna, Austria). While the framework is generic to any transplanted organ, the specific modules and their inputs/outputs are organ specific. A set of linked, organ-specific modules adhering to the COMET framework constitutes an organ specific COMET model.

This study used data from the SRTR. The SRTR data system includes data on all donors, wait-listed candidates, and transplant recipients in the U.S., submitted by the members of the Organ Procurement and Transplantation Network (OPTN). The Health Resources and Services Administration (HRSA), U.S. Department of Health and Human Services provides oversight to the activities of the OPTN and SRTR contractors.

The COMET framework

COMET is an agent-based model (ABM). ABMs are “bottom-up” computer models which mechanistically simulate interactions of individual entities (“agents”) over time to examine aggregate, system-level outcomes.^27,28 Agent types within COMET are donors, candidates, hospitals (in which donors are nested), and transplant centers (in which candidates are nested). Each agent has a set of properties, fixed or time-varying. Fixed properties—candidate diagnosis type or race, for example—can be distributed across a population of agents according to real or hypothetical population characteristics. Time-varying properties represent an agent’s status at a point in time. Each agent in an ABM may undergo status changes probabilistically (e.g., accepting an organ, leaving the transplant list, or dying) according to rules.²⁸ While the interactions of relatively simple agents can give rise to complex system behaviors, an appeal of ABMs like COMET lies in their ability to model complexity with relative transparency and intuitiveness.

COMET’s geographic domain is the U.S., and the model operates over fixed, 24-hour simulated time cycles. Figure 1 depicts the COMET framework. The Donor Generation and Candidate Generation modules produce new synthetic donor and candidate populations, respectively, each day with assigned traits that serve as required inputs for downstream modules. Once a candidate or donor is created, their status may be modified through the actions of specific downstream modules. A Pre-transplant Risk module simulates relevant events affecting waitlist mortality or removal from the waitlist. As new donors enter, a Screening module identifies donor-specific subsets of potentially compatible candidates daily. Next, a Matching module implements specific allocation criteria, and a subsequent Acceptance module models the decision as to whether each candidate would accept an offered organ. Once a candidate receives a transplant, a Post-transplant Risk module models post-transplant survival. A separate Outcomes module compiles population-level statistics.

Figure 1.

Diagram of the COMET framework. Each box represents a model module. COMET, Computational Open-source Model for Evaluating Transplantation.

Module descriptions

Here, we explain the components and logic of version 0.1.0 of each module of the first instance of COMET, COMET-Lung. Table 1 summarizes the standardized inputs and outputs of each module. Tables S1–S4, S6–S8, and S10–S11 (Supplement) provide specifications necessary for development of new module versions. The code for all modules of COMET-Lung is available at https://github.com/ClevelandClinicQHS/COMET. We have created COMET to be open source, meaning the source code is freely available and may be modified by others.

Table 1.

Inputs and Outputs for Each COMET-Lung Simulation Module

Module	Inputs	Outputs
Donor Generation	–	Donor identifier, Hospital of Death, Date of death, Age, Sex, Race/ethnicity, Height, ABO blood Type, > 20 years smoking history (y/n), Cause of death, Donor-after cardiac death (DCD) status, Lungs available (left, right, or both), Procured for transplant (y/n)
Candidate Generation	–	Candidate identifier, Transplant Center, Date added to waitlist, Diagnosis group (A, B, C, or D), Lungs needed (double, single, either), Age at listing, Sex, Race/ethnicity, Height, Weight, ABO blood type, FEV1, FVC, PCO2, PaO2/FiO2 (P/F) ratio, PO2, PCO2 increase of 15% in 6 months (y/n), Mean pulmonary artery pressure, Systolic pulmonary artery pressure, Cardiac Index (L/min/m2), Central venous pressure at rest, Supplemental oxygen frequency, Supplemental oxygen rate, 6-minute walk distance, Baseline bilirubin, Baseline creatinine, Functional status, Diabetes, ECMO, Panel Reactive Antibody, Prior living donor status, Creatinine increase of 150% (y/n), Bilirubin increase of 50%, Continuous Mechanical Ventilation, Additional detailed diagnosis parameters needed to calculate pre-and post-transplant risk
Pre-transplant Risk	From Candidate Generation Module – Candidate identifier, Diagnosis group, Age, Weight, Height, PCO2, PCO2 increase of 15% in 6 months (y/n), Systolic pulmonary artery pressure, Ventilation, Supplemental oxygen, 6-minute walk distance, Baseline bilirubin, Baseline creatinine, Functional status, Detailed diagnosis	Time on waitlist, Whether died on waiting list, Whether removed from waiting list
Screening	From Donor Generation Module – Donor identifier, ABO blood type, Lungs available, Height From Candidate Generation Module – Candidate identifier, Candidate ABO blood type, Lungs needed, Height, Diagnosis group	List of eligible Candidates for a given organ on a given day
Matching (LAS)	From Donor Generation Module – Donor identifier, ABO blood type, Hospital of death From Candidate Generation Module – Candidate identifier, Transplant center, ABO blood type, Diagnosis group, Age at listing, Height, Weight, FVC, PCO2, Systolic pulmonary artery pressure, Cardiac index, Central venous pressure at rest, Ventilation, Supplemental oxygen, 6-minute walk distance, Baseline bilirubin, Baseline creatinine, Functional status, Diabetes, Detailed diagnosis, Oxygen requirement at rest, Continuous Mechanical Ventilation (y/n), Bilirubin increase of 50% (y/n), Creatinine increase 50% (y/n)	Descending-priority ranked list of matching candidates for each available lung or pair of lungs
Matching (CAS)	From Donor Generation Module – Donor identifier, Hospital of death From Candidate Generation Module – Candidate identifier, Transplant center, ABO blood type, Diagnosis group, Age at listing, Height, Weight, FVC, PCO2, Systolic pulmonary artery pressure, Cardiac index, Ventilation, Supplemental oxygen, 6-minute walk distance, Baseline bilirubin, Baseline creatinine, Functional status, Detailed diagnosis, Continuous Mechanical Ventilation (y/n), ECMO	Descending-priority ranked list of matching candidates for each available lung or pair of lungs
Acceptance	From Matching Modules – Candidate-donor matches (ranked) From Donor Generation Module – Donor identifier, Age, Smoking history > 20 years (y/n), Donor after cardiac death (DCD) From Candidate Generation Module – Candidate identifier, Transplant center	Number of offers for each available argan, Whether organ transplanted, Candidate receiving organ, Day of transplant, Distance organ traveled
Post-transplant Risk	From Candidate Generation Module – Candidate identifier, Diagnosis group, Age at listing, Cardiac index, Ventilation, 6-minute walk distance, Baseline creatinine, Functional status, Detailed diagnosis, Continuous mechanical ventilation, ECMO	Date of death
Outcomes	From Pre-transplant Risk Module – Time on waitlist, Removal from waitlist, Died on waitlist (y/n), Date of death From Acceptance Module – Number of offers for each organ, Whether organ transplanted, Candidate receiving organ, Transplant center, Date of transplant, Distance organ traveled From Post-transplant Risk Module – Date of death	Donor count, Candidate count, Transplant volume, Transplant rate, Median time on waitlist, Waitlist death rate, Median distance organ traveled, Median times an organ is offered, Number of procured organs transplanted, Number of recovered organs not utilized, Post-transplant death rate, 1-year post-transplant survival, 2-year post-transplant survival, Median post-transplant survival

CAS, composite allocation score; ECMO, extracorporeal membrane oxygenation; FEV1, forced expiratory volume in 1 second; FiO2, fraction inspired oxygen; FVC, forced vital capacity; LAS, lung allocation score; O2, oxygen; PaO2, partial pressure of arterial oxygen; PCO2, partial pressure of carbon dioxide; PO2, partial pressure of oxygen; PPV, positive pressure ventilation.

Donor and Candidate Generation Modules –

We used hierarchical Bayesian regression to model the joint distribution of key characteristics of lung donors from the SRTR between January 1, 2015, and June 30, 2021. Separately, we modeled the joint distribution of key characteristics for candidates listed between February 19, 2015, and September 1, 2021. This work is described and validated elsewhere.²⁹ Briefly, we developed hierarchical Poisson log-linear regression models to estimate numbers of candidates in each diagnosis group (group A, obstructive lung disease; group B, pulmonary vascular disease; group C, cystic fibrosis and immunodeficiency disorders; group D, restrictive lung diseases) and annual rates at which new donors are listed from each of 1,691 hospitals, and at which new candidates are listed at each of 61 transplant centers. Second, we developed separate probabilistic hierarchical Bayesian networks modeling the joint variability among 10 donor and 36 candidate characteristics relevant to waitlist survival, donor-candidate matching, and post-transplant survival. These 2 modules generate new agents each modeled 24-hour time increment.

Pre-transplant Risk Module –

We based the initial version of the Pre-transplant Risk Module on the CAS waitlist survival model used in the Continuous Distribution system implemented March 9, 2023.³⁰ Module inputs (Table 1 and Table S3) correspond to each of the inputs of the CAS waitlist survival model. We recalibrated the CAS waitlist survival model to address known issues with overestimation of risk³¹ (Supplement).

Screening Module –

Each day of the simulation, for each available lung, the Screening Module outputs a list of candidates who meet compatibility criteria based on matching donor and patient ABO blood types, available donor lungs (double or single), type of transplant needed by candidate (double, single, or either), and donor-candidate height difference. With a Delphi panel of 3 cardiothoracic transplant surgeons, we developed height-based criteria according to candidate diagnosis group (see Table S5).

Matching Modules –

We developed 2 Matching module versions. The first is based on the 2015 LAS calculation,³² the allocation algorithm in effect during the period when the SRTR training data was collected. We also developed a version based on the CAS algorithm in effect since March 2023.³⁰ Tables S6 and S7 list required inputs for each matching module. A matrix of pre-calculated distances between hospitals and transplant centers provides inputs for both modules.

Acceptance Module –

The Acceptance Module determines which candidate, if any, ultimately accepts each organ offer. It is assumed that decisions to accept offers are made by center transplant teams on behalf of candidates. Using a logistic regression model based on SRTR data, the module calculates the probability of accepting an available organ, if offered, for each candidate eligible for that organ. Covariates in the model include donor age > 55, donor smoking history > 20 pack years, donor-after-cardiac-death (DCD) status, offer number, and a center effect³³ (Table S9).

Post-transplant Risk Module –

We based the initial version of the COMET-Lung Post-transplant Risk Module on the CAS post-transplant outcomes model³⁰ (inputs shown in Table S10). We recalibrated the CAS posttransplant model to improve the fit to SRTR data using the same approach applied to the CAS waitlist survival model.

Outcomes Module –

The Outcomes Module (Table S11) receives inputs from the Pre-transplant Risk, Acceptance, and Post-transplant Risk modules, aggregating the experiences of individual COMET agents and calculating summary statistics. These statistics summarize waitlist deaths, transplants, post-transplant deaths, organ utilization, donor-recipient distance, time-on-waitlist, and offer number of accepting candidate.

Validations

Validation 1 tested COMET-Lung’s ability to synthesize a population of virtual candidates who go on to experience outcomes similar to those of a 24-month historical cohort from the SRTR. Our validation cohort consisted of candidates aged 18 and older who were on the waiting list as of January 1, 2018, plus candidates added from January 1, 2018, through December 31, 2019. We used the LAS Matching Module since LAS was the matching algorithm in effect during the validation interval. We compared COMET-Lung outputs to 2018–2019 historical data. To produce stable results and to provide distributions of outcomes, we ran the simulation 1,000 times.

Validation 2 – In the 6 months following the March 9, 2023, transition to CAS-based lung allocation, the U.S. saw a drop of nearly 8% in the proportion of lung transplant recipients with ABO blood type O.³⁴ We assessed COMET-Lung’s ability to predict this shift by comparing outputs under the LAS Matching Module with those under the CAS Matching Module. For this comparison, we used the same 24-month 2018–2019 cohort used in Validation 1.

Example policy simulation

We used COMET-Lung to project the outcomes of hypothetical re-weightings of the CAS in which the Placement Efficiency component of the score (currently weighted at 10%) was alternatively weighted at either 0% or 20%. Placement efficiency penalizes possible matches that involve greater travel distances.

This study was approved as exempt research by the Cleveland Clinic Institutional Review Board and abides by the ISHLT ethics statement.

Results

Validation 1

Table 2 compares the actual outcomes for U.S. lung transplant candidates from January 1, 2018, through December 31, 2019, to the aggregated outcomes from 1,000 runs of COMET-Lung. The simulated rate of transplants per waitlist year closely matched observation (1.25 [IQR 1.23, 1.28] vs 1.26, respectively). The median recipient waited 67 [IQR 22, 191] days for a transplant from a donor who died at a hospital 202 [IQR 102, 377] nautical miles away from their transplant center. This compared to observed values of 44 [IQR 14, 132] days and 165 [70, 243] nautical miles, respectively. COMET-Lung estimated that post-transplant deaths during 2018–2019 would occur at a rate of 0.115 [0.112, 0.118] per patient year, compared to an observed rate of 0.113. COMET-Lung estimates of median annual rate of death among candidates on the waiting list (0.162 [IQR 0.157, 0.167] were marginally lower than the observed waitlist death rate of 0.170. However, the most substantial divergence between predicted and observed waitlist survival did not occur until after 300 days (Figure 2), by which point most candidates had left the waiting list. We compared waitlist survival between each of the 1,000 COMET simulated results and empirical SRTR data using log-rank tests. For purposes of these tests, and because the CAS waitlist mortality model does not provide risk estimates beyond day 365, we censored patients after 365 days on the waiting list (at which point, there were roughly 15% of patients remaining both in the synthetic and real-world cohorts). Of our 1,000 model runs, 98.8% had a log-rank p-value greater than 0.05. Figure 3 compares simulated and observed post-transplant survival during the period of January 1, 2018, through March 12, 2020 (date of U.S. pandemic declaration). Of 1,000 log rank tests comparing modeled to actual post-transplant survival, 98.3% had a p-value greater than 0.05.

Table 2.

Validation 1: Comparison of Observed and Simulated Results for the Period of January 1, 2018 through December 31, 2019

Outcome	Observed (2018-2019) [IQR]	1,000 LAS Simulations median [IQR]
Waitlist Deaths per Waitlist Year	0.170	0.162 [0.157, 0.167]
Transplants per Waitlist Year	1.26	1.25 [1.23, 1.28]
Time to transplant (Days)	44 [14,132]	67 [22,191]
Distance from donor to candidate (Nautical Miles)	165 [70, 243]	202 [102,377]
Post Transplant Deaths per Post-transplant Year	0.113	0.115 [0.112, 0.118]

LAS, lung allocation score.

The simulation was run 1,000 times, each for a period of 730 days (2 years), and the post-transplant period was censored at day 802 (corresponding to date of U.S. pandemic declaration). The LAS simulations used the 2015 LAS formula that was in place during 2018–2019. IQR for simulated time to transplant and nautical miles is represented by the median of the median, [first quartile, third quartile].

Figure 2.

Kaplan-Meier plot of waitlist survival. The light red line reflects the waitlist survival for 2018–2019 lung transplant candidates as listed in the Scientific Registry of Transplant Recipients (SRTR). The black line is the median Kaplan-Meier curve as a result of 1,000 runs using the COMET-Lung Pre-transplant Risk Module. For all Kaplan-Meier estimates, individuals were coded as dying on the waiting list with an event of either death or becoming too sick to receive a transplant. Candidates were censored at the time of transplant or waitlist removal for reasons other than being too sick to transplant. The shaded areas around the lines represent the 95% confidence intervals for the estimates. COMET, Computational Open-source Model for Evaluating Transplantation.

Figure 3.

Kaplan-Meier plot of post-transplant survival. The red line represents post-transplant survival for the 2018–2019 lung transplant candidates as listed in the Scientific Registry of Transplant Recipients (SRTR), with post-transplant survival estimates censored on March 12, 2020 (date of U.S. declared COVID-19 pandemic). The black line is the median Kaplan-Meier curve as a result of 1,000 runs using the COMET-Lung Post-transplant Risk Module. For all Kaplan-Meier estimates, individuals were coded as dying post-transplant if they experienced a post-transplant death and censored otherwise. The shaded areas around the lines represent the 95% confidence intervals for the estimates. COMET, Computational Open-source Model for Evaluating Transplantation.

Validation 2

Table 3 compares the TSAM-predicted and COMET-Lung-predicted distributions of lung transplant volume by ABO blood type under both LAS and CAS to the observed distributions in the 6 months before and after the March 9, 2023 real-world transition from LAS to CAS. While TSAM predicted an increase in transplants for Type O candidates from 45.6% to 51.1% (+5.5%),³⁵ COMET-Lung predicted a six-month decrease in transplants for type O candidates from 45.1% to 37.1% (−8.0%), similar to the observed decrease of 46.5% to 39.0% (−7.6%).³⁴

Table 3.

Validation 2: TSAM-Predicted and COMET-Lung-Predicted vs Observed Changes in Distribution of ABO Blood Types Among Lung Transplant Recipients following the March 9, 2023 Implementation of the Composite Allocation Score (CAS) System

TSAM-predicted35 (based on 2018-2019 SRTR data)	LAS *	CAS *	Change
A	38.6%	34.0%	−4.6%
AB	4.1%	3.8%	−0.3%
B	11.7%	11.1%	−0.6%
O	45.6%	51.1%	+5.5%
COMET-Lung-predicted (based on 2018-2019 SRTR data)	LAS	CAS	Change
A	39.8%	43.9%	+4.1%
AB	3.9%	5.5%	+1.6%
B	11.2%	13.4%	+2.2%
O	45.1%	37.1%	−8.0%
Observed 34	LAS (6 months before transition to CAS)	CAS (First 6 months after transition to CAS)	Change
A	38.7%	43.0%	+4.3%
AB	3.8%	4.5%	+0.7%
B	10.9%	13.6%	+2.7%
O	46.6%	39.0%	−7.6%

CAS, composite allocation score; COMET, Computational Open-source Model for Evaluating Transplantation.

This table shows the distributions of lung transplant volume by ABO blood type as predicted by the Thoracic Simulated Allocation Model (TSAM) and COMET-Lung (both models were based on patients on the transplant list between January 1, 2018–December 31, 2019) under both the Lung Allocation Score (LAS) allocation system and the CAS system. These are compared with the observed distributions in the 6 months before and after the March 9, 2023 transition from LAS to CAS.

^*Median proportion over 1,000 runs.

Policy simulation

We compared outcomes across four scenarios for the same 2018–2019 cohort used in validation. The first column of Table 4 recapitulates the outcomes predicted by COMET-Lung under the LAS. The remaining 3 columns display results for scenarios using the CAS Matching Module with a default Placement Efficiency weight of 10% and with alternative weights of 0% and 20%, respectively. COMET-Lung suggested that transitioning from LAS to CAS during this period would have led to substantial decreases in rates of both waitlist deaths (from 0.162 [IQR 0.157, 0.167] to 0.140 [IQR 0.136, 0.145]) and post-transplant deaths (from 0.115 [IQR 0.112, 0.118] to 0.106 [IQR 0.103, 0.109]). This transition was associated with reduced predicted time-to-transplant (from 67 [IQR 22, 191] to 44 [IQR 10, 163] days) while increasing the distance transplanted organs would travel (from 202 [IQR 102, 377] to 364 [IQR 137, 677] nautical miles). Among the three alternative CAS scenarios, reducing the influence of distance and travel time on CAS led to substantially decreased time-to-transplant, but increased travel distance.

Table 4.

Comparison of Simulated Results (1,000 runs) for the Period January 1, 2018 through December 31, 2019 Under 4 Allocation Scenarios: LAS, CAS with the Currently-Used 10% Placement Efficiency (PE) Weight, CAS with 0% PE Weight, and CAS with 20% PE Weight

Characteristic	LAS median [IQR]	CAS median [IQR]	CAS 0% placement efficiency median [IQR]	CAS 20% placement efficiency median [IQR]
Offer rank of candidate receiving organ	7 [2, 23]	7 [2, 25]	8 [2, 27]	7 [2, 23]
Waitlist deaths per patient year	0.162 [0.157, 0.167]	0.140 [0.136, 0.145]	0.136 [0.132, 0.140]	0.143 [0.139, 0.148]
Transplants per patient year	1.25 [1.23, 1.28]	1.21 [1.18, 1.24]	1.23 [1.20, 1.25]	1.20 [1.18, 1.23]
Time to transplant (Days)	67 [22, 191]	44 [10, 163]	27 [6, 139]	53 [14, 175]
Distance from donor to candidate (Nautical Miles)	202 [102, 377]	364 [137, 677]	787 [436, 1,332]	280 [69, 561]
Post-transplant deaths per patient year	0.115 [0.112, 0.118]	0.106 [0.103, 0.109]	0.104 [0.101, 0.106]	0.107 [0.104, 0.111]

CAS, composite allocation score; LAS, lung allocation score.

The post-transplant period was censored at 802 days (reflecting timing of U.S. pandemic declaration). IQR for time to transplant, nautical miles, and offer rank is represented by the median of the median, [first quartile, third quartile].

Discussion

We report the creation and validation of the initial version of a new transplant simulation model, the Computational Open-source Model for Evaluating Transplantation (COMET). COMET is an agent-based model that simulates interactions of synthetic donors and candidates to examine system-level outcomes for the purpose of guiding clinical practice and policy changes impacting organ transplantation.

This approach offers numerous advantages over current SAMs. First, the creation of realistic, contemporary synthetic populations paves the way for manipulating population parameters to account for ongoing or hypothetical trends in the makeup of donor or candidate populations (e.g., shifts in demographics or disease burden of candidates such as the precipitous decline in the number of Cystic Fibrosis candidates). This use of synthetic populations also allows investigators without SRTR data access to conduct simulation research. Second, an agent-based approach lends itself to studying complex systems such as organ transplantation given its use of heterogeneous “agents” at multiple levels that interact in a shared environment. For example, COMET associates transplant candidates with specific transplant centers and can account for how centers differ in their organ offer acceptance practices and in the extent to which they neighbor other centers “competing” for the same organs. Third, the modular and open-source nature of COMET can foster more rapid improvement of transplant simulation capabilities and their application to specific research questions. Because each module captures a distinct process relevant to transplantation, and because each uses standard, transparent inputs and outputs, a group wishing to develop an enhancement to a particular aspect of COMET (e.g., more realistically modeling organ offer acceptance) can focus on that aspect alone and “plug” their new module into the existing framework.

In the first validation, COMET-Lung slightly underestimated waitlist mortality, though the largest divergence between predicted and observed values occurred well beyond the typical median waiting period for transplants of 3 months,³⁶ leading to minimal impact on simulated transplant activity. Log-rank tests comparing each of 1,000 simulations to reality had p-values above 0.05 98.8% of the time, suggesting a lack of statistically significant difference. The model accurately predicted the rate of transplants as well as post-transplant mortality rate and post-transplant survival. The model overestimated wait times by three weeks, though the estimate was well within the interquartile range of observed values.

The second validation demonstrated COMET-Lung’s ability to accurately predict a real-world precipitous decline in transplant availability for ABO blood type O candidates after transition from the LAS to the CAS allocation system. This unanticipated drop noted in the 6-month United Network for Organ Sharing (UNOS) monitoring report led to an urgent policy change to the CAS algorithm, adjusting the weights assigned to each blood type to give more weight to ABO blood type O candidates.³⁷ The TSAM model used to project the impact of CAS before its implementation did not predict this decline; instead, it projected a modest increase in type O candidates receiving transplants.³⁵

We used COMET-Lung to perform a simulated policy analysis comparing the outcomes stemming from four alternative transplant allocation scenarios among candidates from 2018–2019. With a hypothetical transition from LAS to CAS during this period, simulated waitlist deaths, wait times, and post-transplant deaths declined as medical urgency was prioritized over proximity as an allocation consideration. Time will tell whether a similar effect is seen with the real-life transition to CAS that occurred in March 2023. What is apparent soon after the transition to CAS in the U.S. is that travel distances between donor and transplant hospitals are increasing. Comparing the 6-month periods before and after 2023 implementation of CAS, the observed median travel distance between the donor hospital and transplant center increased from 195 (IQR 78–391) to 353 (IQR 129–663) nautical miles³⁴—a close match to the increase projected by COMET-Lung for a hypothetical 2018 LAS-to-CAS transition: 202 [IQR 102, 377] to 364 [IQR 137, 677] nautical miles.

While the module descriptions and results presented here pertain to lung transplantation, the same COMET framework can be applied to other transplanted organs. Doing so will require developing organ-specific modules. In the U.S., allocation for every other organ is expected to transition to the Continuous Distribution framework as lung transplantation has done with its implementation of CAS. Recent challenges following the March 9, 2023, implementation demonstrate the importance of forecasting the impacts of policy changes as accurately as possible. The present work also has implications beyond the U.S., as adapted forms of U.S. organ allocation algorithms have been implemented by numerous transplant systems including Germany, the Netherlands and Eurotransplant.^38–40 In the year 2021, some version of the LAS governed greater than 50% of lung transplant activity worldwide.^36,41

Limitations –

This work represents the first step in establishing a new simulation framework in transplantation. A limitation at this phase is the lack of an interface with which users may readily tune population parameters in the Donor and Candidate Generation modules. A second limitation is the re-use of existing pre- and post-transplant risk models currently in use for calculating candidates’ CAS. Our recent work suggests that these models may be substantively miscalibrated³¹and that the exclusion of accrued waitlist time from these models may limit their utility for prioritizing candidates.⁴² Finally, there is a limited number of decision-making factors included in the COMET-Lung Acceptance Module. We have included the most relevant evidence-based factors which transplant centers appear to consider in their acceptance decisions, as well as center-specific effects capturing varying levels of conservatism, but more fundamental knowledge is needed regarding the cognitive processes behind acceptance decisions.

Future steps –

In the near term, COMET-Lung can be used to identify potential modifications to the CAS system which will enhance social and biological equity and improve transplant outcomes. Additionally, the model can be adapted to address research and policy questions pertaining to other organs. To extend the utility of COMET beyond U.S. borders, it will be important to adapt COMET modules to embody the allocation rules, demographics, disease burden, and risk profiles of other countries. Achieving these and other goals will require creating effective standards for open-source development and documentation of modules. We hope that by building better simulations, the transplant community can create better policies which more efficiently utilize limited donor organs and extend the life-expectancy of transplant populations.

Abbreviations:

ABM

agent-based model

CAS

Composite Allocation Score

COMET

Computational Open-source Model for Evaluating Transplantation

LAS

Lung Allocation Score

OASIM

Organ Allocation Simulation Model

SRTR

Scientific Registry of Transplant Recipients

SAMs

Simulated Allocation Models