Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2016 Sep 1.
Published in final edited form as: Biol Blood Marrow Transplant. 2015 May 29;21(9):1679–1686. doi: 10.1016/j.bbmt.2015.05.019

Establishment of Definitions and Review Process for Consistent Adjudication of Cause-Specific Mortality After Allogeneic Unrelated Donor Hematopoietic Cell Transplantation

Theresa Hahn 1,#, Lara E Sucheston-Campbell 2,#, Leah Preus 2, Xiaochun Zhu 4, John A Hansen 5, Paul J Martin 5, Li Yan 3, Song Liu 3, Stephen Spellman 6, David Tritchler 7, Alyssa Clay 2, Kenan Onel 8,#, Marcelo Pasquini 4,#, Philip L McCarthy 1,#
PMCID: PMC4537799  NIHMSID: NIHMS695648  PMID: 26028504

Abstract

Clinical trials commonly use adjudication committees to refine endpoints, but observational research or genome-wide association studies rarely do. Our goals were to establish definitions of cause-specific death after unrelated donor allogeneic hematopoietic cell transplantation (URD-HCT), estimate discordance between reported and adjudicated cause-specific death, and identify factors contributing to inconsistency in cause-specific deathdetermination. A consensus panel adjudicated cause-specific deathin 1,484 patients who died within 1 year after HCT, derived from 3,532 acute leukemia or myelodysplasia patients after URD-HCT 2000-2011 reported by 151 U.S. transplant centers to CIBMTR. Deaths were classified as disease-related (DRM) or transplant-related (TRM). The panel agreed with >99% of deaths reported by centers as DRM and 80% reported as TRM. Year of transplant (cohort effect) and disease status significantly influenced agreement between panel and centers. Sensitivity analysis of deaths <100 days post-transplant yielded lowest agreement between the panel and centers for myelodysplastic syndrome patients. Standard pre-defined criteria for adjudicating cause-specific deathled to consistent application to similar clinical scenarios and clearer delineation of cause-specific deathcategories. Other studies of competing events like cancer-specific vs treatment-related mortality would benefit from our results. Our detailed algorithm should result in more consistent reporting of cause-specific deathby centers.

Keywords: Allogeneic HCT, Adjudication, Cause-specific mortality, Acute leukemia

INTRODUCTION

Allogeneic hematopoietic cell transplantation (HCT) offers the only curative therapy for some hematologic malignancy/disorder patients, but has a 1-year post-HCT mortality rate exceeding 30%.1 Allogeneic HCT can precipitate a multi-factorial cascade of events, the sequence and severity of which differs between patients. Not all patients who die after HCT experience all potential post-HCT events. Moreover, surviving patients may experience a similar sequence of events as those who did not survive. Patients’ clinical courses can significantly differ across and within cause-specific mortality. Additionally, co-morbidities carry their own risks and are difficult to discern from HCT-specific causes of death.

As an example, a patient with moderate (not severe) graft-versus host disease (GvHD) treated with multiple systemic immunosuppressive agents develops an infection and dies. The initiating (primary) and contributory (secondary) causes are not easily delineated, leading to ambiguity of whether GvHD or Infection should be reported as the primary cause-specific death. Patients with severe GvHD, requiring prolonged immunosuppression might die of severe GvHD without infection. Likewise, patients can die of infection in the absence of GvHD. Hence GvHD and Infection are not always concurrent causes of death. Discerning the initiating and contributing cause-specific deaths are critical in HCT patients who often have competing and correlated outcomes.

Endpoint Assessment Committees are often used to determine clinical trial endpoints,2-3 but are rarely used for observational research. The only published study investigating cause-specific death (GvHD, infection, disease, other) within the first year post-HCT used the primary cause of death reported by the transplant center.4 Two additional studies examined cause-specific death in HCT patients who had survived beyond 2 and 5 years post-HCT.5-6 The first study defined outcomes (death due to disease recurrence, GvHD or infection) but did not review or adjudicate individual cases,5 while the second study used cause of death reported per the National Death Index in addition to review of medical data for individual cases.6 These methods work for landmark analyses or observational studies that describe changes over time, but genetic studies investigating cause-specific deaths that are incorrectly or inconsistently assigned could result in biased estimation of the association between genetic variants and each cause.7

In preparation for a genome-wide association study (GWAS) of cause-specific mortality after unrelated donor (URD) allogeneic HCT, we convened a consensus panel to review and adjudicate cause-specific deaths to reduce endpoint misclassification and subsequent over- or under-estimation of genetic effects. Our on-going GWAS named DISCOVeRY-BMT: Determining the Influence of Susceptibility-COnveying Variants Related to 1-Year mortality after unrelated donor Blood and Marrow Transplant is designed to investigate donor and recipient genetic factors that contribute to 1-year cause-specific mortality after URD-HCT. We report our cause-specific death definitions, process for adjudication and degree of concordance between the causes of death reported by individual transplant centers and the consensus panel.

SUBJECTS AND METHODS

Research Ethics

All patients and donors provided written informed consent for their clinical data to be used for research purposes and were not compensated for their participation. This study was reviewed and approved by the Roswell Park Cancer Institute Institutional Review Board. All patient data were de-identified. Summary data are provided in this manuscript, with the exception of Supplementary Tables 4 and 5 which contain patient-specific data that have been altered slightly to further protect patient identity and confidentiality.

Study Population

Two independent cohorts were studied to determine the consistency of adjudication results. These cohorts were defined as a training and validation cohort for the main genome-wide association study.

Cohort 1

This cohort included 2,609 10/10 HLA-matched, first, T-cell replete, URD-HCT recipients treated with myeloablative or reduced intensity conditioning regimens from 2000-2008 for acute myeloid or lymphocytic leukemia (AML, ALL) or myelodysplastic syndrome (MDS) who were reported to the Center for International Blood and Marrow Transplant Research (CIBMTR) and had banked biorepository samples from recipient and donor.8 Of 2,609 patients, 1,116 (43%) died within 1-year after HCT.

Cohort 2

This cohort includes 572 patients who had an URD-HCT between 2009-2011 who otherwise met the same criteria as Cohort 1, together with 351 patients who were 8/8 HLA-matched (HLA-DQB1 not assessed) URD-HCT between 2000-2011, but otherwise met the same criteria as Cohort 1. Of 923 patients in Cohort 2, 368 (40%) died within 1-year after HCT.

Patient data and recipient/donor blood samples were contributed by 151 transplant centers. Procedures for the completion and review of CIBMTR data collection forms, as well as cause(s) of death, differ by transplant center. The goal of adjudication was to reduce variability in ascertaining the cause-specific deaths in patients with similar sequences of events leading to death.

Cause of Death Adjudication

The CP consisted of 2 adult HCT physicians (MP, PLM), a pediatric hematologist/oncologist (KO), and a HCT clinical epidemiologist (TH). Causes of death and additional action plans (e.g. request for clinical information from the transplant center) were recorded for each case by an independent co-investigator (XZ) using pre-specified nomenclature and notation. Adjudication of Cohort 1 was completed over 8 months via 3 in-person meetings at CIBMTR (Milwaukee WI) and weekly teleconferences. Adjudication for Cohort 2 was completed over 2 days via an in-person session at CIBMTR.

Case report form summaries were provided to the consensus panel and included detailed data summarized in Supplementary Table 1. Each clinical summary was discussed by the consensus panel using information available in submitted forms, autopsy reports, error correction forms and source documents. Discussions continued until a unanimous consensus was reached regarding the causes of death or that additional information was needed from the transplant center. When additional information was needed before adjudicating cause-specific death, up to 3 data queries requesting source documentation or forms data clarification were submitted to the transplant center.

Cause of Death Category Definitions

The primary cause of death was broadly defined as “Disease-Related Mortality” (DRM - related to leukemia/MDS relapse/progression, including death due to toxicity or infection from post-HCT anti-leukemic therapy) or “Transplant-Related Mortality” (TRM – any cause of death not included in DRM), similar to previous HCT studies.3-6 TRM subtypes were further classified as GvHD, Infection, Organ Failure, and Other. Cause-specific deaths were categorized in a hierarchical manner: Disease, GvHD, Infection, Organ Failure then Other, in descending priority.

Table 1 provides detailed definitions and description of clinical scenarios. Briefly, DRM included documented post-HCT disease progression, relapse or death before day +30 post-HCT in patients with a high disease burden pre-transplant. Autopsy confirming presence of disease was coded as DRM. Treatments such as re-induction chemotherapy, donor lymphocyte infusion and second HCT after the index HCT may have caused “TRM-like” deaths however were coded as DRM due to the hierarchical structure and priority for the cause-specific death definitions.

Table 1.

Definition of Clinical Scenarios for Adjudication of Primary and Secondary Causes of Death

Primary Cause-
specific death		 Secondary
cause-specific
death		 Definition of Clinical Scenarios
Disease None
  • 1)

    Documented disease progression (if not in CR pre-HCT) or relapse (if in CR pre-HCT) after HCT, includes evidence of disease on autopsy only

  • 2)

    Patient was not in CR pre-HCT and had a high disease burden (ie., >50% blasts in BM or PB pre-HCT) with no disease evaluation post-HCT and died before Day+100
GVHD None
  • 1)

    Severe grade III-IV acute GVHD on active treatment with no infections, no organ toxicity, no evidence of disease progression or relapse post-HCT, regardless of disease status pre-HCT

  • 2)

    Severe chronic GvHD on active treatment involving only lung or only GI or multiple organs, refractory to immunosuppressive treatment (>3 lines of therapy), no infections, no organ toxicity, no evidence of disease progression or relapse post-HCT, regardless of disease status pre-HCT
Infection None
  • 1)

    Documented infection causing death with no disease progression or relapse, GvHD or organ toxicity post-HCT. Infection leading to sepsis and multi-organ failure should be coded in this category.

  • 2)

    Documented infection causing death, with a history of acute or chronic GVHD or organ failure that had resolved and was not being treated at the time of diagnosis of the infection. Infection leading to sepsis and multi-organ failure should be coded in this category.
Organ Failure None
  • 1)

    Severe VOD/SOS, acute renal failure requiring dialysis, uncontrolled seizures, cardiac arrest, ARDS/IP or other organ toxicity with maximum acute GvHD grade 0-1 or limited mild chronic GVHD, no serious infections within 2 months of death, no disease progression or relapse post-HCT

  • 2)

    Organ failure after successful treatment of prior acute GvHD grade I-III or mild-moderate chronic GvHD, with resolution of GvHD symptoms and discontinuation of systemic immunosuppression before organ failure onset date
Other None
  • 1)

    Other cause of death in the absence of disease progression/relapse post-HCT, GvHD, infection or organ failure. Other causes includes primary or secondary graft failure, thrombotic or hemorrhagic events, second malignancies, accident, suicide, unknown
GvHD Infection
  • 1)

    Acute GvHD grade III-IV or extensive severe chronic GvHD on active treatment with a documented infection contributing to death; GvHD was severe enough to cause death even if infection did not occur; no evidence of disease progression or relapse post-HCT, no organ toxicity post-HCT
GvHD Infection, Organ Failure
  • 1)

    Acute GvHD grade III-IV or extensive severe chronic GvHD on active treatment with a documented infection and organ toxicity both contributing to death; GvHD was severe enough to cause death even if infection and organ toxicity did not occur; no evidence of disease progression or relapse post-HCT
GvHD Organ Failure
  • 1)

    Acute GvHD grade III-IV or extensive severe chronic GvHD on active treatment with a proximal serious organ failure in a non-GvHD organ (ie, heart, CNS, pancreas) which contributed to death, and no evidence of disease progression or relapse post-HCT, no infections
GvHD Other
  • 1)

    Acute GvHD grade III-IV or extensive severe chronic GvHD on active treatment with an “Other” contributing cause such as TTP/HUS, pulmonary emboli, hemorrhage in an organ not affected by GvHD, and with no evidence of disease progression or relapse post-HCT, no infections, no organ toxicity
Infection GvHD
  • 1)

    Documented infection severe enough to cause death (ie, fungal or viral pneumonia, bacteremia/sepsis, disseminated infection, toxoplasmosis, etc), in the setting of acute GvHD maximum grade II-III or mild to moderate chronic GvHD on active treatment; GvHD contributed to death, was actively being treated at time of death but was not severe enough to cause death without the infection
Infection Other
  • 1)

    Documented infection causing death with an “Other” contributing cause such as TTP/HUS, pulmonary emboli, hemorrhage in an organ not affected by GvHD, and with no evidence of disease progression or relapse post-HCT, no GvHD, no organ toxicity
Other Infection
  • 1)

    An “Other” cause, such as TTP/HUS or hemorrhage or pulmonary emboli, which was severe enough to be fatal, along with a documented infection(s) which contributed to death, and no evidence of disease progression or relapse post-HCT, no GvHD, or resolution of GvHD symptoms and discontinued systemic immunosuppression
Other Organ Failure
  • 1)

    An “Other” cause, such as TTP/HUS or hemorrhage or pulmonary emboli, which was severe enough to be fatal, along with severe organ failure which contributed to death, and no evidence of disease progression or relapse post-HCT, no GvHD, or resolution of GvHD symptoms and discontinued systemic immunosuppression
Open in a new tab

Legend: CR: Complete Remission, HCT: hematopoietic cell transplantation, GvHD: graft-versus-host disease, GI: gastrointestinal tract, VOD/SOS: veno-occlusive disease/sinosidal obstructive syndrome, ARDS/IP: Adult respiratory distress syndrome/interstitial pneumonitis, TTP/HUS: thrombotic thrombocytopenic purpura/hemolytic uremic syndrome, CNS: central nervous system

GvHD deaths included severe acute or chronic GvHD with active treatment at time of death. Infection deaths included bacterial, viral, fungal, and/or protozoan infections causing end organ damage. Organ Failure deaths were defined as transplant-related toxicity not due to disease progression, GvHD or infection. Organ Failure included, for example, veno-occlusive disease/sinusoidal obstructive syndrome (VOD/SOS), non-infectious interstitial pneumonitis, adult respiratory distress syndrome (ARDS), myocardial infarction, and renal failure in the absence of infection and GvHD. “Other” causes of death included rare events: vascular events including hemorrhage or thrombosis (e.g. pulmonary emboli, stroke), secondary malignancies, primary or secondary graft failure, accident, suicide or unknown.

The consensus panel could include an unlimited number of secondary or contributing causes of death. Based on the hierarchical nature of the definitions, secondary causes were included only for TRM and were coded in the same categories as the primary cause (GvHD, Infection, Organ Failure, Other). Secondary causes contributed to death but were not as severe as the primary cause or were closer to the time of death. Rare exceptions (affecting ≤3 cases per category) to the rules were allowed for unusual patient circumstances.

Internal and External Validity

Internal validity was tested using two approaches.9 First, 11 sequential cases from the Cohort 1 were blindly re-reviewed 2 months later. Second, 25 non-sequential cases were randomly selected by a non-panel member (XZ), and blindly re-reviewed by the consensus panel after all cases were adjudicated.

External validity was measured using a fourth in-person meeting at CIBMTR with 2 adult HCT physicians not involved in the study (JAH, PJM). Twenty-one previously adjudicated simple and complex cases from Cohort 1 were selected by a consensus panel member (TH), then adjudicated in the same manner as prior panel meetings.

Statistical Analysis

A first order Agreement Coefficient (AC1) was used to assess transplant center/consensus panel agreement. As with Cohen’s Kappa statistic (K), AC1 values range from 0-1 with a higher value indicating greater concordance.10-11 K was not used as the primary agreement measure due to the unblinded nature of the consensus panel review10,12-13 and the disproportionate prevalence of the competing causes of death,11,14-16 AC1 confidence intervals were taken from the 0.025 and 0.975 quantiles of the bootstrapped AC1 distribution (n=10 000) of the observed transplant center/consensus panel cause-specific death contingency table.17 A Bias Index was calculated to demonstrate the proportion of the total sample that moved into (positive value) or out of (negative value) a cause-specific death category,18-19 and the Prevalence Index was used to measure the proportion of all cases in each transplant center/consensus panel agreement cell.12,14

Several analyses were performed to measure the association with, and dependencies between, covariates and transplant center/consensus panel agreement on primary cause-specific death. Logistic regression models were used to test for patient characteristics associated with transplant center/consensus panel agreement on broad categories of death (DRM vs TRM) and specific TRM categories (GvHD, Organ Failure, Infection). Quasi-symmetric log-linear models were constructed to assess dependencies of each covariate with transplant center/consensus panel agreement on cause-specific death.20-22 Covariates tested include recipient gender and age, year of transplant, disease (AML, ALL, MDS) and disease status (early, intermediate, advanced). All statistical analyses were performed using R statistical software.23

RESULTS

Demographics

Patient, disease and treatment characteristics are shown in Table 2 for both cohorts. Cohort 2 was older and heavier than Cohort 1, had more patients with MDS and advanced disease, less with ALL, and more peripheral blood grafts. Other characteristics were well balanced between the cohorts, with the exception of the planned differences in year of HCT based on the cohort ascertainment.

Table 2.

Recipient Characteristics of the First and Second Cohorts

Recipient
Characteristics		 First Cohort
N=2609
n (%)		 Second Cohort
N=923
n (%)		 P
Age, years 0.002
≥40 1528 (59%) 594 (64%)
<40 1081 (41%) 329 (36%)
Male 1475 (57%) 504 (55%) NS
Race NS
White 2442 (94%) 867 (94%)
Non-White 114 (4%) 44 (5%)
Missing/unknown 53 (2%) 12 (1%)
Disease <0.001
AML 1535 (59%) 572 (62%)
ALL 653 (25%) 124 (13%)
MDS 421 (16%) 227 (25%)
AML Disease Status <0.001
Early 677 (44%) 310 (54%)
Intermediate 386 (25%) 121 (21%)
Advanced 472 (31%) 141 (25%)
ALL Disease Status NS
Early 248 (38%) 57 (50%)
Intermediate 278 (43%) 42 (30%)
Advanced 127 (19%) 25 (20%)
MDS Disease Status <0.001
Early 277 (75%) 83 (37%)
Intermediate -- --
Advanced 144 (25%) 144 (63%)
KPS/LPS NS
90-100 1591 (61%) 518 (56%)
<80 758 (29%) 279 (30%)
Missing 260 (10%) 126 (14%)
Year of HCT --
2000-02 450 (17%) 33 (4%)
2003-05 926 (35%) 90 (9%)
2006-08 1233 (47%) 121 (13%)
2009-11 0 679 (74%)
BMI** 0.011
Underweight 50 (2%) 25 (3%)
Normal 977 (37%) 293 (32%)
Overweight 811 (31%) 310 (34%)
Obese 764 (29%) 295 (32%)
Missing 7 (<1%) 0
Graft Source <0.001
Peripheral Blood 1648 (63%) 660 (72%)
Bone Marrow 961 (37%) 263 (28%)
Open in a new tab

Legend: NS: P>0.1, NA: Not applicable, AML: Acute Myeloid Leukemia, ALL: Acute Lymphoblastic Leukemia, MDS: Myelodysplastic Syndrome, Early: AML/ALL in CR1 or MDS RA/RARS, Intermediate: AML/ALL in CR2, Advanced: AML/ALL not in remission or MDS RAEB1/2, KPS: Karnofsky Performance Score, LPS: Lansky Performance Score, BMI: body mass index.

*

7 patients in cohort 1 and 52 patients in cohort 2 missing KPS/LPS were excluded;

**

BMI definitions for ≥18 years: underweight <18, normal 18 - <25, overweight 25- <30, obese ≥30 mg/kg2; BMI for ≥2 and <18 years: underweight <5th percentile, normal 5 - <85th percentile, overweight ≥85 - <95th percentile, obese ≥95th percentile; BMI for <2 years coded as normal. [Reference 27]

Data Queries and Resolutions

The consensus panel identified 99/1,484 (6.7%) cases as needing additional information from the transplant center in order to determine the causes of death. Transplant center responses were received for 76/99 (77%) of data queries, and cause-specific death was resolved by the consensus panel for all 76 cases. The remaining 23 cases with no response from the transplant center were re-reviewed by the consensus panel for final resolution of cause-specific death.

Internal and External Validity

All measures of internal and external validity showed excellent concordance for both primary and secondary cause-specific death. The 11 sequential cases demonstrated 100% agreement between first and second review, and the randomly selected 25 cases were 95% concordant on primary and secondary cause-specific death, including their order. External reviewers were in 100% agreement on all 21 previously adjudicated cases.

Primary Cause of Death – DRM vs. TRM

For Cohort 1, the primary cause-specific death categorized as DRM vs TRM demonstrated 87.5% agreement between the transplant center and consensus panel (AC1=0.75, 95% CI 0.73-0.80) and the Bias Index was 0.12. The consensus panel agreed with 463/465 cases (99.6% agreement) reported by the transplant center as DRM. The consensus panel agreed with 521/651 cases (80% agreement) reported by the transplant center as TRM. The consensus panel adjudicated the remaining 130 cases as DRM instead of TRM.

For Cohort 2, similar results demonstrated an overall agreement of 89% (AC1=0.78, 95% CI 0.71-0.84). The consensus panel agreed with the transplant center in 98.8% (162/164) for DRM and 81% (165/204) for TRM. The Bias Index was 0.10, indicating little difference in results of TRM/DRM adjudication between the two cohorts.

Primary Cause of Death – Disease vs. GvHD vs. Infection vs. Organ Failure vs. Other

Table 3 summarizes the transplant center/consensus panel agreement on DRM, GvHD, Infection, Organ Failure, and Other cause-specific death categories after adjudicating the primary causes of death for Cohorts 1 and 2, while Figure 1 graphically summarizes the same information for both cohorts combined. Supplementary Table 2 expresses the reclassification by the consensus panel in percentages and Supplementary Table 3 shows the statistical measures of agreement. The overall agreement for Cohort 1 considering all five cause-specific death categories was 73% (AC1=0.67, 95% CI 0.64-0.70). The consensus panel agreed with the transplant center 80 and 99% for GvHD and Disease deaths with the majority of discordance in cases reported by the transplant center as Organ Failure or Other (Table 3, Supplementary Table 2, and Figure 1). Of the 220 cases reported by the transplant center as Organ Failure, 115 (52%) were reclassified by the consensus panel as Disease (mainly, organ failure due to additional therapy for post-HCT relapse), GvHD (which led to multi-organ failure), Infection (which led to sepsis/multi-organ failure) or Other (Table 3, Figure 1). For the 113 cases reported by the transplant center as Other cause-specific death, the consensus panel was able to classify 75 (67%) of them into a more specific category (Table 3, Supplementary Tables 2 and 3, Figure 1).

Table 3.

Distribution of the Most Common Categories of Primary Cause of Death Within One Year after Transplant, Reported by the Transplant Center vs. Adjudicated by the Consensus Panel

Consensus Panel – Cohort 1 N (%)
Agreement
Primary Cause-
specific death		 Disease GvHD Infection Organ Failure Other
Transplant Center Disease 463 0 0 2 0 465 (99.6%)
GvHD 16 105 8 3 0 132 (79.5%)
Infection 44 24 101 12 5 186 (54.3%)
Organ Failure 45 31 27 105 12 220 (47.8%)
Other 25 13 16 21 38 113 (33.6%)
N (%) Agreement 593 (78.1%) 173 (60.7%) 152 (66.4%) 143 (73.4%) 55 (69.1%) 1116
Consensus Panel – Cohort 2 N (%) Agreement
Primary Cause-
specific death		 Disease GvHD Infection Organ Failure Other
Transplant Center Disease 162 0 1 1 0 164 (98.8%)
GvHD 3 38 0 0 1 42 (90.5%)
Infection 9 20 29 6 2 66 (43.9%)
Organ Failure 13 13 8 16 8 58 (27.6%)
Other 14 4 1 7 12 38 (31.6%)
N (%) Agreement 201 (80.6%) 75 (50.7%) 39 (74.4%) 30 (53.3%) 23 (52.2%) 368
Open in a new tab

Figure 1. Visual Summary of the Proportion of Causes of Death in Each Category as Reported by the Transplant Center that were Reclassified by the Consensus Panel, for Both Cohorts.

Figure 1

Open in a new tab

Black boxes indicate >15% of deaths in that TRM category were reclassified

Boxes with dots indicate ≥5 but <15% of deaths in that TRM category were reclassified

White boxes indicate <5% of deaths in that TRM category were reclassified

Grey boxes indicate the percent agreement of the Consensus Panel with the Transplant Center

Similarly, Cohort 2 had an overall agreement of 70% (AC1=0.64, 95% CI 0.58-0.69), with 68% of Other and 72% of Organ Failure deaths reported by the transplant center categorized by the consensus panel into a different category (Table 3), and again the consensus panel agreed with 91-99% of transplant center cause-specific deaths for GvHD and Disease. Additional agreement statistics are summarized in Supplementary Tables 2 and 3.

All Contributing Causes of Death

Table 4 summarizes the transplant center/consensus panel agreement for all cause-specific deaths for Cohorts 1 and 2. Agreement was highest for “Disease, with or without another cause”, followed by “GvHD and Infection, without or without another cause”, “GvHD without infection, with or without another cause”, “Infection without GvHD, with or without another cause” and “Organ Failure without GvHD without Infection, with or without another cause”. Between the 2 cohorts, the consensus panel adjudicated 189 deaths to GvHD and Infection (+/− another contributing causes), 181 deaths to Infection without GvHD (+/− another contributing causes), and 138 deaths to GvHD without Infection (+/− another contributing causes). These 3 clinical scenarios occurred with similar frequencies. Results between the two cohorts were consistent, despite adjudication schedule differences, inclusion of 8/8 HLA matching in Cohort 2, different years of HCT and different patient characteristics (Table 2).

Table 4.

Distribution of the Most Common Combinations of Causes of Death Reported by the Transplant Center vs. Adjudicated by the Consensus Panel

Consensus Panel – Cohort 1
 N (%) Agreement
All Causes of Death Disease +/−
Other		 GvHD +/−
Other
(No Infection)		 GvHD +
Infection
+/− Other		 Infection
+/− Other
(No GvHD)		 Organ Failure
+/− Other
(No GvHD,
No Infection)		 Other
(No GvHD,
Infection,
Organ Failure)
Transplant Center Disease +/−Other 482 0 0 1 2 1 486 (99%)
GvHD +/−Other
(No Infection)		 14 88 25 1
 6 1 135 (65%)
GvHD + Infection +/− Other 7 4 51 1 0 1 64 (80%)
Infection +/− Other
(No GvHD)		 36 2 40 102 10 2 192 (53%)
Organ Failure +/− Other
(No GvHD, No Infection)		 38 10 10 26 84 3 171 (49%)
Other
(No GvHD, Infection or
Organ Failure)		 16 7 4 9 11 21 68 (31%)
N (%) Agreement 593 (81%) 111 (79%) 130 (39%) 140 (73%) 113 (74%) 29 (72%) 1116
Consensus Panel – Cohort 2
 N (%)
Agreement
All Causes of Death Disease +/−
Other		 GvHD +/−
Other
(No Infection)		 GvHD +
Infection
+/− Other		 Infection
+/− Other
(No GvHD)		 Organ Failure
+/− Other
(No GvHD,
No Infection)		 Other
(No GvHD,
Infection,
Organ Failure)
Transplant Center Disease +/−Other 170 0 0 1 1 0 172 (99%)
GvHD +/−Other
(No Infection)		 3 20 14 0
 0 0 37 (54%)
GvHD + Infection +/− Other 2 1 20 1 0 0 24 (83%)
Infection +/− Other
(No GvHD)		 8 3 21 31 2 2 67 (46%)
Organ Failure +/− Other
(No GvHD, No Infection)		 11 2 2 7 14 7 43 (33%)
Other
(No GvHD, Infection or
Organ Failure)		 7 1 2 1 3 11 25 (44%)
N (%) Agreement 201 (85%) 27 (74%) 59 (34%) 41 (76%) 20 (70%) 20 (55%) 368
Open in a new tab

Clinical Scenarios

Supplementary Tables 4 and 5 detail examples of the adjudication results where the transplant center and consensus panel disagreed and agreed, respectively. These examples illustrate the common and rare scenarios that were adjudicated and provide more information about the decisions made and factors considered during the adjudication.

Factors Associated with Transplant Center and Consensus Panel Agreement

The multivariate models indicated that both the transplant centers and consensus panel coded the causes of death based on the patient’s disease (AML, ALL or MDS), hence the proportion of transplant center/consensus panel agreement did not vary by disease. However, the consensus panel relied on disease status (in remission or not) to adjudicate cause-specific deaths, whereas this was not consistently done by the transplant centers. This difference, in combination with the consistent application of cause-specific death definitions by the consensus panel but not transplant centers, accounted for most of the transplant center/consensus panel discordance. In addition, the transplant center cause-specific deathsdemonstrated a cohort effect: agreement between the transplant center and consensus panel was lower earlier in the cohort (HCT in 2000-2003) compared to later (HCT in 2009-2011).

Assessment of Transplant Center and Consensus Panel Agreement among Early Deaths

Table 5 summarizes the results of a sensitivity analysis of transplant center/consensus panel agreement for both cohorts on early deaths before day +100 post-URD-HCT since these were the most difficult to adjudicate. This analysis demonstrates how transplant center/consensus panel agreement is associated with disease status pre-URD-HCT. Agreement was highest for early disease status AML/ALL deaths before day +30 post-HCT (81%), followed by early or intermediate disease AML/ALL deaths before day +100 post-HCT (73-74%). Agreement was lowest among early disease risk MDS deaths before day +30 post-HCT (18%).

Table 5.

Sensitivity Analysis of Agreement between Reported and Adjudicated Cause-Specific Death < 100 Days after URD-HCT

Timing of Death Occurrence
After URD-HCT		 Disease Risk Rate of Agreement
between TC & CP		 Summary of Consensus Panel Changes
AML & ALL patients, Cohorts 1 & 2
Day 0-30, N=121 deaths (4%) of 2884 patients
Early 26/32 (81%) 6 cases were recoded as a different TRM category
Intermediate 13/24 (54%) 11 cases were recoded as a different TRM category
Advanced 32/65 (49%) 6 cases were recoded as a different TRM category
27 cases were recoded from TRM to DRM
Day 0-100, N=478 deaths (17%) of 2884 patients
Early 110/150 (73%) 38 cases were recoded as a different TRM category
3 cases were recoded from TRM to DRM
Intermediate 80/108 (74%) 25 cases were recoded as a different TRM category
3 cases were recoded from TRM to DRM
Advanced 141/220 (64%) 33 cases were recoded as a different TRM category
46 cases were recoded from TRM to DRM
MDS patients, Cohorts 1 & 2
Day 0-30, N=21 deaths (3%) of 648 patients
Early 2/11 (18%) 6 cases were recoded as a different TRM category
3 cases were recoded from TRM to DRM
Advanced 6/10 (60%) 1 case was recoded as a different TRM category
1 case were recoded from DRM to TRM
2 cases were recoded from TRM to DRM
Day 0-100, N=122 deaths (19%) of 648 patients
Early 39/63 (62%) 20 cases were recoded as a different TRM category
4 cases were recoded from TRM to DRM
Advanced 35/59 (59%) 13 cases were recoded as a different TRM category
9 cases were recoded from DRM to TRM
2 cases were recoded from TRM to DRM
Open in a new tab

Legend: AML: Acute Myeloid Leukemia, ALL: Acute lymphoblastic Leukemia, MDS: Myelodysplastic Syndrome, URD-HCT: unrelated donor allogeneic hematopoietic cell transplantation, TC: Transplant Center, CP: Consensus Panel

AML+ALL Disease Risk = Early: 1st complete remission, Intermediate: ≥2nd complete remission, Advanced: Not in complete remission at URD-HCT

MDS Disease Risk = Early: Refractory anemia with or without ringed sideroblasts or not otherwise specified, Advanced: all other MDS subtypes

Recommendations for Future Adjudication Panels

Based on our findings, the following groups should be prioritized for future studies needing adjudication of cause-specific death: all MDS patients who died before day +100, and all patients regardless of disease or disease status whose cause of death was reported as “Other” or “Organ Failure”. While the rate of agreement was higher for AML/ALL patients in remission pre-HCT, and for deaths reported as due to “GVHD” or “Infection”, there was still about a 20-27% discordance between reported and adjudicated death. This rate of discordance is enough to bias analyses of cause-specific mortality, and thus these groups are recommended for adjudication as well. Based on our report of 99% agreement between reported and adjudicated death, cause of death reported as recurrence or relapse of disease does not need to be adjudicated. This represented 42% of all deaths in our study population.

DISCUSSION

Application of our algorithm to future studies should focus efforts on adjudication of cases reported as TRM, particularly Infection, Organ Failure or Other, as well as AML and ALL patients who were not in remission at the time of HCT and all MDS patients. The current CIBMTR guidance provided to transplant centers for reporting causes of death uses the Centers for Disease Control and Prevention and World Health Organization definition for the underlying cause of death, defined as “the disease or injury that initiated the chain of events that led directly or inevitably to death”.24-26 However, many transplant centers report the last event in a cascade of multiple events. We defined the cause-specific death categories based on the CIBMTR guidance, our clinical experience, prior publications of similar studies and our intent to study genetic contributions to important, common causes of death. Other investigators studying the risk factors for these outcomes, or evaluating a multi-state modelling approach to define the risk of one phenotype (eg, infection) after developing another (eg, GvHD) could attribute causes of death with different definitions, prioritization or hierarchy.

Having precise categories of cause-specific death with pre-specified definitions allowed the consensus to categorize most of the Other cause-specific death cases into more specific categories (GvHD, Infection, Organ Failure, Disease). The ability to attribute more than one cause-specific death to each case by incorporating both timing and severity of each outcome, allowed better separation of cases into single vs multi-factorial causes of death. For patients with relapsed disease, the primary cause-specific death is defined as Disease, even if patients died with complications including infection, organ failure, GvHD. This practice is not uniformly applied in clinical research and varied over time in the CIBMTR cohort. This lack of uniformity between centers and over time could be due to changes in data manager education, the real-time reporting of individual cases by >150 transplant centers over a 12 year period, changes to the CIBMTR forms over time, availability of medical records (before/after EMRs) or a lack of standards for how to select and prioritize cause-specific deaths when multiple competing serious events occur.

Our consensus panel methodically adjudicated all deaths in the DISCOVeRY-BMT study with internal and external validity controls.9 There were no validation inconsistencies using short (2-day) or prolonged (8-month) review. Compact and prolonged schedules represent feasible designs for future studies, as long as quality assurance measures are used. This study was possible because the investigator team understood the critical importance of the review process and was dedicated to accuracy and consistency.

This approach to defining cause-specific death will enable us to perform a reproducible and accurate evaluation of genetic variants associated with TRM. We evaluated the potential impact of endpoint misclassification on our future GWAS study (DISCOVeRY-BMT) using the methods described by Wray, et al7 in the context of our adjudication results. Since 20% of TRM cases reported by the transplant center were reclassified as DRM, the estimate of total genetic variance explained by SNPs for TRM and DRM would decrease by 25-33% depending on the extent to which genetic variants contribute to TRM. Therefore, if we had not performed the adjudication, the total amount of true genetic variance contributing to these clinical endpoints after URD-HCT would have been difficult to assess and the results would have been subject to confounding by an increased false positive discovery rate.

Our findings are applicable to any GWAS investigating competing events, such as cancer-specific mortality vs treatment-related mortality, especially when multiple centers contribute data. Adjudication of the primary endpoint in observational research helps to eliminate potential cohort effects, reduces outcome heterogeneity and increases the likelihood of discovering true genetic associations. Our detailed and easy to use outcome algorithm (Table 1) provides the standards and definitions necessary to yield more consistent and accurate reporting of cause-specific death by participating CIBMTR centers.

Supplementary Material

Highlights.

  • Clinical scenarios/definitions of cause-specific death after URD-HCT are presented

  • High rate of agreement between reported and adjudicated disease-related deaths

  • Discordance of 20% between reported and adjudicated transplant-related deaths

  • Discordance varied among GVHD, Infection, Organ Failure and Other causes of death

ACKNOWLEDGEMENTS

John P. Klein, PhD, Department of Biostatistics, Medical College of Wisconsin, provided guidance on study design and statistical methodology. This study was supported by a grant from the National Heart, Lung, and Blood Institute (R01 HL102278) to Drs. Hahn and Sucheston-Campbell. The Bioinformatics Core at Roswell Park Cancer Institute is partially funded by a grant from the National Cancer Institute (P30 CA016056). The Center for International Blood and Marrow Transplant Research is supported by Public Health Service Grant/Cooperative Agreement U24-CA076518 from the National Cancer Institute (NCI), the National Heart, Lung and Blood Institute (NHLBI) and the National Institute of Allergy and Infectious Diseases (NIAID); a Grant/Cooperative Agreement 5U10HL069294 from NHLBI and NCI; a contract HHSH250201200016C with Health Resources and Services Administration (HRSA/DHHS); two Grants N00014-12-1-0142 and N00014-13-1-0039 from the Office of Naval Research; and grants from *Actinium Pharmaceuticals; Allos Therapeutics, Inc.; *Amgen, Inc.; Anonymous donation to the Medical College of Wisconsin; Ariad; Be the Match Foundation; *Blue Cross and Blue Shield Association; *Celgene Corporation; Chimerix, Inc.; Fred Hutchinson Cancer Research Center; Fresenius-Biotech North America, Inc.; *Gamida Cell Teva Joint Venture Ltd.; Genentech, Inc.;*Gentium SpA; Genzyme Corporation; GlaxoSmithKline; Health Research, Inc. Roswell Park Cancer Institute; HistoGenetics, Inc.; Incyte Corporation; Jeff Gordon Children’s Foundation; Kiadis Pharma; The Leukemia & Lymphoma Society; Medac GmbH; The Medical College of Wisconsin; Merck & Co, Inc.; Millennium: The Takeda Oncology Co.; *Milliman USA, Inc.; *Miltenyi Biotec, Inc.; National Marrow Donor Program; Onyx Pharmaceuticals; Optum Healthcare Solutions, Inc.; Osiris Therapeutics, Inc.; Otsuka America Pharmaceutical, Inc.; Perkin Elmer, Inc.; *Remedy Informatics; *Sanofi US; Seattle Genetics; Sigma-Tau Pharmaceuticals; Soligenix, Inc.; St. Baldrick’s Foundation; StemCyte, A Global Cord Blood Therapeutics Co.; Stemsoft Software, Inc.; Swedish Orphan Biovitrum; *Tarix Pharmaceuticals; *TerumoBCT; *Teva Neuroscience, Inc.; *THERAKOS, Inc.; University of Minnesota; University of Utah; and *Wellpoint, Inc. The views expressed in this article do not reflect the official policy or position of the National Institute of Health, the Department of the Navy, the Department of Defense, Health Resources and Services Administration (HRSA) or any other agency of the U.S. Government. *Corporate Members

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

This study was previously presented at an oral session during the annual Tandem BMT meeting in February 2014 and was published as an abstract in Biol Blood Marrow Transplant 20: S35, 2014 (abstract #20).

Conflict-of-Interest

The co-authors have no conflicts of interest to disclose.

REFERENCES

  • 1).Majhail NS, Chitphakdithai P, Logan B, et al. Significant improvement in survival after unrelated donor hematopoietic cell transplantation in the recent era. Biol Blood Marrow Transplant. 2015;21:142–150. doi: 10.1016/j.bbmt.2014.10.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2).Walovitch R, Yao B, Chokron P, Le H, Bubley G. Subjective endpoints in clinical trials: the case for blinded independent central review. J Clin Trials. 2013;5:111–17. [Google Scholar]
  • 3).Copelan E, Casper JT, Carter SL, et al. A scheme for defining cause of death and its application in the T cell depletion trial. Biol Blood Marrow Transplant. 2007;13:1469–76. doi: 10.1016/j.bbmt.2007.08.047. [DOI] [PubMed] [Google Scholar]
  • 4).Gratwohl A, Brand R, Frassoni F, et al. Cause of death after allogeneic haematopoietic stem cell transplantation (HSCT) in early leukaemias: an EBMT analysis of lethal infectious complications and changes over calendar time. Bone Marrow Transplant. 2005;36:757–769. doi: 10.1038/sj.bmt.1705140. [DOI] [PubMed] [Google Scholar]
  • 5).Wingard JR, Majhail NS, Brazauskas R, et al. Long-term survival and late deaths after allogeneic hematopoietic cell transplantation. J Clin Oncol. 2011;29:2230–39. doi: 10.1200/JCO.2010.33.7212. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6).Martin PJ, Counts GW, Appelbaum FR, et al. Life expectancy in patients surviving more than 5 years after hematopoietic cell transplantation. J Clin Oncol. 2010;28:1011–16. doi: 10.1200/JCO.2009.25.6693. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7).Wray NR, Lee SH, Kendler KS. Impact of diagnostic misclassification on estimation of genetic correlations using genome-wide genotypes. Eur J Hum Gen. 2012;20:668–674. doi: 10.1038/ejhg.2011.257. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8).Pasquini MC, Wang Z, Horowitz MM, Gale RP. 2010 report from the Center for International Blood and Marrow Transplant Research (CIBMTR): current uses and outcomes of hematopoietic cell transplants for blood and bone marrow disorders. Clin Transpl. 2010;210:87–105. [PubMed] [Google Scholar]
  • 9).Dechartres A, Boutron I, Roy C, Ravaud P. Inadequate planning and reporting of adjudication committees in clinical trials: recommendation proposal. J Clin Epidemiol. 2009;62:695–702. doi: 10.1016/j.jclinepi.2008.09.011. [DOI] [PubMed] [Google Scholar]
  • 10).Gwet KL. Inter-Rater Reliability: Dependency on Trait Prevalence and Marginal Homogeneity. Stat Methods IRR Assessment. 2002;2:1–9. [Google Scholar]
  • 11).Gwet KL. Computing inter-rater reliability and its variance in the presence of high agreement. Br J Math Stat Psychol. 2008;61:29–48. doi: 10.1348/000711006X126600. [DOI] [PubMed] [Google Scholar]
  • 12).Banerjee M, Capozzoli M, McSweeney L, Sinha D. Beyond Kappa: A Review of Interrater Agreement Measures. Canadian J Stat. 1999;27:3–23. [Google Scholar]
  • 13).Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther. 2005;85:257–68. [PubMed] [Google Scholar]
  • 14).Hoehler FK. Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity. J Clin Epidemiol. 2000;53:499–503. doi: 10.1016/s0895-4356(99)00174-2. [DOI] [PubMed] [Google Scholar]
  • 15).Wongpakaran N, Wongpakaran T, Wedding D, Gwet KL. A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Med Res Methodol. 2013;13:61. doi: 10.1186/1471-2288-13-61. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16).Ludbrook J. Statistical techniques for comparing measurers and methods of measurement: a critical review. Clin Exp Pharmacol Physiol. 2002;29:527–36. doi: 10.1046/j.1440-1681.2002.03686.x. [DOI] [PubMed] [Google Scholar]
  • 17).Efron B. Better bootstrap confidence intervals (with discussions) J Amer Stat Assoc. 1987;82:171–200. [Google Scholar]
  • 18).Hoehler FK. Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity. J Clin Epidemiol. 2000;53:499–503. doi: 10.1016/s0895-4356(99)00174-2. [DOI] [PubMed] [Google Scholar]
  • 19).Cicchetti DV, Feinstein AR. High agreement but low kappa, II: resolving the paradoxes. J Clin Epidemiol. 1990;43:551–558. doi: 10.1016/0895-4356(90)90159-m. [DOI] [PubMed] [Google Scholar]
  • 20).Agresti A. Series in probability and statistics. 3rd ed Wiley; Hoboken, NJ: 2013. Categorical data analysis; p. 714. [Google Scholar]
  • 21).Agresti A, Lang JB. Quasi-symmetric latent class models, with application to rater agreement. Biometrics. 1993;49:131–9. [PubMed] [Google Scholar]
  • 22).Gottard A, Marchetti GM, Agresti A. Quasi-Symmetric Graphical Log-Linear Models. Scand J Stat. 2011;38:447–465. [Google Scholar]
  • 23).R Core Development Team . R: A language and environment for statistical computing. R Foundation for Statistical Computing; Vienna, Austria: 2012. [Google Scholar]
  • 24). [Accessed 2/27/15];Instructions for Recipient Death Data (Form 2900) http://www.cibmtr.org/DataManagement/TrainingReference/Manuals/DataManagement/Documents/CIBMTR%20Recipient%20Death%20Data%20Form%202900.pdf.
  • 25).CDC [Accessed 2/27/15];Multiple Causes of Death 1999-2013. http://wonder.cdc.gov/wonder/help/mcd.html.
  • 26).World Health Organization [Accessed 2/27/15];Health Topics: Mortality. http://www.who.int/topics/mortality/en/
  • 27). [Accessed 7/11/14];Use and Interpretation of the WHO and CDC growth charts for children from birth to 20 years in the United States. http://www.cdc.gov/nccdphp/dnpa/growthcharts/resources/growthchart.pdf.

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

NIHMS695648-supplement.docx (27.1KB, docx)

RESOURCES