Visual Abstract
Keywords: molecular biology, rejection, kidney transplantation, multiplex polymerase chain reaction
Abstract
Background
The Banff Classification for Allograft Pathology recommendations for the diagnosis of kidney transplant rejection includes molecular assessment of the transplant biopsy. However, implementation of molecular tools in clinical practice is still limited, partly due to the required expertise and financial investment. The reverse transcriptase multiplex ligation-dependent probe amplification (RT-MLPA) assay is a simple, rapid, and inexpensive assay that permits simultaneous evaluation of a restricted gene panel using paraffin-embedded tissue blocks. The aim of this study was to develop and validate a RT-MLPA assay for diagnosis and classification of rejection.
Methods
A retrospective cohort of 220 kidney transplant biopsies from two centers, which included 52 antibody-mediated rejection, 51 T-cell–mediated rejection, and 117 no-rejection controls, was assessed. A 17-gene panel was identified on the basis of relevant pathophysiological pathways. A support vector machine classifier was developed. A subset of 109 biopsies was also assessed using the Nanostring Banff Human Organ Transplant panel to compare the two assays.
Results
The support vector machine classifier train and test accuracy scores were 0.84 and 0.83, respectively. In the test cohort, the F1 score for antibody-mediated rejection, T-cell–mediated rejection, and control were 0.88, 0.86, and 0.69, respectively. Using receiver-operating characteristic curves, the area under the curve for class predictions was 0.96, 0.89, and 0.91, respectively, with a weighted average at 0.94. Classifiers' performances were highest for antibody-mediated rejection diagnosis with 94% correct predictions, compared with 88% correct predictions for control biopsies and 60% for T-cell–mediated rejection biopsies. Gene expression levels assessed by RT-MLPA and Nanostring were correlated: r = 0.68, P < 0.001. Equivalent gene expression profiles were obtained with both assays in 81% of the samples.
Conclusions
The 17-gene panel RT-MLPA assay, developed here for formalin-fixed paraffin-embedded kidney transplant biopsies, classified kidney transplant rejection with an overall accurate prediction ratio of 0.83.
Podcast
This article contains a podcast at https://dts.podtrac.com/redirect.mp3/www.asn-online.org/media/podcast/CJASN/2023_04_10_CJN10100822.mp3
Introduction
Kidney transplantation rejection is a major cause of allograft dysfunction and graft loss.1 To diagnose rejection, histological assessment of a kidney transplant biopsy according to Banff guidelines remains the standard of care. However, it suffers from several limitations, such as high interobserver variability, sample bias in the case of focal disease, lack of sensitivity and/or specificity of some criteria (e.g., arteritis), and semiquantitative scoring.2
In addition to the histological assessment of kidney transplant biopsies, molecular phenotyping has emerged, providing a better understanding of the pathophysiological process responsible for rejection.3,4 Progressively, molecular tools, essentially microarray5 (Affymetrix) and mRNA quantification6,7 (Nanostring), have been refined to be used in clinical practice. The Banff consortium has identified genes of interest for molecular testing and recognized the validity of these molecular tools8–10 as an adjunct to histological diagnosis. A validated gene expression signature can now fulfill the second criterion of antibody-mediated rejection—“evidence of current/recent antibody interaction with vascular endothelium”9—or substitute the presence of donor-specific antibodies (DSAs) in the third criterion of antibody-mediated rejection.
Despite the clear interest of these molecular tools, their implementation in clinical practice is still limited, which can be explained by several factors.11 First, the proposed technologies require a financial investment, both because of equipment-related expenses and reagents required for sample processing: around $275 per sample for the 770-gene Nanostring Banff Human Organ Transplant (B-HOT) panel and around $2000 per sample for DNA microarrays.10 Second, these technologies are based on the simultaneous analysis of several hundred genes, which requires expertise for data processing and interpretation. Third, although lists of genes of interest have been identified, there are, to date, no guidelines for conducting and interpreting these molecular tests.
Considering these limitations, the aim of this study was to develop and validate a molecular assay that could be easily implemented in kidney transplant centers for diagnosis and classification of rejection. We used a reverse transcriptase multiplex ligation-dependent probe amplification (RT-MLPA) assay, which enables the simultaneous evaluation of a panel of approximately 20 genes. This simple, rapid, and inexpensive (reagent-related expenses estimated at 15–20€) assay requires a capillary sequencer for fragment-size analysis and can be performed with the same formalin-fixed paraffin-embedded tissue block used for light microscopy examination. We selected and validated a gene panel and proposed an open access online tool enabling automatized interpretation of results.
Methods
Study Population
Kidney transplant biopsies were randomly selected among the Rouen University Hospital pathology database and from biopsies from the Imperial College Healthcare National Health Service Trust database with previous RNA extraction.7,12,13 These databases included biopsies performed between 2010 and 2020. Biopsies were classified according to the Banff 2019 classification without considering the molecular biology criterion.9 For the diagnosis of active antibody-mediated rejection, these three criteria were required: (1) histologic evidence of acute tissue injury, defined by microvascular inflammation (g>0 and/or ptc>0) in the absence of glomerulonephritis or arteritis or acute thrombotic microangiopathy in the absence of any other cause; (2) linear C4d staining >1 by immunofluorescence or >0 by immunohistochemistry or microvascular inflammation defined by [g+ptc] ≥2; and (3) serologic evidence of DSA or C4d staining. For the diagnosis of chronic active antibody-mediated rejection, criterion 1 was replaced by transplant glomerulopathy (cg>0) with no evidence of chronic thrombotic microangiopathy or glomerulonephritis, with the same criteria as active antibody-mediated rejection for criteria 2 and 3. Acute T-cell–mediated rejection was defined by interstitial inflammation involving >25% of nonsclerotic cortical parenchyma (i>1) with tubulitis (t>1) or arteritis (v≥1). Other no-rejection biopsies were defined as controls, including protocol and indication biopsies with decreased graft function, to avoid limited challenge bias. The exclusion criteria were as follows: unknown DSA status, biopsies from ABO incompatible kidney transplant, biopsies classified as C4d without evidence of rejection, borderline for T-cell–mediated rejection, or mixed rejection. Considering the shared phenotype between BK virus nephropathy and T-cell–mediated rejection, biopsies with an intrarenal polyomavirus load level of >0% were also excluded.
Data Collection
Clinical and biological data were retrospectively collected. In both centers, DSA was monitored once a year and at the time of biopsy, if required, with Labscreen mixed beads as the screening test and Labscreen single antigen beads for identification (One Lambda, CA). The threshold for positivity was a mean fluorescence intensity more than 500.
RNA Extraction
In both centers, kidney transplant biopsies were collected with an 18-gauge spring-loaded needle. Paraffin blocks were stored at room temperature according to local practices. For each biopsy, four to six consecutive 20-μm sections were cut and stored at room temperature until RNA extraction. Manual extraction was performed using deparaffinization (#19093) and a RNeasy formalin-fixed paraffin-embedded (#73504) kit from Qiagen (Hilden, Germany), according to the supplier protocol. RNA concentration and quality were assessed with the NanoDrop 2000 Spectrophotometer (Thermo Fisher Scientific, Waltham, MA). Extracted RNA was stored at −80°C until the RT-MLPA assay.
For the French samples, 84 paraffin blocks were selected and successfully extracted. Mean RNA concentration was 41.4±37.1 ng/µl with a 260/280 ratio = 1.9±0.3. The molecular assessment failed in six of 84 samples (7%). One hundred fifty-six UK samples were selected, with a mean RNA concentration of 54.1±40.5 ng/µl and a 260/280 ratio = 2.0±0.1. The RT-MLPA assay succeeded in 142 (91%) of them.
RT-MLPA Assay
To develop the RT-MLPA assay, only the French biopsies were used. Probes against genes of interest were designed across exon-exon boundaries to avoid any unwanted amplifications of genomic DNA. All 5′ gene-probes, complementary to the first exon of interest, included at their 5′ end a sequence complementary to a universal primer U1 (GTGCCAGCAAGATCCAATCTAGA) followed by a gene-specific sequence. All 3′ gene-probes, complementary to the second exon of interest, included a phosphorylated 5′ end allowing ligation to the corresponding 5′ probe, then a gene-specific sequence, and a sequence complementary to a universal primer U2 (TCCAACCCTTAGGGAACCC) at their 3′ end. Between universal and gene-specific sequences, spacers (repeated TAC sequences) were inserted to allow separation and identification of PCR products according to their size. All probes were obtained from Eurogentec (Seraing, Belgium). Sequences of the probes used in the 17-gene panel are reported in Supplemental Table 1.
The RT-MLPA is a multiplex semiquantitative assay. The protocol for RT-MLPA, adapted from Mareschal et al.,13 is described in Supplemental Methods. To minimize RNA quantity bias, all expression values are divided by the mean of all genes measured in the sample, and the base 2 logarithms of these ratios plus 1 are used as normalized expression values.13,14 A Z-score normalization (mean 0 and variance 1) was also applied to all datasets to enable gene expression comparison. Free web interfaces were developed to facilitate data processing (http://92.222.23.215/RTMLPA/index.php?p=signin) and interpretation (https://kidney-transplant-classifier.herokuapp.com/). This software also provides graphical representation of gene expression from the raw fragment analyzer files (.fsa).
Nanostring Assay
In addition to RT-MLPA, RNA from 109 of the UK biopsies was also assessed using Nanostring nCounter (NanoString Technologies, Seattle, WA) with the Banff Human Organ Transplant panel (B-HOT panel),10 which contains 758 literature-derived genes of interest,6,12,13,16 12 internal reference genes, and 14 synthetic positive and negative controls. RNA was extracted according to the methods described above. The protocol for the Nanostring assay is described in Supplemental Methods.
Absolute expression levels of genes used in the RT-MLPA 17-gene panel were extracted from the Nanostring data. To enable Nanostring to RT-MLPA data comparison, absolute expression values were normalized using the same method described above.
Biopsy Classifier
To assign each biopsy into one of the three diagnosis categories, a support vector machine classifier was developed. The cohort was split between a training cohort (70% of the cohort) and a validation cohort (30%) with a stratification on the diagnosis category. Random oversampling was performed to compensate minority classes. To report classifier's performances, the following scores were used: the precision score, which is the ratio of correctly predicted samples (true positive) to all predicted samples (true and false positives); the recall score (i.e., sensitivity), which is the ratio of correctly predicted samples (true positive) to all samples in the class (true positive and false negative); and the specificity score, which is the ratio of true negative to true negative and false positive. F1 score, which is the harmonic mean of precision and recall scores, and accuracy, defined as the ratio of accurate predictions, were also used. Weighted average scores were calculated by taking into account the proportion of each diagnosis class.
Statistical Analyses
Quantitative data are presented with median [25th–75th percentiles] or mean ± SD according to their distribution. Qualitative data are presented with number of patients and percentages. The Student test, Mann–Whitney U test, and Kruskal–Wallis test were used for continuous variables. Fisher or chi-square tests were used for categorical variables, depending on sample sizes. Correlations were tested with the Spearman rank correlation test. A two-sided P value of <0.05 was considered statistically significant. Statistical analyses were conducted using GraphPad Prism version 6.04, GraphPad Software, La Jolla, CA, and Python programming language with SciPy, NumPy, and Pandas libraries.
Ethical Considerations
The clinical and research activities being reported are consistent with the principles of the Declaration of Istanbul as outlined in the Declaration of Istanbul on Organ Trafficking and Transplant Tourism. Biopsies collected during the routine care of patients were used for secondary research. Informed written consent for such secondary use was obtained from all patients before biopsy. The study was approved by the local institutional review board (notice No. 474).
Results
Cohort Characteristics
We analyzed the kidney biopsies from 220 kidney transplant recipients (Supplemental Figure 1), including 117 of 220 (53%) control biopsies, 52 of 220 (24%) antibody-mediated rejection biopsies, and 51 of 220 (23%) T-cell–mediated rejection biopsies. Control samples consisted of protocol and indication biopsies with a wide spectrum of diagnosis: 51 of 117 (44%) were considered as normal or with acute tubular necrosis, 38 of 117 (33%) biopsies showed tubulointerstitial fibrosis, 12 of 117 (10%) showed glomerulonephritis, 11 of 117 (9%) showed other features, and five of 117 (4%) showed microvascular inflammation (Supplemental Table 2). Clinical, biological, and pathological characteristics of the whole cohort and according to the diagnosis are reported in Table 1. Owing to the multicentric nature of the cohort, some differences in clinical, biological, or histological features were observed within each diagnosis group according to their geographic origin (Supplemental Table 3).
Table 1.
Characteristics of selected kidney transplant patients whose kidney biopsies were analyzed at Rouen University Hospital (Rouen, France) or Imperial College Healthcare National Health Service Trust (London, United Kingdom)
| Characteristics | All Patients N=220 |
AMR, N=52 |
TCMR, N=51 |
No Rejection, N=117 |
|---|---|---|---|---|
| Age at biopsy, yr | 52 [40–61] | 47 [42–57] | 54 [42–61] | 52 [40–62] |
| Female, n (%) | 61 (28) | 16 (31) | 31 (61) | 14 (12) |
| Cause of kidney failure, n (%) | ||||
| Diabetes | 43 (20) | 5 (10) | 12 (23) | 26 (22) |
| Other glomerulonephritis | 51 (23) | 15 (29) | 10 (20) | 26 (22) |
| Nephrosclerosis | 13 (6) | 8 (15) | 3 (6) | 2 (2) |
| Cystic kidney disease | 24 (11) | 5 (10) | 5 (10) | 14 (12) |
| Tubulointerstitial or uropathy | 25 (11) | 9 (17) | 6 (12) | 10 (9) |
| Other | 10 (5) | 0 | 1 (2) | 9 (8) |
| Unknown | 54 (25) | 10 (19) | 14 (27) | 30 (26) |
| Prior solid organ transplantation, n (%) | 34 (15) | 12 (23) | 10 (20) | 12 (10) |
| Living donor, n (%) | 53 (24) | 12 (23) | 10 (20) | 31 (27) |
| Mismatch HLA | ||||
| A | 1.2±0.7 | 1.2±0.7 | 1.2±0.7 | 1.2±0.7 |
| B | 1.3±0.7 | 1.4±0.7 | 1.5±0.6 | 1.2±0.7 |
| DR | 1.1±0.7 | 1.2±0.8 | 1.2±0.7 | 1.0±0.7 |
| DSA at transplant | ||||
| Class I | 23 (11) | 10 (19) | 5 (10) | 8 (7) |
| MFI of immunodominant anti-class I | 1820±3030 | 4900±6223 | 640±156 | 950±218 |
| Class II | 23 (11) | 13 (25) | 5 (10) | 5 (4) |
| MFI of immunodominant anti-class II | 2748±4402 | 6675±6333 | 620±84 | 1010±173 |
| Induction therapy, n (%) | ||||
| Antithymocyte globulin | 28 (13) | 15 (29) | 10 (20) | 3 (3) |
| Basiliximab | 57 (26) | 14 (27) | 15 (29) | 28 (24) |
| Alemtuzumab | 135 (61) | 23 (44) | 26 (51) | 86 (74) |
| Maintenance immunosuppressive therapy, n (%) | ||||
| Belatacept and antiproliferative | 9 (4) | 2 (4) | 6 (12) | 1 (1) |
| Belatacept alone | 1 (0.5) | 0 | 1 (2) | 0 |
| CNI and antiproliferative | 107 (49) | 36 (69) | 27 (53) | 44 (38) |
| CNI alone | 86 (39) | 13 (25) | 17 (33) | 56 (48) |
| CNI and mTORi | 13 (6) | 1 (2) | 0 | 12 (10) |
| mTORi and antiproliferative | 4 (2) | 0 | 0 | 4 (3) |
| Corticosteroids | 152 (69) | 36 (69) | 33 (65) | 83 (71) |
| Time from kidney transplantation to biopsy, mo | 13 [2–55] | 40 [7–113] | 9 [4–16] | 11 [1–61] |
| Indication biopsy | 212 (96) | 52 (100) | 50 (98) | 110 (94) |
| Donor-specific antibodies at biopsy | ||||
| Class I, n (%) | 20 (9) | 14 (27) | 2 (4) | 4 (3) |
| MFI of immunodominant anti-class I | 4474±4368 | 4234±4323 | 4000±4808 | 5550±5495 |
| Class II, n (%) | 49 (22) | 37 (71) | 4 (8) | 8 (7) |
| MFI of immunodominant anti-class II | 7159±5794 | 8352±5950 | 3668±4475 | 3388±2961 |
| Plasma creatinine at biopsy, mg/dl, µmol/L | 2.7±2.0 | 2.1±1.0 | 4.0±3.0 | 2.3±1.4 |
| Proteinuria at biopsy, g/g of creatininuria | 0.8±1.3 | 1.0±1.6 | 0.7±0.7 | 0.7±1.4 |
| Banff score | ||||
| cg | 0.4±0.9 | 1.2±1.3 | 0.1±0.2 | 0.1±0.5 |
| ci | 1.0±0.9 | 1.2±0.8 | 0.8±0.8 | 1.0±1.0 |
| ct | 1.0±0.9 | 1.1±0.8 | 1.0±0.8 | 1.0±0.9 |
| cv | 1.3±1.0 | 1.3±0.9 | 1.1±0.9 | 1.3±1.0 |
| ptc | 0.6±0.9 | 1.4±0.8 | 0.3±0.6 | 0.1±0.3 |
| t | 0.7±1.1 | 0.2±0.5 | 2.3±0.7 | 0.1±0.4 |
| i | 0.7±1.1 | 0.3±0.7 | 2.1±0.6 | 0.0±0.1 |
| g | 0.5±0.9 | 1.8±0.9 | 0.1±0.4 | 0.1±0.3 |
| ah | 0.9±1.1 | 1.2±1.3 | 0.4±0.6 | 1.0±1.2 |
| v | 0.1±0.4 | 0.1±0.4 | 0.3±0.7 | 0.0±0.0 |
| ti | 0.9±1.0 | 0.9±0.9 | 2.1±0.7 | 0.5±0.7 |
| iIFTA | 0.9±1.1 | 1.0±1.1 | 1.6±1.2 | 0.1±0.4 |
| g+ptc | 1.0±1.5 | 3.1±1.3 | 0.5±0.7 | 0.2±0.5 |
| i+t | 1.2±2.0 | 0.5±0.9 | 4.3±1.4 | 0.1±0.5 |
| ci+ct | 2.0±1.7 | 2.3±1.6 | 1.6±1.5 | 2.0±1.8 |
| cv+ah | 2.1±1.7 | 2.4±1.5 | 1.3±1.2 | 2.3±1.7 |
| C4d | 0.4±0.8 | 1.3±1.0 | 0.2±0.5 | 0.0±0.0 |
Results are expressed by mean ± SD or median [25th–75th percentiles]. P values refer to t tests performed between the three diagnostic categories. Among antibody-mediated rejection samples, mean [i+t] Banff score was 0.5 +/− 0.9. Indeed, 15/52 (29%) samples presented an isolated i or isolated t lesion, not reaching criteria for T-cell mediated or borderline for T-cell rejection. Among T-cell–mediated rejection samples, 17 of 51 (33%) presented a positive [g+ptc] score without reaching antibody-mediated rejection criteria. AMR, antibody-mediated rejection; TCMR, T-cell–mediated rejection; DSA, donor-specific antibody; MFI, mean fluorescence intensity; CNI, calcineurin inhibitors; mTORi, mammalian target of rapamycin inhibitors.
Gene Panel
The RT-MLPA assay allows the simultaneous evaluation of a panel up to 20 genes. On the basis of the knowledge of pathways involved in the pathophysiology of rejection and previously published data about molecular diagnosis in kidney transplantation using Affymetrix and Nanostring, 30 genes of interest were selected.3,4,7,8,17–19 Six successive gene panels were tested on the French samples to select the most discriminant genes to identify antibody-mediated rejection, T-cell–mediated rejection, or control biopsies. Finally, a 17-gene panel was identified, including genes significantly associated with at least one of these diagnoses (Table 2). This panel included several pathophysiological pathways: monocytes/macrophages-associated genes (ADAMDEC1, CCL18, CD68), endothelial-associated genes (CAV1, PECAM, PLA1A, ROBO4), T-cell–associated transcripts (CTLA4, IFNG, PRF1), NK-cell–associated transcripts (CCL4, FCGR3, GNLY, KLRD1), B-cell–related transcripts (CD72), and inflammation-related transcripts (CARD16, CXCL11).
Table 2.
Relative gene expression values assessed by the RT-MLPA assay according to diagnosis group in the French samples
| Gene | AMR (N=27) | TCMR (N=20) | Control (N=31) | P Value, Kruskal–Wallis t test | P Value, AMR versus TCMR | P Value, AMR versus Control | P Value, TCMR versus Control |
|---|---|---|---|---|---|---|---|
| CARD16 | 0.53±0.48 | 0.77±0.77 | 1.13±0.80 | 0.1 | 1.0 | 0.01a | 0.3 |
| CD72 | 0.74±0.32 | 0.85±0.40 | 0.72±0.32 | 0.2 | 0.4 | 1.0 | 0.4 |
| CD68 | 1.33±0.28 | 1.50±0.41 | 1.87±0.34 | <0.001a | 0.6 | <0.001a | 0.002a |
| CCL4 | 1.35±0.27 | 1.30±0.28 | 1.03±0.26 | <0.001a | 1.0 | <0.001a | 0.003a |
| CTLA4 | 0.20±0.16 | 0.42±0.34 | 0.15±0.16 | 0.01a | 0.3 | 0.7 | 0.01a |
| PLA1A | 1.50±0.24 | 0.95±0.22 | 1.38±0.34 | <0.001a | <0.001a | 0.5 | <0.001a |
| ROBO4 | 0.78±0.31 | 0.37±0.25 | 0.79±0.47 | <0.001a | <0.001a | 1.0 | 0.002a |
| KLRD1 | 0.95±0.28 | 0.99±0.27 | 0.79±0.38 | 0.03a | 1.0 | 0.1 | 0.07 |
| FCGR3 | 1.12±0.37 | 1.47±0.39 | 1.04±0.41 | 0.004a | 0.02a | 1.0 | 0.005a |
| GNLY | 0.97±0.43 | 0.74±0.25 | 0.64±0.38 | 0.009a | 0.3 | 0.007a | 0.7 |
| CXCL11 | 0.82±0.42 | 0.82±0.48 | 0.45±0.34 | 0.002a | 1.0 | 0.004a | 0.02a |
| CCL18 | 0.39±0.30 | 1.03±0.65 | 0.35±0.32 | <0.001a | <0.001a | 1.0 | <0.001a |
| CAV1 | 1.10±0.26 | 0.63±0.34 | 1.02±0.38 | <0.001a | <0.001a | 1.0 | <0.001a |
| PECAM | 1.87±0.24 | 1.27±0.35 | 1.94±0.41 | <0.001a | <0.001a | 1.0 | <0.001a |
| PRF1 | 1.16±0.28 | 1.25±0.35 | 0.87±0.32 | <0.001a | 1.0 | 0.006a | <0.001a |
| ADAMDEC1 | 0.13±0.15 | 0.58±0.49 | 0.09±0.13 | <0.001a | 0.003a | 1.0 | <0.001a |
| IFNG | 0.18±0.12 | 0.34±0.26 | 0.09±0.10 | <0.001a | 0.4 | 0.04a | <0.001a |
RT-MLPA, reverse transcriptase multiplex ligation-dependent probe amplification; AMR, antibody-mediated rejection; TCMR, T-cell–mediated rejection.
aValues P < 0.05.
Biopsy Classification
Unsupervised hierarchical analysis of gene expression levels is shown in Supplemental Figure 2. Normalized principal components analysis (PCA) representation in 3D showing variance scores of 28%, 14%, and 11% is represented in Figure 1.
Figure 1.
Principal component analysis shows variance scores of 28%, 14%, and 11% (PC1, PC2, and PC3, respectively). AMR, antibody-mediated rejection; TCMR, T-cell–mediated rejection.
Considering the effect of DSA for the diagnosis of antibody-mediated rejection and to provide a molecular tool that reflects daily clinical practice, support vector machine classifiers were based on gene expression values and DSA presence or absence on the closest serum to the biopsy. Support vector machine classifiers were randomly generated on the basis of a 70%/30% split of the entire cohort. Among these classifiers, one was randomly selected. Its train score accuracy was 0.84, and its test score accuracy was 0.83. To assess the influence of the initial 70%/30% split on the selected classifier, 500 random splits were performed to recalculate train/test scores. Scores of this selected classifier according to these random splits followed a Gaussian distribution (Supplemental Figure 3) with mean train and test scores at 0.88±0.02 and 0.77±0.05, respectively. To assess the generalization of this model, a 15-fold cross-validation was performed (Supplemental Figure 4): mean F1 score = 0.75±0.18, precision score = 0.78±0.17, and recall score = 0.76±0.20. In the test cohort, the F1 scores for antibody-mediated rejection, T-cell–mediated rejection, and control were 0.88, 0.69, and 0.86, respectively. Altogether, the weighted F1 score of the test cohort was 0.83 (Figure 2).
Figure 2.
Support vector classifier (SVC) report and confusion matrix in train (upper panel) and test (lower panel) cohorts. The cohort was randomly split into a training set (70% of the cohort) and validation set (30%), keeping identical proportions of each diagnosis category. For each diagnosis, precision score is the ratio of correctly predicted samples (true positive) to all predicted samples (true and false positives). Recall score (i.e., sensitivity) is the ratio of correctly predicted samples (true positive) to all samples in the class (true positive and false negative). Specificity score is the ratio of true negative to true negative and false positive. F1 score is the harmonic mean of precision and recall scores. Accuracy is the ratio of accurate predictions. Weighted average scores were calculated by taking into account the proportion of each diagnosis class. In the test cohort, performances for TCMR were lower; however, 89% of control biopsies and 94% of antibody-mediated rejection biopsies were correctly predicted. Altogether, the weighted F1 score of the test cohort was 0.83 and accuracy was 0.84. In the confusion matrices, rows are true diagnoses and columns are diagnoses predicted by the classifier.
Receiver-operating characteristic (ROC) curves were also constructed to determine the area under the curve (AUC) for each specific class versus the two others. AUC for antibody-mediated rejection, T-cell–mediated rejection, and control predictions were 0.96, 0.89, and 0.91, respectively, with a weighted average at 0.94 (Supplemental Figure 5).
In the test cohort, 60% of T-cell–mediated rejection biopsies were correctly predicted, 33% as control, and 7% as antibody-mediated rejection. Among control biopsies, 88% were correctly predicted as control, 6% as antibody-mediated rejection, and 6% as T-cell–mediated rejection. For antibody-mediated rejection class prediction, performances were higher: 94% of biopsies were correctly predicted and 6% were predicted as control. Support vector machine features’ importance for each diagnosis is depicted in Supplemental Figure 6. Twenty samples were used to test the influence of analytical variations between the two centers on diagnosis prediction: All samples were assigned to the same class in both centers (Supplemental Table 4).
Nanostring
Gene expression levels from the RT-MLPA assay were compared with those from the Nanostring B-HOT panel. Molecular profiles assessed by Nanostring were available for 109 UK samples. Comparisons of clinical and biological data of samples assessed or not by the Nanostring B-HOT panel are reported in Supplemental Table 5. All 17 genes of our panel assessed by Nanostring, but one (CCL4), were differentially expressed in a statistically significant manner between diagnostic groups (Table 3). The normalization process used for RT-MLPA data was also applied to the Nanostring data, which showed that relative gene expression levels were also statistically different between diagnosis groups (Supplemental Table 6).
Table 3.
Absolute gene expression values assessed by the Nanostring B-HOT panel according to diagnosis group in the UK samples
| Gene | AMR (N=24) | TCMR (N=6) | Control (N=79) | P Value, Kruskal–Wallis t test | P Value, AMR versus TCMR | P Value, AMR versus Control | P Value, TCMR versus Control |
|---|---|---|---|---|---|---|---|
| CARD16 | 79±40 | 67±21 | 43±21 | <0.001a | 0.99 | <0.001a | <0.001a |
| CD72 | 80±41 | 58±14 | 44±31 | <0.001a | 0.99 | <0.001a | 0.04a |
| CD68 | 578±264 | 462±84 | 414±223 | <0.001a | 0.99 | <0.001a | 0.2 |
| CCL4 | 51±51 | 38±21 | 29±22 | 0.1 | 0.99 | 0.2 | 0.7 |
| CTLA4 | 49±33 | 60±40 | 26±26 | <0.001a | 0.99 | <0.001a | 0.03a |
| PLA1A | 244±119 | 78±28 | 97±43 | <0.001a | <0.001a | <0.001a | 0.99 |
| ROBO4 | 341±107 | 207±41 | 273±69 | <0.001a | 0.001a | 0.02a | 0.06 |
| KLRD1 | 47±34 | 46±19 | 30±17 | 0.03a | 0.99 | 0.2 | 0.2 |
| FCGR3 | 571±381 | 314±167 | 181±202 | <0.001a | 0.6 | <0.001a | 0.06 |
| GNLY | 255±123 | 123±50 | 78±46 | <0.001a | 0.3 | <0.001a | 0.2 |
| CXCL11 | 434±478 | 70±39 | 75±251 | <0.001a | 0.04a | <0.001a | 0.6 |
| CCL18 | 55±32 | 109±63 | 42±38 | 0.001a | 0.3 | 0.06 | 0.006a |
| CAV1 | 228±112 | 165±158 | 177±83 | 0.04a | 0.08 | 0.2 | 0.6 |
| PECAM | 1464±424 | 1184±683 | 1059±237 | <0.001a | 0.04a | 0.001a | 0.99 |
| PRF1 | 119±43 | 83±22 | 46±29 | <0.001a | 0.99 | <0.001a | 0.01a |
| ADAMDEC1 | 22±15 | 33±11 | 15±12 | 0.001a | 0.1 | 0.1 | 0.003a |
| IFNG | 27±20 | 36±12 | 21±19 | 0.007a | 0.2 | 0.3 | 0.01a |
B-HOT, Banff Human Organ Transplant; AMR, antibody-mediated rejection; TCMR, T-cell–mediated rejection.
aValues P < 0.05.
Finally, gene expression levels assessed by these two assays were compared. Nanostring-normalized gene expression profiles were comparable with those obtained with the RT-MLPA assay (Figure 3). Normalized gene expression levels correlated with an r coefficient of 0.68 (P < 0.001, Figure 4). Comparisons for each gene are presented in Supplemental Table 7: A significant correlation was observed for 11 of the 17 genes. Importantly, in 81% of the 109 samples, a correlation in gene expression levels was observed with a P value of <0.05 (Supplemental Figure 7). Although no Nanostring-based classifier was built here, this suggests that both techniques can be used with comparable results.
Figure 3.

Gene expression profiles obtained with the RT-MLPA and Nanostring assays were comparable. Results are expressed for each gene as the mean (symbols) and SD (whiskers) of relative gene expression value in each diagnosis category after data normalization. Profiles obtained using these two assays were comparable for most of the genes, leading to consistent molecular profiles. RT-MLPA, reverse transcriptase multiplex ligation-dependent probe amplification.
Figure 4.

Normalized gene expression levels assessed by the RT-MLPA and Nanostring assays are correlated. Using the Spearman rank test, r value was 0.68 with P < 0.001.
Discussion
In this study, we analyzed gene expression in formalin-fixed paraffin-embedded kidney transplant biopsies using the RT-MLPA assay. We identified a 17-gene panel for the diagnosis and classification of rejection and validated a classifier. We also provided an open access tool for data processing and interpretation, allowing for replication of this assay in other kidney transplantation centers. Finally, we compared these results to data obtained using the Nanostring B-HOT panel.
The interest of the RT-MLPA assay has already been shown in several fields, such as hemato-oncology and inflammatory bowel diseases as well as molecular diagnosis of rejection in heart transplantation.20–23 It is a simple assay on the basis of fragment-size analysis, requiring a capillary sequencer, available in a large majority of kidney transplantation centers. As probes hybridize on a small length of DNA (20–40 base pairs), the RT-MLPA assay allows testing samples conserved in fixative solutions known to induce some level of RNA degradation. We were thus able to use formalin-fixed paraffin-embedded blocks in our study, without the need for an extra kidney transplant biopsy core.24 Moreover, interpretation of the results from our restricted gene panel did not require specific bioinformatic expertise, and reagent-related expenses were low, estimated at 15–20€ per sample. Finally, the whole procedure from RNA extraction to results reporting could be performed in <24 hours. Altogether, this assay seems appropriate for its implementation in routine practice, providing clinicians with an apprehensible molecular tool.
The gene panel was chosen according to pathophysiological data and previously published studies with PCR, Affymetrix, or Nanostring.3,4 Several successive gene panels were evaluated until the final 17-gene panel on the basis of its ability to discriminate antibody-mediated rejection, T-cell–mediated rejection, and no-rejection controls. Because these genes were also present in the Nanostring B-HOT panel,10 it was possible to compare the RT-MLPA semiquantitative results to quantitative Nanostring counts in a subcohort of patients. Although this subcohort showed slight differences in Banff scores as compared with the rest of the cohort, analyses on the basis of absolute quantification of gene expression confirmed that the chosen gene panel was discriminant using the two techniques. Molecular profiles obtained with RT-MLPA or after normalization of Nanostring data were also comparable, supporting the relative quantification approach with RT-MLPA. However, no Nanostring-based classifier was built here because Nanostring results were not available for the 220 samples of the entire cohort. This 17-gene panel could certainly be optimized with additional and/or more discriminant genes identified during ongoing large multicentric molecular studies.10 As data from RT-MLPA and Nanostring were comparable here, one could expect that the final recommended panel may be assessed by RT-MLPA with reliable results. Of note, the RT-MLPA assay can process one sample at a time, whereas the Nanostring assay is designed for 12-sample runs. On the whole, a kidney transplantation center could acquire one of these techniques depending on the available local facilities and number of biopsies and molecular tests performed per year.
On the basis of a restricted 17-gene panel, we were able to build a classifier with a weighted average F1 score at 0.83 and a misclassified biopsy rate at 17% in the test cohort. The highest performances were observed for the antibody-mediated rejection category. Indeed, in the test cohort, 94% of antibody-mediated rejection biopsies were correctly classified, and only 6% of control biopsies and 7% of T-cell–mediated rejection biopsies were misclassified into the antibody-mediated rejection category. Importantly, graft function was similar in the antibody-mediated rejection and control groups as control biopsies were predominantly for-cause biopsies with altered graft function and histology, which reflects the real-life situations and emphasizes the potential clinical applicability of this molecular tool. Consequently, the assay could be considered as a helpful tool in antibody-mediated rejection with an incomplete phenotype or to substitute the Banff criterion 2 or 3 for antibody-mediated rejection. By contrast, classifier performances for T-cell–mediated rejection were lower, with a high rate of T-cell–mediated rejection biopsies misclassified as no rejection, exposing to the risk of not treating a T-cell–mediated rejection. However, this should be tempered by the fact that the classifier does not consider clinical data or graft function. Indeed, T-cell–mediated rejection often occurs in a particular clinical setting, for example, decreased immunosuppression or treatment switch likely associated with AKI, which affects interpretation of pathological and molecular data. This underlines that the molecular tool should be considered as a help rather than a substitute to pathological tools.
Our study has some limitations. First, mixed rejections were excluded. Indeed, because this category is not individualized in the Banff classification and there is to date no clear molecular phenotype of mixed rejection,25–27 we selected only well-defined outcomes to build the classifier. BK virus nephropathy was also excluded because recent works showed that the overlap of immune-related genes between T-cell–mediated rejection and BK virus nephropathy hampers their concomitant evaluation.28 Moreover, although biopsies were selected in two different centers to develop and validate the classifier on the basis of a random-split strategy, our results could be strengthened by a higher number of samples and a real-life prospective evaluation of the model predictive performances.29 However, some level of heterogeneity in our cohort, linked to the two-center recruitment and likely reflecting the known variability in histopathological evaluation,2,24 did not impair the generalization of the selected model. At last, this study was not designed to analyze the effect of molecular diagnosis on patient management and graft survival. Indeed, further studies are needed to evaluate whether this assay could afford early diagnosis of rejection and risk stratification and assist clinicians in the treatment strategy, for example, in patients with de novo DSA.
We developed a simple molecular tool on the basis of a restricted gene panel that can be used on formalin-fixed paraffin-embedded kidney transplant blocks in a routine-compatible strategy. Moreover, we have provided an automated classifier, which enables rapid adoption of this tool by other transplantation centers. Finally, we compared the results obtained by the RT-MLPA assay with those obtained with the Nanostring assay and showed that a gene panel can be adapted from one of these techniques to the other.
Supplementary Material
Acknowledgments
The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. The authors are thankful to the French Society of Transplantation; the French Society of Nephrology, Dialysis and Transplantation; and the Charles Nicolle's Foundation for the financial support of the first author.
Footnotes
Published online ahead of print. Publication date available at www.cjasn.org.
Disclosures
S. Candon reports research funding from CSL Behring and honoraria from GILEAD. C. Roufosse reports consultancy agreements with Achillion, Rigel Pharmaceutical, and UCB. All remaining authors have nothing to disclose.
Funding
The study was supported by the Association pour le développement des recherches sur le rein artificiel en Normandie (Grant No. 2020-1234-001). This research project was also supported by the NIHR Imperial Biomedical Research Centre (BRC). Dr. Roufosse's research activity is made possible with generous support from Sidharth and Indira Burman. Some of the human samples used in this research project were obtained from the Imperial College Healthcare Tissue Bank (ICHTB). ICHTB is supported by the National Institute for Health Research (NIHR) Biomedical Research Centre based at Imperial College Healthcare NHS Trust and Imperial College London. ICHTB is approved by Wales REC3 to release human material for research (17/WA/0161).
Author Contributions
Conceptualization: Dominique Bertrand, Edvin Candon, Sophie Candon, Fanny Drieux, Tristan de Nattes
Data curation: Jack Beadle, Fanny Drieux, Arnaud François, Tristan de Nattes, Candice Roufosse, Frederic Toulza.
Formal analysis: Tristan de Nattes.
Funding acquisition: Dominique Bertrand, Sophie Candon, Dominique Guerrot, Tristan de Nattes, Candice Roufosse.
Investigation: Jack Beadle, Edvin Candon, Arnaud François, Tristan de Nattes, Candice Roufosse, Frederic Toulza.
Methodology: Sophie Candon, Fanny Drieux, Tristan de Nattes, Philippe Ruminy.
Project administration: Sophie Candon.
Software: Edvin Candon, Tristan de Nattes.
Supervision: Sophie Candon.
Visualization: Tristan de Nattes.
Writing – original draft: Tristan de Nattes.
Writing – review & editing: Jack Beadle, Dominique Bertrand, Edvin Candon, Sophie Candon, Fanny Drieux, Arnaud François, Dominique Guerrot, Tristan de Nattes, Candice Roufosse, Philippe Ruminy, Frederic Toulza.
Data Sharing Statement
Data supporting the findings of this study are available from the corresponding author on request.
Supplemental Material
This article contains the following supplemental material online at http://links.lww.com/CJN/B637.
Supplemental Methods. RT-MLPA and Nanostring assays.
Supplemental Table 1. Probes sequences used in the RT-MLPA 17-gene panel.
Supplemental Table 2. Spread of diagnoses in controls groups.
Supplemental Table 3. Clinical and biological characteristics of samples.
Supplemental Table 4. Inter-laboratory reproducibility.
Supplemental Table 5. Clinical and biological characteristics of samples assessed or not by the Nanostring B-HOT panel.
Supplemental Table 6. Relative gene expression values assessed by the Nanostring B-HOT panel.
Supplemental Table 7. Correlation analyses of relative gene expression levels obtained by the RT-MLPA or the Nanostring assay.
Supplemental Figure 1. Flow chart of samples selection.
Supplemental Figure 2. Heat map of unsupervised hierarchical analysis of gene expression levels assessed by the RT-MLPA assay.
Supplemental Figure 3. Influence of the initial random 70/30 split on train and test scores.
Supplemental Figure 4. Fifteen-fold cross-validation of the selected model.
Supplemental Figure 5. Performances of the classifier assessed by ROC analyses.
Supplemental Figure 6. Relative importance of support vector classifier features for each diagnosis category.
Supplemental Figure 7. Normalized gene expression levels assessed by RT-MLPA and Nanostring assays in matched samples.
References
- 1.Sellarés J de Freitas D Mengel M, et al. Understanding the causes of kidney transplant failure: the dominant role of antibody-mediated rejection and nonadherence. Am J Transplant. 2012;12(2):388–399. doi: 10.1111/j.1600-6143.2011.03840.x [DOI] [PubMed] [Google Scholar]
- 2.Furness PN, Taub N; Convergence of European Renal Transplant Pathology Assessment Procedures (CERTPAP) Project. International variation in the interpretation of renal transplant biopsies: report of the CERTPAP Project1. Kidney Int. 2001;60(5):1998–2012. doi: 10.1046/j.1523-1755.2001.00030.x [DOI] [PubMed] [Google Scholar]
- 3.Halloran PF, Famulski KS, Reeve J. Molecular assessment of disease states in kidney transplant biopsy samples. Nat Rev Nephrol. 2016;12(9):534–548. doi: 10.1038/nrneph.2016.85 [DOI] [PubMed] [Google Scholar]
- 4.Halloran PF Venner JM Madill-Thomsen KS, et al. Review: the transcripts associated with organ allograft rejection. Am J Transplant. 2018;18(4):785–795. doi: 10.1111/ajt.14600 [DOI] [PubMed] [Google Scholar]
- 5.Halloran PF Reeve J Akalin E, et al. Real time central assessment of kidney transplant indication biopsies by microarrays: the INTERCOMEX Study. Am J Transplant. 2017;17(11):2851–2862. doi: 10.1111/ajt.14329 [DOI] [PubMed] [Google Scholar]
- 6.Adam B Afzali B Dominy KM, et al. Multiplexed color-coded probe-based gene expression assessment for clinical molecular diagnostics in formalin-fixed paraffin-embedded human renal allograft tissue. Clin Transplant. 2016;30(3):295-305. doi: 10.1111/ctr.12689 [DOI] [PubMed] [Google Scholar]
- 7.Dominy KM Willicombe M Al Johani T, et al. Molecular assessment of C4d-positive renal transplant biopsies without evidence of rejection. Kidney Int Rep. 2019;4(1):148–158. doi: 10.1016/j.ekir.2018.09.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Haas M Loupy A Lefaucheur C, et al. The Banff 2017 Kidney Meeting Report: revised diagnostic criteria for chronic active T cell–mediated rejection, antibody‐mediated rejection, and prospects for integrative endpoints for next‐generation clinical trials. Am J Transplant. 2018;18(2):293–307. doi: 10.1111/ajt.14625 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Roufosse C Haas M Loupy A, et al. The Banff 2019 Kidney Meeting Report (I): updates on and clarification of criteria for T cell- and antibody-mediated rejection. Am J Transplant. 2021;21(5):1689–1690. doi: 10.1111/ajt.16419 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Mengel M Loupy A Haas M, et al. Banff 2019 Meeting Report: molecular diagnostics in solid organ transplantation - consensus for the Banff Human Organ Transplant (B-HOT) gene panel and open source multicenter validation. Am J Transplant. 2020;20(9):2305–2317. doi: 10.1111/ajt.16059 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Schinstock CA Sapir-Pichhadze R Naesens M, et al. Banff survey on antibody-mediated rejection clinical practices in kidney transplantation: diagnostic misinterpretation has potential therapeutic implications. Am J Transplant. 2019;19(1):123–131. doi: 10.1111/ajt.14979 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Toulza F Dominy K Cook T, et al. Technical considerations when designing a gene expression panel for renal transplant diagnosis. Sci Rep. 2020;10(1):17909. doi: 10.1038/s41598-020-74794-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Toulza F Dominy K Willicombe M, et al. Diagnostic application of transcripts associated with antibody-mediated rejection in kidney transplant biopsies. Nephrol Dial Transplant. 2022;37(8):1576–1584. doi: 10.1093/ndt/gfab231 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Mareschal S Ruminy P Bagacean C, et al. Accurate classification of germinal center B-Cell–Like/Activated B-Cell–Like diffuse large B-cell lymphoma using a simple and rapid reverse transcriptase–multiplex ligation-dependent probe amplification assay: a CALYM Study. J Mol Diagn. 2015;17(3):273–283. doi: 10.1016/j.jmoldx.2015.01.007 [DOI] [PubMed] [Google Scholar]
- 15.Eldering E Spek CA Aberson HL, et al. Expression profiling via novel multiplex assay allows rapid assessment of gene regulation in defined signalling pathways. Nucleic Acids Res. 2003;31(23):e153. doi: 10.1093/nar/gng153 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Dominy KM Roufosse C De Kort H, et al. Use of quantitative real time polymerase chain reaction to assess gene transcripts associated with antibody-mediated rejection of kidney transplants. Transplantation. 2015;99(9):1981–1988. doi: 10.1097/TP.0000000000000621. [DOI] [PubMed] [Google Scholar]
- 17.Reeve J Sellarés J Mengel M, et al. Molecular diagnosis of T cell-mediated rejection in human kidney transplant biopsies. Am J Transplant. 2013;13(3):645–655. doi: 10.1111/ajt.12079 [DOI] [PubMed] [Google Scholar]
- 18.Halloran PF, Venner JM, Famulski KS. Comprehensive analysis of transcript changes associated with allograft rejection: combining universal and selective features. Am J Transplant. 2017;17(7):1754–1769. doi: 10.1111/ajt.14200 [DOI] [PubMed] [Google Scholar]
- 19.Aubert O Loupy A Hidalgo L, et al. Antibody-mediated rejection due to preexisting versus de novo donor-specific antibodies in kidney allograft recipients. J Am Soc Nephrol. 1912-19232017;28(6):1912–1923. doi: 10.1681/ASN.2016070797 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Gologan S Iacob R Iancu D, et al. Inflammatory gene expression profiles in Crohn’s disease and ulcerative colitis: a comparative analysis using a reverse transcriptase multiplex ligation-dependent probe amplification protocol. J Crohns Colitis 2013;7(8):622–630. doi: 10.1016/j.crohns.2012.08.015 [DOI] [PubMed] [Google Scholar]
- 21.Bobée V Ruminy P Marchand V, et al. Determination of molecular subtypes of diffuse large B-cell lymphoma using a reverse transcriptase multiplex ligation-dependent probe amplification classifier: a CALYM Study. J Mol Diagn. 2017;19(6):892–904. doi: 10.1016/j.jmoldx.2017.07.007 [DOI] [PubMed] [Google Scholar]
- 22.Adam N Coutance G Drieux F, et al. RT-MLPA as a new tool for the molecular diagnosis of allograft rejection in formalin paraffin-embedded endomyocardial biopsies. J Heart Lung Transplant. 2018;37(4):S79. doi: 10.1016/j.healun.2018.01.179 [DOI] [Google Scholar]
- 23.Drieux F Ruminy P Abdel-Sater A, et al. Defining the signatures of peripheral T-cell lymphoma, with a targeted 20-markers gene expression profiling assay (RT-MLPA). Haematologica. 2020;105(6):1582–1592. doi: 10.3324/haematol.2019.226647 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.von Ahlfen S, Missel A, Bendrat K, Schlumpberger M. Determinants of RNA quality from FFPE samples. PLoS One. 2007;2(12):e1261. doi: 10.1371/journal.pone.0001261 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Madill-Thomsen K Perkowska-Ptasińska A Böhmig GA, et al. Discrepancy analysis comparing molecular and histology diagnoses in kidney transplant biopsies. Am J Transplant. 2020;20(5):1341–1350. doi: 10.1111/ajt.15752 [DOI] [PubMed] [Google Scholar]
- 26.Vaulet T Divard G Thaunat O, et al. Data-driven derivation and validation of novel phenotypes for acute kidney transplant rejection using semi-supervised clustering. J Am Soc Nephrol. 2021;32(5):1084–1096. doi: 10.1681/ASN.2020101418 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Callemeyn J, Lamarthée B, Koenig A, Koshy P, Thaunat O, Naesens M. Allorecognition and the spectrum of kidney transplant rejection. Kidney Int. 2022;101(4):692–710. doi: 10.1016/j.kint.2021.11.029 [DOI] [PubMed] [Google Scholar]
- 28.Adam BA Kikic Z Wagner S, et al. Intragraft gene expression in native kidney BK virus nephropathy versus T cell-mediated rejection: prospects for molecular diagnosis and risk prediction. Am J Transplant. 2020;20(12):3486–3501. doi: 10.1111/ajt.15980 [DOI] [PubMed] [Google Scholar]
- 29.Collins GS Dhiman P Andaur Navarro CL, et al. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open. 2021;11(7):e048008. doi: 10.1136/bmjopen-2020-048008 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
Data supporting the findings of this study are available from the corresponding author on request.



