Error Reduction in Leukemia Machine Learning Classification With Conformal Prediction

Abstract

PURPOSE

Recent advances in machine learning have led to the development of classifiers that predict molecular subtypes of acute lymphoblastic leukemia (ALL) using RNA-sequencing (RNA-seq) data. Although these models have shown promising results, they often lack robust performance guarantees. The aim of this study was three-fold: to quantify the uncertainty of these classifiers, to provide prediction sets that control the false-negative rate (FNR), and to perform implicit error reduction by transforming incorrect predictions into uncertain predictions.

METHODS

Conformal prediction (CP) is a distribution-agnostic framework for generating statistically calibrated prediction sets whose size reflects model uncertainty. In this study, we applied an extension called conformal risk control to three RNA-seq ALL subtype classifiers. Leveraging RNA-seq data from 1,227 patient samples taken at diagnosis, we developed a multiclass conformal predictor ALLCoP, which generates statistically guaranteed FNR-controlled prediction sets.

RESULTS

ALLCoP was able to create prediction sets with specified FNR tolerances ranging from 7.5% to 30%. In a validation cohort, ALLCoP successfully reduced the FNR of the ALLIUM RNA-seq ALL subtype classifier from 8.95% to 3.5%. For patients whose subtype was not previously known, the use of ALLCoP was able to reduce the occurrence of empty predictions from 37% to 17%. Notably, up to 34% of the multiple-class prediction sets included the PAX5alt subtype, suggesting that increased prediction set size may reflect secondary aberrations and biological complexity, contributing to classifier uncertainty. Finally, ALLCoP was validated on two additional RNA-seq ALL subtype classifiers, ALLSorts and ALLCatchR.

CONCLUSION

Our results highlight the potential of CP in enhancing the use of oncologic RNA-seq subtyping classifiers and also in uncovering additional molecular aberrations of potential clinical importance.

INTRODUCTION

In the past decade, there has been an explosion in the number of diagnostic machine learning (ML) models developed for precision oncology, with the promise of delivering increasingly accurate diagnostics and personalized treatment.^1,2 acute lymphoblastic leukemia (ALL), the most common cancer in children and a highly heterogeneous disease, has seen the development of numerous classifiers linking transcriptomic footprints to subtype-defining chromosomal aberrations.^3-7 ALL subtypes hold prognostic value, aid in monitoring disease progression, and help determine treatment intensity.^8,9 Before the introduction of next generation sequencing (NGS)–based diagnostics, patients with ALL were subtyped at diagnosis using techniques such as G-banding, fluorescence in situ hybridization, and reverse transcription polymerase chain reaction. At present, whole-genome sequencing and whole-transcriptome sequencing (WTS) are emerging as alternative methods to determine the subtype without a priori knowledge of the underlying genomic aberrations.¹⁰ However, some patients remain unclassified even with the use of WTS analysis pipelines,¹¹ making ML subtyping classifiers a useful alternative. Yet, despite increasing efforts to implement data for clinical decision making in hematology,¹² numerous challenges remain to the deployment of diagnostic ML models, ranging from insufficient regulatory frameworks for artificial intelligence (AI) to the poor clinical applicability of such models.^13,14

CONTEXT

• Key Objective

• Can conformal prediction be used to reduce the error of models that use RNA-sequencing data to predict the molecular subtypes of acute lymphoblastic leukemia (ALL)?

• Knowledge Generated

• The conformal predictor ALLCoP reduced the false-negative rate of the ALLIUM classifier from 8.95% to 3.5% in a validation cohort. For patients with ALL whose subtype was not previously known, the use of ALLCoP was able to reduce the occurrence of empty predictions from 37% to 17%.

• Relevance (F.P.-Y. Lin)

• Conformal prediction offers an alternative framework for representing machine learning outputs instead of point prediction scores. This approach has the potential to improve transparency and reduce missed diagnoses, as comprehensively demonstrated by this study, with applications extending beyond leukemia diagnostic classification.*

• *Relevance section written by JCO Clinical Cancer Informatics Deputy Editor Frank Po-Yen Lin, PhD, FRACP, MBChB, FAIDH.

Among these challenges is the quantification of reliability and uncertainty in ML classification models. Out of the box, classifiers typically output a naïve point prediction, a top-scoring class or k top-scoring classes with no metric of uncertainty, while the underlying probabilistic scores for each class are not calibrated to empirical probability and are therefore not interpretable as confidence scores. Furthermore, traditional ML models are evaluated with population-based metrics, which do not give useful indications of uncertainty for individual patients and leave no explanation for classifier errors when they occur.¹⁵

Conformal prediction (CP), a distribution-agnostic and model-agnostic framework, addresses this problem by replacing point predictions with prediction sets containing all true classes at a user-specified probability, using an independent calibration data set to determine softmax thresholds for inclusion of classes in these sets.^16-19 In addition to providing a mathematically proven, statistically guaranteed prediction set, CP aids in the human interpretability of ML outputs: a CP set containing a single class shows that the classifier is highly certain about the prediction, while a larger set shows a higher degree of uncertainty, and an empty set indicates that the model does not recognize the input, suggesting an out-of-distribution instance. Cresswell et al²⁰ showed that the use of CP sets, with size inherently quantifying uncertainty, can improve human decision making compared with models simply outputting the k top-scoring classes. Finally, CP can be used for implicit error reduction: It is possible to select a lower error rate for the classifier (increased accuracy) at the expense of larger, less certain prediction sets (decreased precision).

To date, medical applications of CP remain few.²¹ A handful of studies have shown CP applied to drug discovery,²² disease course prediction in multiple sclerosis,²³ lung tissue microscopy,²⁴ as well as oncology, including prostate²⁵ and breast cancer²⁶ classification. A 2006 study applied CP to a support vector machine trained to predict five ALL and three AML subtypes using microarray-generated gene expression (GEX) data up to a 95% confidence level.²⁷ However, to our knowledge, to date, there have been no applications of CP to RNA-sequencing (RNA-seq) classifiers for leukemia or any other cancer.

In this study, we applied CP to three ALL subtype classifiers using RNA-seq data from 1,227 patients from five different ALL cohorts^3,28-30 encompassing 14 ALL subtypes. A conformal predictor, ALLCoP, was first calibrated and cross-validated using predictions from the ALLIUM classifier.³ In a validation data set, ALLCoP was able to reduce the false-negative rate (FNR) from 8.95% to 3.5%. ALLCoP was then used to create prediction sets for 126 samples whose subtype was unknown at diagnosis. Finally, ALLCoP was validated on two additional classifiers, ALLSorts⁴ and ALLCatchR.⁵

METHODS

Data

Publicly available GEX counts were obtained from five different studies^3,28-30 comprising RNA-seq data from a total of 1,227 diagnostic pediatric and young adult ALL samples. After preprocessing, 1,042 samples were subjected to ALL subtype prediction using ALLIUM,³ while 292 samples were subjected to the ALLCatchR⁵ and ALLSorts⁴ models. Of the ALLIUM predictions, samples whose subtype was denoted as unknown in the original studies (n = 126) and samples with multiple known subtypes (n = 65) were analyzed separately, leaving 851 samples for ALLCoP calibration and validation. The model and data preprocessing are specified in the Data Supplement, together with cohort details (Table S1); batch effects and correction are illustrated in the Data Supplement (Figs S1 and S2).

Conformal Prediction

The conformal predictor ALLCoP used the classifier outputs as inputs, both for calibration and the subsequent formation of prediction sets (Data Supplement, Fig S3). We used split CP, which requires calibration and validation using a data set independent from the training data set of the underlying model,¹⁹ and conformal risk control, which allows the application of CP to data sets with multiple true classes.³¹ The latter produces prediction sets at a user-defined tolerance on the FNR, denoted α, and an FNR-controlling softmax threshold value, lamhat, above which classes are included in the prediction sets (Data Supplement, Fig S4).

We first validated ALLCoP. Using the ALLIUM predictions for all samples with a single known subtype (n = 851), a series of cross-validation experiments were configured whereby each experiment consisted of multiple runs and in each run, the prediction data set was shuffled and split: 90% of predictions were used for calibration of the conformal predictor using the conformal risk control³¹ algorithm and 10% were used for validation. ALLCoP was then recalibrated in different configurations to generate prediction sets for subsets of the data (Data Supplement).

RESULTS

Uncertainty in ALL Subtype Classification

Our aim was to investigate whether applying CP could quantify the uncertainty and reduce the error rate of RNA-seq ALL subtype classifiers. We applied ALLIUM³ to 1,042 orthogonal RNA-seq samples from patients with ALL from five studies^3,28-30 (Data Supplement, Table S2). The distribution of their known molecular subtypes is shown in Figure 1A.

FIG 1.

RNA-sequencing data sets and ALLIUM predictions used as input for ALLCoP. (A) The subtype distributions for the 1,042 samples used for the ALLIUM implementation of ALLCoP. The remaining panels visualize ALLIUM predictions for the 851 samples with a single known subtype, including (B) softmax scores output by the model, stratified by true subtype and colored by predicted subtype; (C) the class membership of ALLIUM prediction sets, formed using a softmax threshold of 0.5; and (D) a heatmap mapping each true subtype to the mean softmax score per predicted subtype, with the green column showing the mean softmax scores of false positives and the purple column showing the mean size of the ALLIUM prediction sets with softmax threshold = 0.5.

For the 851 samples with a single known subtype, the softmax scores of the ALLIUM predictions were unequally distributed by true subtype, with, for example, T-ALL and ZNF384-r showing consistently high scores for the correct subtypes only and with other subtypes such as BCR::ABL1 and BCR::ABL1-like receiving high scores in more than 1 class (Fig 1B).

Using a threshold of 0.5, we formed prediction sets comprised of all subtypes whose ALLIUM softmax score surpassed this value. Of the samples, 691 (81.2%) resulted in single-class prediction sets, most frequently high hyperdiploid (n = 181) or ETV6::RUNX1 (n = 117). However, some subtypes were consistently found to co-occur with others, such as PAX5 P80R; 78.6% of prediction sets containing this class also contained the subtype PAX5alt (Fig 1C).

ALLIUM has difficulty distinguishing between these two subtypes, although the true subtype typically scored higher, with true PAX5 P80R receiving a mean softmax score of 0.96 versus 0.85 for PAX5alt. Again, using a softmax score cutoff of 0.5, the mean prediction set size, meaning the number of classes predicted per individual sample, varied per subtype, with the highest mean set sizes observed for PAX5 P80R, BCR::ABL1, BCR::ABL1-like, and MEF2D-r (Fig 1D).

Empirical Error Rate Selection and FNR Validation

Next, we developed and applied a conformal predictor (ALLCoP) to the ALLIUM predictions (softmax scores), to (1) evaluate error reduction to predefined levels and (2) enable reporting of multiple potentially true subtype calls.³²

Using ALLCoP, we created prediction sets using a range of FNR tolerance α levels (0.05-0.50) and corresponding FNR-controlling softmax threshold values, lamhat, above which classes are included in the output prediction sets (Fig 2A; Data Supplement, Table S3).

FIG 2.

Empirical ALLCoP error rate selection and FNR validation for the ALLIUM classifier. (A) The output of the constraint function that selects the FNR-controlling model softmax threshold lamhat for a range of error rate values α. (B) From an experiment of 1000 ALLCoP runs for a range of α values, shown are the proportions of resulting prediction sets that were empty, certain (size = 1), or uncertain (size ≥2). The gray dashed line and bolded number indicate the α value at which the highest proportion of certain prediction sets occurs. (C) The mean FNRs of prediction sets produced in an experiment of 1000 ALLCoP runs with error rate α = .15, empirically testing the overall FNR. (D) The FNRs of the ALLCoP prediction sets produced at a range of α values, versus the FNR of the uncalibrated ALLIUM classifier outputs, defined as a set containing the single top-scoring subtype, in yellow. FNR, false-negative rate.

At α = .15, ALLCoP produced the highest proportion of high certainty, single-class prediction sets (81.6%), and relatively few sets representing no prediction (5.7% empty sets) or uncertain prediction (12.7% of sets with size ≥2; Fig 2B). The mean FNR of the prediction sets was 14.82%, in line with the selected value of α = .15 (Fig 2C).

In experiments of 1,000 runs per α value, the FNR of ALLCoP prediction sets with $\alpha \in \left\{0.1,0.2,0.3\right\}$ were then compared against the FNR of the uncalibrated ALLIUM classifier outputs. In line with the conformal statistical guarantee, the FNRs of the ALLCoP prediction sets were 9.6%, 19.8%, and 30.0%, while the mean FNR for ALLIUM was 15.3% (Fig 2D). FNRs for ALLCoP at all three error rates were at or below the corresponding α values, indicating that the ALLCoP statistical guarantee holds across a diverse range of conditions, irrespective of the underlying model's performance.

Performance by ALL Subtype

Next, we evaluated ALLCoP performance by true ALL subtype. In order to obtain generalizations across the entire dataset, we cross-validated with 10 thousand runs per α value with $\alpha \in \left\{0.075,0.1,0.15\right\}$ using predictions from the 851 single-subtype samples. We evaluated the FNR and set sizes of the resulting prediction sets, stratified by subtype.

At α = .075, the FNRs for the different subtypes ranged from 0% to 28% (Fig 3A), while the set sizes ranged from 1.0 to 2.50 (Fig 3B). At α = .1, the FNRs ranged from 0% to 38% (Fig 3C) and set sizes from 0.98 to 2.0 (Fig 3D). At α = .15, the FNRs ranged from 0% to 55% (Fig 3E) and set sizes from 0.93 to 1.90 (Fig 3F).

FIG 3.

ALLCoP prediction sets generated from ALLIUM predictions, by subtype. (A) Mean FNRs for prediction sets, α = .075. (B) Mean sizes for prediction sets, α = .075. (C) Mean FNRs for prediction sets, α = .1. (D) Mean sizes for prediction sets, α = .1. (E) Mean FNRs for prediction sets, α = .15. (F) Mean sizes for prediction sets, α = .15. FNR, false-negative rate.

For subtypes ZNF384-r, TCF3::PBX1, and MEF2D-r, the FNR remained 0% across all three α values, indicating high classifier certainty, with the correct class always included in the respective prediction sets. BCR::ABL1-like had the highest FNR across all α values, from 28% at α = .075 to 55% at α = .15, and the highest set size at α = .075 (2.50), indicating that high-uncertainty classes are increasingly included in prediction sets at lower α values, at the cost of larger prediction sets.

Implicit Error Reduction in a Validation Data Set

After cross-validating the conformal guarantee in ALLCoP and assessing its performance across ALL subtypes, we then recalibrated instances of ALLCoP to produce prediction sets for three data subsets: first, for samples with a single known subtype; second, for samples with multiple known subtypes; and finally, for samples of unknown subtype. We aimed to show that CP can be used to reduce error in validation data with a known subtype and to produce fewer empty predictions for samples with an unknown subtype.

Using ALLIUM predictions from the St Jude Cloud samples with a single known subtype (n = 594),²⁸ we recalibrated ALLCoP with $\alpha \in \left\{0.075,0.1,0.15\right\}$ and obtained prediction sets for the samples with a single known subtype from the other cohorts (n = 257)^3,29,30 (Data Supplement, Table S4).

In this validation set, 234 ALLIUM predictions were correct, 19 were wrong, and four were empty. Of these 23 samples with incorrect or empty predictions, ALLCoP prediction sets contained the correct class for six patients at α = .15, 13 patients at α = .1, and 14 patients at α = .075. In this validation set, the uncalibrated ALLIUM model had an FNR of 8.95%, which was reduced to 6.61% at α = .15, 3.89% at α = .10, and 3.50% at α = .075; the trade-off was an increasing mean set size: 1.11 at α = .15, 1.31 at α = .10, and 1.53 at α = .075 (Data Supplement, Table S5). A comparison between ALLIUM class predictions, ALLCoP sets, and sets containing classes where the softmax score was >1-α is shown for α = .075 (Fig 4A), α = .1 (Fig 4B), and α = .15 (Fig 4C). The mean FNR and mean set size are stratified by subtype in the Data Supplement (Table S6). Notably, the FNR of BCR::ABL1-like, which was 90.9% in the uncalibrated ALLIUM output, was reduced to 36.36% at α = .075. Also of note, PAX5alt frequently co-occurred with other subtypes, appearing in 10 of 29 multiclass prediction sets at α = .15 (34.48%), 20 of 65 at α = .10 (30.77%), and 31 of 103 (30.10%) at α = .075.

FIG 4.

Concordance between ALLIUM single-class point predictions, ALLCoP sets, and sets of classes where the softmax score was >1-α in the validation data set with a single known subtype, at preselected FNRs of (A) α = .075, (B) α = .1, and (C) α = .15. FNR, false-negative rate.

We then used the same conformal predictors to obtain prediction sets for samples with multiple known subtypes (n = 65)^28,30 (Data Supplement, Table S7). Briefly, the uncalibrated ALLIUM model had an FNR of 61.54%, which was reduced to 23.85% at α = .075 (Data Supplement, Table S8; performance by subtype in the Data Supplement, Table S9). The prediction sets are visualized in the Data Supplement (Fig S5).

Prediction Sets for Unknown Subtype Patients

We calibrated three instances of ALLCoP using all patients with a single known subtype (n = 851) at error rates $\alpha \in \left\{0.075,0.1,0.15\right\}$ and used them to generate prediction sets for samples with an unknown subtype from all studies (n = 126; Data Supplement, Table S10).

Using the uncalibrated model, ALLIUM issued predictions for 97 patients at the default model softmax threshold of 0.5, leaving 29 empty predictions (23%). At $\alpha$ = .15, 34 of the ALLCoP prediction sets were empty (27%; Fig 5A), but at $\alpha$ = .10, this number dropped to 26 empty sets (21%; Fig 5B), and at $\alpha$ = .075, only 21 of the sets were empty (17%; Fig 5C). Across all three error rates, the most commonly predicted subtypes were PAX5alt, DUX4-r, and iAMP21.

FIG 5.

ALLCoP prediction sets for 126 patients with previously unclassified B-ALL using predictions from ALLIUM. The upset plots represent class membership counts, with unattached dots representing single-class prediction sets and connected dots representing multiclass sets. Visualized are prediction sets for preselected FNRs of (A) α = .15, resulting in 34 empty sets, 78 certain sets, and 14 uncertain sets; (B) α = .10, resulting in 26 empty sets, 80 certain sets, and 20 uncertain sets; and (C) α = .075, resulting in 21 empty sets, 69 certain sets, and 36 uncertain sets. FNR, false-negative rate.

Validation on Additional RNA-Seq ALL Subtype Classifiers

ALLIUM is one of numerous RNA-seq classifiers for molecular subtype determination in ALL.^4-6 To evaluate the generalizability of ALLCoP to other classifiers, we selected two other models, ALLCatchR⁵ and ALLSorts,⁴ and generated predictions for samples that were not used for training these models^3,29 (Data Supplement, Fig S6; Data Supplement, Tables S11-S12). The softmax scores generated by ALLCatchR resulted in discrete distributions (Fig 6A). The subtypes exhibiting the highest uncertainty were iAMP21 (high softmax scores for BCR::ABL1-like, ETV6::RUNX1-like, and high hyperdiploid) and PAX5alt (high softmax scores for BCR::ABL1-like). Similar to ALLCatchR, ALLSorts was most uncertain in predicting iAMP21, often confusing it for the same classes as ALLCatchR (Fig 6B). ALLCatchR generated largely certain predictions, with the mean prediction set size (using a softmax threshold of 0.5) never surpassing 1.0 and remaining over 0.99 for all classes, except PAX5alt (0.93) and iAMP21 (0.69; Fig 6C). Similarly, ALLSorts had a mean set size of over 0.90 for all classes, except KMT2A-r (0.83) and iAMP21 (0.50), although it had more uncertain sets, with a mean set size >1.0 for BCR::ABL1-like (1.14), ZNR384-r (1.12), and PAX5alt (1.07; Fig 6D). In ALLSorts, as with ALLIUM prediction sets, PAX5alt was the subtype most frequently observed in multiclass predictions, although in the discrete model outputs of ALLCatchR, this was not observed.

FIG 6.

Softmax predictions and ALLCoP results for two ALL RNA-seq classifiers, ALLCatchR and ALLSorts. All softmax scores output by the model, stratified by true subtype and colored by predicted subtype, for (A) ALLCatchR and (B) ALLSorts. Heatmaps mapping each true subtype to the mean softmax score per predicted subtype, with the green column showing the mean softmax scores of false positives and the purple column showing the mean size of the prediction sets with softmax threshold = 0.5, for (C) ALLCatchR and (D) ALLSorts. FNR of ALLCoP prediction sets versus uncalibrated model outputs for (E) ALLCatchR and (F) ALLSorts, with the uncalibrated model outputs in yellow. FNR, false-negative rate; RNA-seq, RNA-sequencing.

Next, we applied ALLCoP to the softmax scores and evaluated the output (Data Supplement, Tables S13-S14). Similarly to ALLIUM, ALLCoP produced prediction sets at defined FNRs for AllCatchR and ALLSorts (Figs 6E and 6F). Empirical error rate selection was performed for the two classifiers, showing an optimal error rate of α = .1 for ALLCatchR (softmax threshold of 0.44) and α = .05 for ALLSorts (softmax threshold of 0.18; Data Supplement, Fig S7). These results further demonstrate the robustness and generalizability of ALLCoP for RNA-seq ALL classifiers.

DISCUSSION

In this study, we present the first application of conformal risk control, to our knowledge, to RNA-seq–based ML classification of ALL. Recent and forthcoming legislation, such as the EU AI Act,³³ emphasizes regulatory requirements including transparency, explainability, and accountability in AI systems.^34,35 In this context, the quantification of model robustness and confidence is essential.³⁶ This necessitates methodologies beyond the uncalibrated softmax outputs typically generated by classifiers, which do not reflect empirical probabilities.^37,38

CP, by providing statistical performance certification for predictions, supports future clinical translation of RNA-seq classifiers. At present, a small subset of International Consensus Classification (ICC)–recognized subtypes^8,39 are used to guide therapeutic decision making for ALL treatment. However, recent studies have demonstrated the utility of increasingly fine-grained molecular subtyping both for predicting outcomes and tailoring treatment intensity.⁴⁰ This research landscape is fluid, with both the continuous emergence of novel subtypes⁴¹ and the refinement of existing ICC subtypes.⁴² Established subtypes that show clinical significance include PAX5alt, BCR::ABL1-like, ETV6::RUNX1-like, and MEF2D-r,⁴³ which are among the classes whose ALLIUM predictions exhibited high levels of uncertainty in this study. CP can identify ambiguous cases among these clinically relevant subtypes, flagging them for further human assessment and contributing to more reliable classification outcomes. As such, this framework has the potential to facilitate the translation of refined molecular risk stratification to clinical use and to eventually improve patient outcomes.

The variability of CP set size, referred to as set adaptivity, serves as a valuable metric of uncertainty. This reflects both the performance of the classifier and the irreducible uncertainty arising from the inherent complexity of the features being classified. In our study, the PAX5alt subtype most frequently appeared in multiclass prediction sets generated using ALLIUM³ and ALLSorts.⁴ Given that alterations in the PAX5 gene are observed in over one third of patients with B-cell precursor ALL, as reported in prior studies,^32,44 the multiclass CP sets containing PAX5alt may not solely indicate classifier uncertainty but could also point to biologically relevant secondary aberrations with potential clinical importance.

Finally, we explored the concept of implicit error reduction within the framework of ALL RNA-seq classifiers. In the validation cohorts, we found that lowering the α value resulted in the expected reduction in the mean FNR of the prediction sets. Likewise, for samples with unknown subtypes, a lower α value resulted in fewer empty prediction sets, demonstrating improved model utility.

However, this approach involves a trade-off: Reducing the α value increases the likelihood of including the true class in the prediction sets but results in larger and less precise sets.⁴⁵ The ability to select an α value tailored to specific requirements offers significant flexibility. For instance, a higher tolerance for error may be acceptable in research contexts where exploratory analysis is prioritized. Conversely, high-stakes applications demand stricter error rates to ensure reliable and actionable predictions.^46,47 Obtaining regulatory approval for medical advice applications, for instance, may require performing risk analysis to determine an appropriate error rate,⁴⁸ although AI best practices and regulatory requirements are still in development.⁴⁹ The adaptability of CP underscores its potential as a versatile tool for integrating ML into both research and clinical workflows, balancing precision with reliability on the basis of user-defined thresholds.

In summary, this study demonstrates the potential of CP to enhance the robustness, transparency, and adaptability of RNA-seq–based ML classifiers for ALL subtyping. By addressing both predictive uncertainty and error management, our findings pave the way for integrating advanced AI methodologies into clinical workflows while aligning with emerging regulatory requirements.

PREPRINT VERSION

Preprint available on https://www.biorxiv.org/content/10.1101/2024.12.11.627902v1.

DATA SHARING STATEMENT

A data sharing statement provided by the authors is available with this article at DOI https://doi.org/10.1200/CCI-24-00324. All data^3,28-30 (Data Supplement, Table S1) and code^50-53 (Data Supplement, Table S15) used in this study are publicly available.

AUTHOR CONTRIBUTIONS

Conception and design: Mariya Lysenkova Wiklander, Olga Krali, Jessica Nordlund

Financial support: Jessica Nordlund

Administrative support: Jessica Nordlund

Provision of study materials or patients: Jessica Nordlund

Collection and assembly of data: Mariya Lysenkova Wiklander, Olga Krali, Jessica Nordlund

Data analysis and interpretation: Mariya Lysenkova Wiklander, Dave Zachariah, Olga Krali

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/cci/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).