Establishing guidelines to harmonize tumor mutational burden (TMB): in silico assessment of variation in TMB quantification across diagnostic platforms: phase I of the Friends of Cancer Research TMB Harmonization Project

Abstract

Background

Tumor mutational burden (TMB), defined as the number of somatic mutations per megabase of interrogated genomic sequence, demonstrates predictive biomarker potential for the identification of patients with cancer most likely to respond to immune checkpoint inhibitors. TMB is optimally calculated by whole exome sequencing (WES), but next-generation sequencing targeted panels provide TMB estimates in a time-effective and cost-effective manner. However, differences in panel size and gene coverage, in addition to the underlying bioinformatics pipelines, are known drivers of variability in TMB estimates across laboratories. By directly comparing panel-based TMB estimates from participating laboratories, this study aims to characterize the theoretical variability of panel-based TMB estimates, and provides guidelines on TMB reporting, analytic validation requirements and reference standard alignment in order to maintain consistency of TMB estimation across platforms.

Methods

Eleven laboratories used WES data from The Cancer Genome Atlas Multi-Center Mutation calling in Multiple Cancers (MC3) samples and calculated TMB from the subset of the exome restricted to the genes covered by their targeted panel using their own bioinformatics pipeline (panel TMB). A reference TMB value was calculated from the entire exome using a uniform bioinformatics pipeline all members agreed on (WES TMB). Linear regression analyses were performed to investigate the relationship between WES and panel TMB for all 32 cancer types combined and separately. Variability in panel TMB values at various WES TMB values was also quantified using 95% prediction limits.

Results

Study results demonstrated that variability within and between panel TMB values increases as the WES TMB values increase. For each panel, prediction limits based on linear regression analyses that modeled panel TMB as a function of WES TMB were calculated and found to approximately capture the intended 95% of observed panel TMB values. Certain cancer types, such as uterine, bladder and colon cancers exhibited greater variability in panel TMB values, compared with lung and head and neck cancers.

Conclusions

Increasing uptake of TMB as a predictive biomarker in the clinic creates an urgent need to bring stakeholders together to agree on the harmonization of key aspects of panel-based TMB estimation, such as the standardization of TMB reporting, standardization of analytical validation studies and the alignment of panel-based TMB values with a reference standard. These harmonization efforts should improve consistency and reliability of panel TMB estimates and aid in clinical decision-making.

Background

Immune checkpoint inhibitors (ICIs) have recently emerged as a pillar of cancer care, providing the potential for durable responses and improved survival for patients across multiple cancer types.1–3 An intensive clinical development pipeline investigating ICIs is ongoing as a result. However, not all patients with cancer respond to ICIs, with modest response rates for several approved indications (approximately 20% or less in lung cancer, bladder cancer and cancers of the head and neck, among others) and high treatment costs. There is a crucial interest in the development of biomarker assays to predict which patients are most likely to respond and benefit from ICIs, and to improve clinical decision-making and disease management.4 5

Expression of the programmed cell death ligand protein-1 (PD-L1) by immunohistochemistry (IHC) has been studied extensively as a biomarker of response to anti-PD-L1 and programmed cell death protein 1 (PD-1) therapy. Several assays have been developed to quantify tumor PD-L1 immuno-positivity; however, quantitation is imperfect, and lack of standardization across platforms and scoring systems precludes assay interchangeability.6 Tumor mutational burden (TMB), which measures the number of somatic mutations per megabase (Mb) of the interrogated genomic sequence of a tumor, has been most recently identified as a biomarker of response to ICIs in several cancer types. High TMB is associated with improved outcomes in patients with melanoma treated with cytotoxic T-lymphocyte-associated protein 4 (CTLA-4) blockade7–9 and PD-1/PD-L1 blockade across several cancer types, including melanoma,10 11 non-small-cell lung carcinoma,12–15 bladder cancer,16 microsatellite instability cancers3 17 and pan-tumor cohorts.18–20 High TMB has also been associated with improved outcomes in patients treated with a combination of PD-1/PD-L1 and CTLA-4 inhibitors.21–24

Initial assessments of TMB involved whole exome sequencing (WES) of matched tumor tissue and normal specimens using next-generation sequencing (NGS).3 8–10 However, WES is not currently routine in clinical practice due to substantial cost and turnaround time, which has led assay manufacturers and commercial and academic labs to develop targeted NGS panels. These targeted panels, which cover several hundred genes, are already routinely used in clinical practice, and are currently being adapted to estimate TMB. TMB estimated from targeted NGS panels has generally correlated well with TMB determined by WES, however the reliability of this technology is still being assessed.13–16 20 22 25–30

There are several targeted NGS panels at different stages of development that estimate TMB. To date, the Foundation Medicine FoundationOne CDx test31 is currently the only Food and Drug Administration (FDA)-approved panel, which includes TMB as part of its tumor profiling claim, while the Memorial Sloan Kettering Cancer Center MSK-IMPACT (Integrated Mutation Profiling of Actionable Cancer Targets)32 has received FDA authorization. Additionally, there are many more commercial and laboratory-developed test panels currently under development. Each panel has unique features integrated into their design that may impact TMB estimation. For example, each panel may include different numbers and types of genes, use different sequencing platforms, have different methods of filtering germline mutations, incorporate different mutation types in the quantification of TMB and use proprietary bioinformatics protocols to calculate TMB.33 34 Thus, TMB estimates will vary according to the targeted panel used.35 This is a crucial time to understand the differences in TMB estimation across panels, standardize the way TMB is reported, begin to harmonize methods for TMB quantification and identify optimal approaches to promote TMB alignment across different targeted NGS panels.

Friends of Cancer Research (Friends) convened a consortium of key stakeholders, including diagnostic manufacturers, academics, pharmaceutical companies, the National Cancer Institute and the FDA, to recommend best practices and approaches for TMB measurement, validation, alignment and reporting well ahead of the adoption of this powerful biomarker in clinical decision-making. Leveraging the expertise and insights of this comprehensive group of stakeholders, the Friends TMB harmonization project seeks to establish a uniform approach to measure and report TMB across different sequencing panels by harmonizing the definition of TMB, proposing best practices for analytic validation studies and ensuring consistency of TMB calculation through alignment with a universal reference standard. The project consists of a stepwise approach broken down into three phases: phase I, reported here, comprises the in silico analysis, which by using publicly available data from The Cancer Genome Atlas (TCGA) representing 32 cancer types, aims to identify the theoretical variability of panel-derived TMB estimates (panel TMB) relative to a common, standardized WES-derived TMB (WES TMB) across various panels. Building on the results of the in silico analysis, phase II will analyze human tumor clinical sample material to objectively measure variation across panels using patient formalin-fixed paraffin-embedded (FFPE) tissue samples. This empirical analysis will also compare panel TMB results to an agreed on universal reference standard, consisting of a collection of human tumor-derived reference cell lines that span a clinically meaningful TMB dynamic range. FFPE tissue samples will also be used to validate the use of the cell line standard. Finally, phase III will involve a clinical study that seeks to retrospectively analyze samples from patients treated with ICIs to evaluate optimal cut-off values that will help guide the clinical application of TMB (see online supplementary figure 1).

The need for harmonization of TMB is a global effort, which is portrayed by the representation of national and international diagnostic companies in the consortium. Moreover, in seeking to complement the consortium’s work, the Friends TMB harmonization project has partnered with the technical comparability study conducted by Quality in Pathology in Germany,36 leading to the identification of common and panel-specific factors that influence TMB estimation and the development of global recommendations, which have been published previously.33

Due to the large scale and collaborative nature of this effort, study results will greatly contribute to understanding and refining how to best quantify and interpret TMB as a biomarker, help establish standards that will facilitate harmonization across different testing platforms and inform future harmonization efforts that seek to ensure consistency across diagnostic platforms.

Methods

In silico dataset

Mutation calls generated using Multi-Center Mutation calling in Multiple Cancers (MC3) WES data from TCGA project were used for this analysis.37 Variants that overlapped with the CCDS, using bedtools (-wa option)38 were extracted from the publicly available mc3.v0.2.8.PUBLIC.maf file (https://gdc.cancer.gov/about-data/publications/pancanatlas). Finally, the data were filtered for any overlap or redundancy using the ‘merge’ function. The consortium created a final bed file that covered 32.102 Mb of the genome after intersecting the data found in the MAF files and filtering for any overlap or redundancy (see online supplementary methods). The final bed file size was used as the denominator for calculating WES TMB in this study. Three different consortium laboratories independently calculated WES TMB using the same dataset and analytical methodology with 100% concordance.

Ten thousand two hundred ninety-five tumor samples with matched normal initially composed part of the cohort. Only samples with at least one variant which PASSED variant review filter were used (see online supplementary methods for variant quality filters). Low quality samples based on variant filters and those with low purity were also removed from further analysis. The remaining cases (n=8291) were randomly assigned to training (n=4157) and validation (n=4134) datasets with similar median candidate mutations and cancer types (online supplementary figure 2). Participants, though not required, could use the ‘training’ set for their own algorithm or parameter testing. However, all analyses described herein were conducted using the validation dataset.

The evaluations reported in the present study are those comparing panel TMB to WES TMB on the validation set, with no adjustments made to the panel TMB algorithms once the validation set analyses began. All analyses focused on tumors for which WES TMB was ≤40 because >98% of the TCGA dataset tumors investigated had TMB ≤40 in the TCGA dataset and all members of the consortium agreed that this range would have the greatest relevance for clinical decision-making. Of the 4134 tumors initially represented in the validation set, 4065 remained after excluding those with WES TMB >40. All results were blinded to the entire consortium, with the exception of the project statistician and data manager (LMMS and DMM) who were regarded as neutral parties not affiliated with any of the participating laboratories.

Statistical analysis

Statistical analyses interrogated the relationship between WES TMB and panel TMB values. The first analysis focused on the combined data from all 32 tumor types. Spearman’s R correlation values were calculated, and scatterplots and difference plots were created to assess linearity of the relationship between panel TMB and WES TMB and to evaluate whether variance of panel TMB was constant across the range of WES TMB values.

Next, the 32 tumors were divided into three strata according to the number of samples within each tumor type that had TMB values spanning the range 0–40 mut/Mb (see online supplementary methods and figure 3). Stratum 1 contained eight tumor types (see online supplementary table 1A—stratum 1) displaying a good distribution of TMB values spanning the range of interest (0–40 mut/Mb). Seventy-seven per cent of samples (1257/1627) had TMB ≤10 mut/Mb, 19% (306/1627) had TMB 10–40 mut/Mb and 4% (64/1627) had TMB ≥40 mut/Mb and were thus eliminated from further analyses. Stratum 2 was represented by 11 tumor types (see online supplementary table 1B—stratum 2) whose samples had generally low TMB values (≤10 mut/Mb, 98%, 1723/1754), and only 1.5% (26/1754) of samples had TMB 10–40 mut/Mb. Only five samples (0.29%) had TMB ≥40 mut/Mb and were thus eliminated from further analyses. Stratum 3 was represented by 13 tumor types (see online supplementary table 1C—stratum 3) whose samples had very low TMB values (≤5 mut/Mb, 99.5%, 749/753) and only 4 samples (0.5%) had samples with TMB between 5 and 10 mut/Mb. Regression modeling using weighted least squares was implemented to account for the heteroscedasticity in errors, referring to the variability in panel TMB values about the fitted regression line, which was observed to increase with the mean and with WES TMB. This modeling was conducted for all strata, although we focused on stratum 1 considering strata 2 and 3 provided less stable and unreliable estimates due to the large number of samples that concentrated in the lower end of the TMB range.

For each regression, the mean panel TMB was modeled as a simple linear function of the WES TMB, and five different models for the error variance were considered (see online supplementary methods). Restricted maximum likelihood analysis using the gls function available in the R package nlme was performed to estimate the model parameters and select a best fitting variance structure based on minimum Akaike information and Bayesian information criteria.

Whole exome analysis

The whole exome analysis of the TCGA MC3 validation dataset used an agreed on methodology to calculate the WES TMB values, termed the Uniform TMB Calculation Method (see online supplementary table 2). The goal of phase I of this harmonization study is to assess the theoretical variability across panels. Given that the participating panels were at different stages of development and had different sensitivity levels, the consortium decided to use the Uniform TMB Calculation Method, which would enable the selection of high-quality variants that all laboratories were able to assess as part of their panels. The consortium created a custom bed file covering 32.102 Mb of the genome which was used to calculate the reference WES TMB values. The calculated WES TMB values comprised the reference dataset for this study. The uniform method for analysis of WES TMB included minimum thresholds for median target coverage (median 300X as this was identified as the point where sensitivity for the lower allele frequency variants drops drastically) (see online supplementary figure 4), variant allele frequency (≥0.05), read depth (≥25) and variant count (≥3), and synonymous variants were excluded.

Panel analysis

Each participating laboratory calculated TMB from the subset of the exome restricted to the genes covered by their targeted panel and using their own unique bioinformatics pipeline (panel TMB). If available, the laboratory’s bioinformatics analysis has been reported in table 1.

Table 1.

Description of the 11 participating diagnostic NGS panels

Laboratory	Panel name	# genes	Total region covered (Mb)	TMB region covered* (Mb)	Type of exonic mutations included in TMB estimation	Published performance characteristics (ref.)
ACT Genomics	ACTOnco+	440	1.8	1.12	Non-synonymous†, synonymous	NA
AstraZeneca	AZ600	607	1.72	1.72	Non-synonymous, synonymous	NA
Caris	SureSelect XT	592	1.60	1.40	Non-synonymous	Vanderwalde et al40
Foundation Medicine	FoundationOne CDx‡	324	2.20	0.80	Non-synonymous, synonymous	Frampton et al41 Chalmers et al25 Fabrizio et al42 US FDA SSED31
Guardant Health	GuardantOMNI§	500	2.15	1.00	Non-synonymous, synonymous	Quinn et al43
Illumina	TSO500 (TruSight Oncology 500)	523	1.97	1.33	Non-synonymous, synonymous	NA
Memorial Sloan Kettering Cancer Center	MSK-IMPACT¶	468	1.53	1.14	Non-synonymous	Cheng et al,44 Zehir et al,30 US FDA32
NeoGenomics	NeoTYPE Discovery Profile for Solid Tumors	372	1.10	1.03	Non-synonymous, synonymous	NA
Personal Genome Diagnostics	PGDx elio tissue complete	507	2.20	1.33	Non-synonymous, synonymous	Wood et al45
QIAGEN	QIAseq TMB panel	486	1.33	1.33	Non-synonymous, synonymous	NA
Thermo Fisher Scientific	Oncomine Tumor Mutation Load Assay	409	1.70	1.20	Non-synonymous	Chaudhary et al46 Endris et al35

*Coding region used to estimate TMB regardless of the size of the region assessed by the panel.

†Non-synonymous mutations include single nucleotide variants, splice-site variants and short insertions and deletions (indels).

‡FoundationOne CDx assay has been approved by the US FDA as an IVD.31

§GuardantOMNI is a plasma-based circulating tumor DNA assay.

¶MSK-IMPACT assay has been authorized by the US FDA32

NA, not available.

The panel-derived TMB datasets were sent to a neutral third party (DMM) who assigned coded identifiers to the laboratories to mask which laboratory contributed each dataset. All subsequent data analyses were conducted by LMMS and DMM. Participating laboratories were not involved in the analyses and were not provided the key to the coded lab identifiers.

Results

In silico assessment of theoretical TMB variation across panels

Eleven academic and commercial laboratories with targeted gene panels in different stages of development participated in this study (table 1). The size of the coding region used to estimate TMB from these gene panels ranged between 0.80 and 1.72 Mb. And the number of genes in each of the gene panels ranged between 324 and 607 genes. All participating laboratories included exonic somatic non-synonymous, frameshift and splice site variants and short indels when estimating TMB. Eight panels (8/11, 73%) also included synonymous variants in their estimation. Each laboratory used their own bioinformatics algorithms and workflows, which were optimized using the sequencing methods, mutation types and filters that best suited their own panel specifications. Since the participating panels were in different stages of development, only a few had published panel performance characteristics (table 1).

The WES TMB values were calculated using the TCGA MC3 Mutation Annotation Format (MAF) validation dataset and an agreed on methodology (see ‘Whole exome analysis’ section and online supplementary table 2). The panel TMB values on the same validation dataset were estimated by down-sampling to the regions covered by each of the laboratories’ panels and applying their own bioinformatics algorithms. To prevent the misinterpretation of this study’s results as an interlab performance study, all laboratories agreed for the results to be blinded with respect to the lab generating each dataset.

First, all 32 cancer types in the TCGA MC3 dataset were investigated together using weighted linear regression analysis (generalized least squares, see ‘Methods’ section). Some variation was observed across panels, with Spearman’s rank correlation values (R) ranging from 0.79 to 0.88, and slope values ranging from 0.87 to 1.47 (figure 1A, online supplementary figure 5). Eight laboratories (73%) had slope values >1, demonstrating an overestimation of TMB. Panel factors that may influence TMB overestimation were not assessed due to the blinded study design but may have included the type of mutations counted for the panel TMB value (eg, synonymous alterations included in panel TMB that were excluded from the WES estimation), among others.

Figure 1.

Estimated regression lines for panel tumor mutational burden (TMB) as a function of whole exome sequencing (WES) TMB for each of the 11 participating laboratories analyzing (A) all cancer types combined and (B) stratum 1 cancer types combined. Solid lines represent the fitted regression lines. Red dashed line represents 45^o line.

In silico assessment of theoretical TMB variation across panels by cancer type

A limitation of analyzing all cancer types together is the variable distribution of TMB across different cancer types, with some cancer types displaying large dynamic ranges of TMB values up to several hundred mutations per Mb and others with very limited distributions with very few samples reaching 20 mutations per Mb (see online supplementary figure 3). To account for this limitation, cancer types were categorized into strata by their distribution of WES TMB values. Stratum 1 (n=1563 samples with <40 mut/Mb) had samples with a good distribution of WES TMB values covering 0–40 mut/Mb, which enabled a more robust regression analysis across a clinically relevant TMB range. The eight cancer types in stratum 1 were: bladder urothelial carcinoma (BLCA, n=195), colon adenocarcinoma (COAD, n=128), head and neck squamous cell carcinoma (HNSC, n=232), lung adenocarcinoma (LUAD, n=228), lung squamous cell carcinoma (LUSC, n=228), skin cutaneous melanoma (SKCM, n=166), stomach adenocarcinoma (STAD, n=189) and uterine corpus endometrial carcinoma (UCEC, n=197).

Regression analyses restricted to stratum 1 tumors revealed an association between WES TMB and panel TMB similar to that for all cancer types analyzed together (Spearman’s R: 0.81–0.90 and slope 0.80–1.32, figure 1B, per laboratory online supplementary figure 6 and table 3). The slopes calculated when stratum 1 tumors were analyzed were consistently lower than when all cancers were analyzed. The greatest differences in slope values when comparing slopes estimated for all cancers and for stratum 1 tumors only, were observed for labs 8 (all cancers 1.47 vs stratum 1 1.32) and 9 (all cancers 1.24 vs stratum 1 1.1) (both ∆ 0.15), while labs 4 (all cancers 0.904 vs stratum 1 0.897) and 2 (all cancers 1.087 vs stratum 1 1.076) had the least differences (∆ 0.007 and 0.01, respectively). When stratum 1 tumors were analyzed, only six laboratories (55%) reported overestimation of TMB with slope values >1.

Regression analyses with stratum 2 and 3 were not robust, as the WES TMB values did not adequately cover the entire clinically meaningful range (see online supplementary figures 7 and 8, and table 3).

Lastly, the eight cancers in stratum 1 were analyzed separately. UCEC, BLCA and COAD had the broadest range of slope values (UCEC: range 0.755–1.602, ∆ 0.847; BLCA: range 1.042–1.79, ∆ 0.748; COAD: range 0.75–1.486, ∆ 0.736) (figure 2, online supplementary table 4), and most laboratories consistently overestimated these cancer types, with BLCA as the only cancer type for which all 11 laboratories (100%) consistently overestimated their panel TMB values relative to WES TMB. Conversely, LUAD, LUSC and HNSC had the tightest range of slope values with no consistent bias to overestimating or underestimating TMB (LUAD: range 0.817–1.135, ∆ 0.318; LUSC: range 0.741–1.099, ∆ 0.358; HNSC: range 0.854–1.244, ∆ 0.39).

Figure 2.

Estimated regression lines for panel tumor mutational burden (TMB) as a function of whole exome sequencing (WES) TMB for the eight cancer types within stratum 1. All cancer types had a good distribution of WES TMB values from 0 to 40 mut/Mb. Solid lines represent the fitted regression lines. Red dashed line represents 45^o line. BLCA, bladder urothelial carcinoma; COAD, colon adenocarcinoma; HNSC, head and neck squamous cell carcinoma; LUAD, lung adenocarcinoma; LUSC, lung squamous cell carcinoma; SKCM, skin cutaneous melanoma; STAD, stomach adenocarcinoma; UCEC, uterine corpus endometrial carcinoma.

Defining the theoretical variation in TMB across panels and by cancer type

Prediction limits for the observed panel TMB at fixed WES TMB (5, 10, 15 and 20 mut/Mb) were calculated to quantify the variability around the regression line at those selected WES TMB values. The limits were designed to capture approximately 95% of the panel TMB values expected to be observed at a given WES TMB. Some laboratories had consistently tighter (narrower) prediction intervals, while others demonstrated more variability (wider intervals), but for all laboratories, the prediction intervals became tighter with decreasing WES TMB value, indicating greater variability in panel TMB at larger WES TMB values (figure 3). Generally, the prediction intervals observed for each participating laboratory were similar, with laboratories demonstrating intervals that spanned as small as ±4.7 mut/Mb or as large as ±12.3 mut/Mb when the WES TMB was 10 mut/Mb, which is a TMB threshold that has been previously used to define a TMB-high cohort using NGS panels.33 When prediction limits were assessed by strata, the variability of the intervals was very large for cancer types in strata 2 and 3 compared with stratum 1 because most TMB values for the cancers in these strata accumulate in the lower end of the TMB spectrum, thus resulting in more uncertainty in the fitted regression lines and wide scatter in panel TMB values around those lines (see online supplementary figure 9). When the eight stratum 1 cancers were analyzed separately, prediction intervals at the discreet value of WES TMB=10 mut/Mb were observed to be wider for BLCA and UCEC, while LUAD, LUSC, HNSC and SKCM had the tightest intervals (figure 4). This is similar to the observed variation in fitted regression lines for BLCA and UCEC across laboratories (figure 2). The theoretical variability around the regression was also seen to increase (wider intervals) with increasing TMB value in individual cancer types (see online supplementary figure 10).

Figure 3.

Ninety-five per cent prediction intervals for panel tumor mutational burden (TMB) estimated at discreet whole exome sequencing (WES) TMB values (5, 10, 15 and 20 mut/Mb), by laboratory across all laboratories. Blue arrows represent the estimated mean panel TMB for each laboratory. Red dashed line represents the discreet WES TMB value at which prediction interval is calculated.

Figure 4.

Ninety-five per cent prediction intervals for panel TMB (x-axis) of stratum 1 tumor types at whole exome sequencing tumor mutational burden (WES TMB) 10 mut/Mb by laboratory (y-axis). Blue arrows represent the estimated mean panel TMB for each laboratory. Red dashed line indicates WES TMB value=10 mut/Mb. BLCA, bladder urothelial carcinoma; COAD, colon adenocarcinoma; HNSC, head and neck squamous cell carcinoma; LUAD, lung adenocarcinoma; LUSC, lung squamous cell carcinoma; SKCM, skin cutaneous melanoma; STAD, stomach adenocarcinoma; UCEC, uterine corpus endometrial carcinoma.

Discussion

Eleven laboratories with distinct NGS targeted gene panels and bioinformatic approaches participated in phase I of the Friends of Cancer Research TMB Harmonization Project and provided early insights into the theoretical variability across different targeted gene panels that estimate TMB. The goal of the first phase of the project was to describe the variability in TMB estimates across several uniquely designed panel-based diagnostic assays and to further elucidate the theoretical variation in TMB quantification using an in silico approach with a large publicly available dataset with high-quality reads and a common reference TMB standard calculated from the entire exome. Moreover, dependence of the association between panel TMB and WES TMB on cancer type was investigated.

Variability in panel TMB across different panels was observed, with some panels consistently overestimating or underestimating TMB, suggesting that panel size and composition, as well as laboratories’ bioinformatics algorithms, including types of mutations counted and variant filters used in the TMB calculation, were likely contributors to the differences. Because of the blinded design of this study, the influence of these factors on panel TMB variability was not evaluated in this early phase of the project but will be assessed in the following empirical phase to be reported subsequently. Additionally, other studies have recently reported on the impact of panel size, DNA input and variant filtering on panel-based TMB estimates.35 39

The study evaluated a robust dataset containing 32 cancer types, with very few cases having TMB >40 mut/Mb (n=69/4134, 1.7%), so it was not possible to robustly estimate the association between panel TMB and WES TMB for cases with values >40 mut/Mb. TMB data were thus capped at 40 mut/Mb and a linear relationship was used to model the relationship in that range. Factoring the limited dynamic range of TMB values observed in some cancer types, a subset of eight cancers was identified and named stratum one for the primary analysis. Stratum 1 cancer types included lung, bladder, head and neck, skin, colon, uterine and gastric cancers, all of which have been shown to respond to immune checkpoint inhibitors. Evaluating these cancers separately revealed distinct levels of variability in the association between panel TMB and WES TMB across panels, with some cancer types having less variability (eg, lung and head and neck cancers), and some having greater variability (eg, uterine, bladder and colon cancers). As our initial findings suggest that panels may perform differently on certain cancer types, further work is required to understand the factors contributing to any disease-specific TMB variability, and the relationships beyond the analyzed TMB range. However, the composition of the panels’ genes, types of mutations counted or methods used to train their respective TMB algorithms could be future areas of focus.

Despite these cancer-dependent findings, our study found that panel TMB values were strongly correlated with WES TMB across laboratories. Additionally, the calculated 95% prediction intervals permitted estimation of the linear relationship between panel TMB and WES TMB as well as quantification of the range in which 95% of the observed TMB panel values would be expected to fall for tumors with various fixed WES TMB values. This provides a framework for understanding the theoretical variability likely to be incurred in the clinical application of TMB estimation across panels, but also suggests that harmonization of TMB estimates could be achieved through alignment using external reference materials. There is still, however, much that can be done to improve the reliability of using NGS panels for TMB estimation.

The selection of high-quality variants from the TCGA MC3 dataset was used to assess the theoretical variability of TMB across panels in this study ensuring the interpretability of the findings where the assessment of variability was limited to factors such as panel size and composition or bioinformatics pipeline, instead of perceived differences regarding sensitivity and specificity of individual variant calling. However, we acknowledge that in a clinical setting the estimation of TMB from FFPE tissue may introduce variants of lesser quality and panels should aim to validate the sensitivity and specificity of individual variant calling separately from TMB validation.

As TMB measurements are most likely to be impactful in treatment decisions for stratum one cancer types, including these tumors as part of a laboratory’s analytical validation studies to achieve optimal accuracy and consistency is critical. On the other hand, it is also important to recognize that there are cancer types with generally low TMB values that may have a few cases with high TMB values that may benefit from reliable panel TMB results. Moreover, because of the cancer type-dependent distribution of TMB values, studies aiming to evaluate the clinical utility of TMB and determine optimal TMB cut-offs for treatment decisions may need to account for specific cancer types. This would be consistent with a recent report that found that in patients who received ICI, those who had high TMB had longer survival than those who had low TMB, but TMB-high cut-offs were cancer-type dependent.20

The Friends of Cancer Research TMB Harmonization Consortium includes the participation of several leading commercial and academic laboratories as well as a diverse group of stakeholders, who together identified opportunities for standardization to promote the harmonization of TMB estimation. These have led the consortium to recommend best practices for panel developers that seek to promote consistency in alignment and facilitate commutability across panels table 2. These recommendations revolve around the following three items.

Table 2.

Consensus recommendations for the standardization of analytical validation studies of targeted NGS panels that estimate TMB

Parameters	Recommendations
Accuracy*	Accuracy or agreement should be measured by comparing the TMB values generated by the assay requiring validation against reference TMB values generated either from: A comparable companion diagnostic approved by a regulatory agency, such as the FDA, if available, OR A WES assay with validated performance characteristics and using an accepted WES TMB calculation method, such as the common method reported in this study (see online supplementary table 1). The minimum number of samples used for evaluation of accuracy should be at least 30. Samples should have TMB values that span the entire analytical range being investigated (0–40 mut/Mb is recommended currently). TMB as a continuous score: This analysis will characterize the analytical performance of the assay over the analytical range of interest per the intended use. Quantification of performance based on TMB as a continuous variable should include an appropriate regression analysis and a scatter plot showing the association between the panel and reference TMB values. Additionally, quantification of performance should examine the pointwise prediction intervals for panel TMB values as obtained from the regression analysis, at predefined reference TMB values. A description of the absolute deviation from the mean should also be reported. For TMB as a categorical call: accuracy should be analytically validated using a single or multiple discreet TMB cut-off values within the analytical range of interest. Quantification of performance of TMB as a categorical call should be based on 2×2 agreement tables to inform the positive per cent agreement, negative per cent agreement and overall per cent agreement informed by 2×2 agreement tables. For assays pursuing a companion diagnostic claim, accuracy should be examined using a predetermined discrete cut-off value investigated in a study using clinical samples covering the spectrum of conditions from a defined, or intent-to-treat (ITT) population. If the ITT population includes multiple cancer types, stratified analyses should be conducted. Alteration level agreement: as a supplemental analysis, the agreement between alteration level calls (including single nucleotide variants (SNVs) and short indels) between the platform and the reference should be provided for each alteration included in the TMB panel and restricted to the overlapping genomic regions between the two assays. Report the concordance of variant calls between variants identified by WES and panel as a function of the panel variant allele frequency (VAF). Characterize the percentage of tests passing QC by reporting first pass acceptability rate and overall acceptability rate (after samples have been retested, if necessary).
Precision*	Precision should be evaluated using several samples. For each sample, separate analyses should be performed as described in the TMB as a continuous score and TMB as a categorical call sections below. Analytical validation of precision of TMB as both a continuous score and a categorical call will improve reliability of TMB as a biomarker. Because TMB is a composite estimate composed of different variants, its precision should be evaluated as a composite score (mut/Mb). TMB as a continuous score: Precision studies of quantitative TMB estimates should evaluate the mean, SD and coefficient of variation of TMB values obtained from testing aliquots of the same sample under stipulated precision conditions (eg, replicates, runs, instruments, lots, operators) for a range of samples (5–6 samples with 20 TMB results distributed across the precision conditions each) with TMB values within the analytical range (0–40 mut/Mb is recommended currently), and include different levels of tumor content and VAF values. Note: identification of the TMB range to be evaluated should be guided by the most recently published clinically relevant studies. Precision of the TMB score should be estimated using a variance component analysis to estimate between-run, within-run, between instruments, between lots and between operator SD for each sample. Quantification of performance should include calculation of repeatability and within-lab SD for each sample corresponding to several discreet TMB values, given that a single average value of variation (eg, coefficient of variation pooled across several samples having different TMB levels) may not best reflect the changing variability across the TMB range. TMB as a categorical call: consistency of the categorical calls (eg, TMB high vs TMB low according to a discreet cut-off) should be evaluated based on both repeatability and within-laboratory precision for the TMB results according to a single or multiple discreet cut-off values. For repeatability, calculate the per cent of TMB high calls (if majority call is high) or per cent of TMB low calls (if majority call is low) between replicate samples tested under the same lab conditions. For within laboratory precision, calculate the per cent of TMB high calls (if majority call is high) or per cent of TMB low calls (if majority call is low) and the mean TMB score from replicate samples tested under varying within-lab conditions. Note: the number of aliquots tested per sample should be sufficient to account for the various sources of assay variability, such as the ones described above (TMB as a continuous score). Moreover, the number of samples tested should be similar. Emphasis should be placed on evaluating samples with TMB values: Significantly below cut-off (approximates limit of blank, expect TMB low almost 100% of time). Near and below cut-off (expect TMB low 95% of time). Near and above cut-off (expect TMB high 95% of time). Significantly above cut-off (expect very high TMB almost 100% of time). Alteration level precision: the evaluation of precision for each individual alteration call used to estimate TMB is not necessary but may be performed as an exploratory analysis to provide insight into the mechanisms that contribute to the TMB score variability. Per cent tumor content should be collected when evaluating precision and reported, if applicable. Characterize the percentage of tests passing QC by reporting first pass acceptability rate and overall acceptability rate (after samples have been retested, if necessary).
Sensitivity*	The impact of tumor content of a sample on the TMB categorical call (high, low) should be evaluated using multiple samples, taking into consideration the precision of the TMB score as a function of decreasing tumor content. Undiluted samples should have a range of expected TMB scores, a range of VAF values for somatic mutations and a ratio of SNVs and indels that are representative of clinical samples. The evaluation of panel sensitivity to tumor content should be done using: Samples: 6–10 undiluted samples where each sample is diluted to at least 5 levels of tumor content. Each sample should have a dilution series ranging from well above and below the expected sensitivity limits for tumor content. Note: It is likely that matched normal will be required to generate each respective dilution value. Consideration should be given to technical and biological factors that may impact the choice of the normal sample and design of the dilution series. At each dilution level, at least 10 replicate samples should be tested. For each sample, the evaluation should include a calculation of the per cent of TMB high (above a predefined TMB threshold) across replicates at each dilution and a probit regression of per cent TMB high versus tumor content. From the regression, report the estimated tumor content where the probability of detecting TMB high is 95%.
Limit of blank*	Non-tumor samples should be used to establish the limit of blank for TMB, yielding results close to, but not always equal to 0 mut/Mb. Considerations should be given to technical and biological factors, such as age of patient and distance from tumor lesion, among others.
Percentage of tests passing QC	Report percentage of tests passing TMB QC metrics in routine testing. Example QC metrics for TMB might include: median exon coverage, coverage uniformity, contamination rate.

*The definitions of the terms used in this table are based on the Clinical and Laboratory Standards Institute Harmonized Terminology Database at http://htd.clsi.org/.

FDA, Food and Drug Administration; NGS, next-generation sequencing; QC, quality control; TMB, tumor mutational burden; WES, whole exome sequencing.

1. Ensure reporting consistency: TMB should be reported in mutations/megabase (mut/Mb)

The current practice of reporting WES-derived TMB values as number of somatic mutations, while panel-derived TMB values are reported as a density of somatic mutations per Mb of genomic region covered by the panel (mut/Mb), precludes the aggregate analysis of TMB being derived from WES or targeted panels, especially since the size of the exome interrogated using different platforms may not be consistent. Reporting TMB as mutations/megabase (mut/Mb) in order to keep these values consistent and comparable is recommended by the consortium.

2. Analytical validation studies for TMB estimation should be standardized to include assessment of analytical accuracy, precision and sensitivity

Size of targeted gene panel, technical sensitivity of the assay and pre-analytical and analytical variables are known to contribute to variability in panel-based TMB estimates.33 The same in silico data were used by every participating laboratory, which created a theoretical setting that focused the investigation on potential sources of variability that are unique to the technical specifications of the panel (eg, size and composition), and the bioinformatics approaches of each laboratory (eg, mutation types counted and germline and hotspot mutation filtering). Some of these factors cannot be easily modified and standardized across laboratories as panel assays are, for the most part, proprietary and have been designed to optimize their respective technical specifications and conditions. However, harmonization of TMB estimates may be achieved across laboratories by ensuring that the analytical validation studies for each panel follow a standard approach including alignment of panel TMB values to an external reference standard. Recommendations for analyzing accuracy, precision and sensitivity of TMB values to tumor content when used both as a continuous score and a categorical call have been proposed by the consortium (table 2). These recommendations will ensure that regardless of the type of panel or bioinformatics pipeline a laboratory decides to use, TMB estimates are held to a standard of acceptable reliability.

3. Consistency across panels could be ensured through alignment of panel TMB values to WES-derived universal reference standard

Comparison to WES TMB is currently the most recognized way to determine accuracy of panel TMB. However, it should be noted that differences in performance between panel TMB and WES TMB are to be expected based on differences in coverage depth between the two methods, with typically greater depth and higher variability observed in panels.

Universal reference standards, with TMB values spanning a clinically relevant range (eg, 0–40 mut/Mb), represent a promising tool to achieve alignment or calibration in order to ensure consistency of the TMB estimation across platforms, regardless of known sources of variability. An ideal reference standard for TMB estimation should be generated from a renewable source and its TMB values should be calculated using WES. To mitigate differences resulting from comparisons using multiple different WES assays, a universally accepted, predefined bioinformatics pipeline and statistical methods should be implemented. A calibration curve generated using the reference standard should be used to normalize and compare across panels, which should promote alignment and aid in the analytical validation of panel TMB values.

Conclusions

Harmonization of methodologies for the accurate measurement of complex continuous biomarkers is an ongoing effort. The Friends of Cancer Research TMB Harmonization Project has convened key stakeholders early in the development of NGS assays that estimate TMB to more effectively identify avenues for the harmonization of estimation approaches and to emphasize the need for the uptake and implementation of these harmonization recommendations. The results included in this report are the initial results from this stepwise approach, but future studies will focus on assessing the feasibility of using tumor-derived cell lines as external reference standards to help facilitate alignment of panel TMB values. Additional empirical analyses will also be conducted to investigate the influence of biologic factors (eg, specimen type, cancer type) and technical factors (eg, sequencing technology) on panel TMB, continue refining best practices for panel assessment of TMB, and developing alignment approaches to improve interchangeability between TMB estimates generated from different targeted gene panels.

Lastly, the collaborative efforts of the TMB Harmonization Consortium will serve as a framework for future harmonization initiatives that seek to standardize complex quantitative biomarker assays and promote the reliability of biomarker testing.