Consistency of BRCA1 and BRCA2 Variant Classifications Among Clinical Diagnostic Laboratories

Abstract

Background

Genetic tests of the cancer predisposition genes BRCA1 and BRCA2 inform significant clinical decisions for both physicians and patients. Most uncovered variants are benign, and determining which few are pathogenic (disease-causing) is sometimes challenging and can potentially be inconsistent among laboratories. The ClinVar database makes de-identified clinical variant classifications from multiple laboratories publicly available for comparison and review, per recommendations of the American Medical Association (AMA), the American College of Medical Genetics (ACMG), the National Society for Genetic Counselors (NSGC), and other organizations.

Methods

Classifications of more than 2000 BRCA1/2 variants in ClinVar representing approximately 22,000 patients were dichotomized as clinically actionable or not actionable and compared across up to seven laboratories. The properties of these variants and classification differences were investigated in detail.

Results

Per-variant concordance was 98.5% (CI 97.9%–99.0%). All discordant variants were rare; thus, per patient concordance was estimated to be higher: 99.7%. ClinVar facilitated resolution of many of the discordant variants, and concordance increased to 99.0% per variant and 99.8% per patient when reclassified (but not yet resubmitted) variants and submission errors were addressed. Most of the remaining discordances appeared to involve either legitimate differences in expert judgment regarding particular scientific evidence, or were classifications that predated availability of important scientific evidence.

Conclusions

Significant classification disagreements among the professional clinical laboratories represented in ClinVar are infrequent yet important. The unrestricted sharing of clinical genetic data allows detailed interlaboratory quality control and peer review, as exemplified by this study.

Introduction

Hereditary breast and ovarian cancer (HBOC) is a cancer predisposition syndrome resulting from inherited (i.e., germline) loss-of-function mutations in the BRCA1 or BRCA2 genes (collectively, BRCA1/2). Such pathogenic (disease-causing) genetic variants result in a 40–80% lifetime risk of developing breast cancer, an 11–40% risk of ovarian cancer, and striking increases in the risk of male breast, pancreatic, and prostate cancers [1, 2]. Up to 10% of breast cancers are caused by these genes [3, 4]. Approximately 1/250 individuals of European descent are born with a pathogenic variant in BRCA1/2, and prevalence is much higher in certain populations—for example, Ashkenazi Jews [5, 6].

Decades of clinical testing and research have uncovered tens of thousands of BRCA1/2 genetic variants across the human population (http://brcaexchange.org/). The great majority of these variants are benign, conferring no increased cancer risk, whereas others are pathogenic. Still others are considered variants of uncertain significance (VUS) when the current scientific evidence for or against pathogenicity is inadequate or conflicting. To help standardize variant interpretation, the American College of Medical Genetics (ACMG) and the Association for Molecular Pathology (AMP) jointly issued revised guidelines [7] for variant classification. Although more comprehensive and specific than earlier guidelines [8, 9], these guidelines still require laboratory directors to use expert judgment in evaluating the quality of available evidence. Thus, classifications of some genetic variants may vary among laboratories. Moreover, scientific evidence continually evolves, which can change the classifications of some variants over time. Whereas VUS, likely benign, and benign variants in BRCA1/2 are not medically actionable, pathogenic and likely pathogenic variants are, warranting consideration of additional screening, prevention, or treatment options [6, 10, 11]. Thus, rigorous and consistent variant interpretation is critical to patient care.

Variant classifications can also potentially conflict if one laboratory has access to proprietary data that are unavailable to others. In the 1990s, Myriad Genetics patented the BRCA1/2 genes and prohibited testing by other laboratories [12, 13]. Myriad continued as the sole provider for nearly 20 years until the patents were overturned. The company used its monopoly to accumulate a substantial database of variants that it ceased releasing publicly in 2006 and from which it claims a competitive advantage [14–16]. This practice is contrary to the recommendations of the American Medical Association (AMA), the American College of Medical Genetics (ACMG), the National Society for Genetic Counselors (NSGC), and other organizations [17] (Resolution 519, adopted by AMA House of Delegates in 2013, http://www.ama-assn.org/ama/pub/news/news/2013/2013-06-18-new-ama-policies-annual-meeting.page; http://nsgc.org/p/bl/et/blogid=47&blogaid=330). Recognizing that shared knowledge about genetic variants is critical to high-quality medical care, the National Institutes of Health established ClinVar, a public database of clinically observed genetic variants, their pathogenicity classifications from various laboratories, and a summary of the scientific evidence used in those classifications [18–22]. While many commercial and academic laboratories collaboratively submit data to ClinVar, others (including Myriad) do not. Nevertheless, a substantial Myriad data set has been submitted by ordering clinicians and patients through the Sharing Clinical Reports Project (SCRP) (https://www.clinicalgenome.org/data-sharing/sharing-clinical-reports-project-scrp/) [13]. In this study, we used publicly available data from ClinVar to assess agreement among clinical laboratories for classifications of BRCA1/2 variants.

Methods

Classifications of BRCA1/2 variants were extracted from the ClinVar May 2016 release. Variants in ClinVar are classified as pathogenic, likely pathogenic, VUS, likely benign, or benign, consistent with ACMG/AMP terminology [7]. Laboratory-specific classification categories (e.g., deleterious instead of pathogenic or polymorphism instead of benign) are mapped to the standardized nomenclature. Our inclusion criteria restricted analysis to data from (a) licensed clinical laboratories with (b) at least 200 classified BRCA1/2 variants in ClinVar, among which (c) most (>50%) were less than 5 years old. We thus excluded data from research laboratories, consortia, smaller (possibly less experienced) clinical laboratories, and older data sets. ClinVar submissions that we knew were pending (i.e., provided to ClinVar but not yet merged into a monthly release) were incorporated. Duplicate entries were identified and merged, and clearly erroneous entries were repaired or removed. For details, see supplemental methods. The complete dataset used in our analysis is also provided as a supplement.

To compare potential clinical impact, we dichotomized classifications into positive (pathogenic, likely pathogenic) or not positive (benign, likely benign, VUS). Although many laboratories exclude benign and likely benign variants from clinical reports, these variants are often submitted to ClinVar and many are available for comparison [23]. An exception is SCRP, which is derived directly from clinical reports and thus benign and likely benign variants are underrepresented from this submitter (Table 1, Supplemental Table 1).

Table 1.

Data included in this study.

List of ClinVar submitters and counts of classified BRCA1 and BRCA2 variants included in this study. Comparable variants are those with classifications from two or more of the listed submitters. Currently, detailed evidence underlying individual variant classifications is provided to ClinVar only by GeneDx and Invitae for BRCA1/2. There are known biases in the Counsyl data. The SCRP data are derived directly from clinical reports, which usually exclude benign and likely benign variants except when reclassifications are provided. Ambry and Counsyl had not submitted BRCA1/2 updates for more than one year, although all data sets shown here met our study inclusion criteria. See methods and supplemental methods for details.

ClinVar submitter	Classified variants	Comparable variants	Full name In ClinVar	Most recent classification	Evidence provided	Note
Ambry	2792	1613	Ambry Genetics	Feb, 2015
SCRP/Myriad	2327	1351	Sharing Clinical Reports Project	Dec, 2015		Benign and likely benign variants are under reported.
Invitae	1998	1367	Invitae	Mar, 2016	Yes
GeneDx	1216	957	GeneDx	Oct, 2015	Yes
Counsyl	272	256	Counsyl	Feb, 2015		No VUS submitted
CHEO	257	220	Molecular Genetics Diagnostic Laboratory, Children’s Hospital of Eastern Ontario	Dates not provided
Emory	203	183	Emory Genetics Laboratory	June, 2015
Total	5124	2006

Abbreviations: VUS: variants of uncertain significance.

General population allele frequencies for these variants were determined by using the ExAC database [24], the 1000 Genomes Project phase 3 database [25], and the Exome Variant Server (http://evs.gs.washington.edu/EVS/). Common variants were defined as those with allele frequencies over 1% in any of these databases. We used a separate sequential series of over 30,000 patients clinically tested for BRCA1/2 to measure allele frequencies in a clinical population. These patients’ variants were also part of the ClinVar dataset described above.

Confidence intervals (CIs) were computed using the Wilson method [26]. Evaluation of the scientific evidence underlying discordant classifications followed the most recent ACMG/AMP recommendations [7]. Our methods for estimating the number of patients expected to have discordant variants are detailed in the supplement.

Results

There were 5124 BRCA1/2 variants submitted to ClinVar by seven groups that met our inclusion criteria (Table 1). Of these variants, 2006 had classifications from two or more laboratories available for comparison. We call these “comparable variants” (Supplemental Figure 1). The remaining variants had been submitted by only a single source. Nearly 90% of these variants (1769/2006) were rare, having allele frequencies less than 0.05% in all of the general population databases we examined and less than 0.1% in our clinical database (Figure 1). We estimate that the comparable variants represent testing of approximately 22,000 patients (Figure 2). The comparable variants were a representative subset of the 5124 ClinVar variants in overall properties, with an expected bias away from rare variants, albeit small in magnitude (Table 2). The ClinVar data were also representative of those seen in clinical practice, with some submitter-specific exceptions (Table 1; Supplemental Table 1).

Figure 1.

Histograms of ClinVar variants (1a) by population allele frequency in ExAC and (1b) prevalence in our clinical database. By either measure, most variants in ClinVar are very rare, although the vast majority of variants observed in patients are repeated occurrences of a small number of common and intermediate frequency variants.

Figure 2.

Total number of distinct genetic variants observed as patients were added to the clinical database. Because many variants are rare, new variants continue to be accumulated even after many patients have been sequenced. From these data, we estimate that the comparable ClinVar variants in this study (n = 2006) correspond to the number that would be observed if roughly 22,000 patients had been tested by the same laboratories.

Table 2.

Breakdown by allele frequency and variant type.

Total variant counts, classification concordance, and clinical prevalence by allele frequency bin. Rare variants are further broken down by variant type. Percentages indicate the fraction of all variants in this study (column 2), comparable variants (those with two or more submitters) (column 3), and concordance for each variant class (columns 4 and 5). Clinical prevalence (column 6) indicates the fraction of patients in our clinical database who carry one or more such variants, regardless of pathogenicity (many are indeed classified benign). Because patients may carry multiple variants, concordance and prevalence percentages do not sum meaningfully. Variants in the last exon of BRCA1 or BRCA2 were counted as such and were not counted in another applicable category.

	All variants (% of total)	Comparable variants (% of total)	Concordant variants	Concordance (%)	Clinical prevalence (%)	Note
Full data set	5124 (100)	2006(100)	1976	98.5	100
By allele frequency
Common variants	62(1.2)	55 (2.7)	55	100	100
Intermediate	200 (3.9)	182(9.1)	182	100	18.43
Rare variants	4862 (94.9)	1769(88.2)	1739	98.3	12.74
Rare variants by variant type
Missense	2333 (45.5)	913(45.5)	895	98.0	6.35
Truncating	1210 (23.6)	439(21.9)	438	99.8	2.67	The one discordance is a submission error.
Silent	542(10.6)	173(8.6)	173	100.0	1.93
Intronic	253 (4.9)	73 (3.6)	71	97.3	1.53
Last exon	209(4.1)	71 (3.5)	69	97.2	0.428	Both discordances are historical discrepancies.
Canonical splice site	188(3.7)	60 (3.0)	55	91.7	0.310
In-frame indel	81 (1.6)	31 (1.5)	29	93.5	0.323
Copy number del/dup	44 (0.86)	8 (0.40)	8	100.0	0.191	Underrepresented category in ClinVar
Alu insertion	2 (0.04)	1 (0.05)	1	100.0	0.013	Underrepresented category in ClinVar

Abbreviations: del/dep, copy-number deletion or duplication; indel, sequence insertion or deletion.

We compared variant classifications in terms of whether they would or would not potentially impact clinical management (see methods). On a per-variant basis, we found high concordance: 98.5% of the comparable variants (1976/2006, CI 97.9–99.0%) had concordant classifications among all submitters. Only 30/2006 showed discordance between any two submitters (see supplemental data for details). Pairwise concordance between laboratories was also high, varying between 97.2% and 100.0% (Supplemental Table 2).

Importantly, the 30 clinically significant discordances were in rare variants that, by definition, are present in very few patients. Based on the prevalence distribution of variants in clinical testing, we calculated the expected concordance on a per-patient basis to be 99.7% (Figure 3; see supplemental methods for details). An independent calculation based on population allele frequencies confirmed this result. This concordance rate is similar to that reported (99.8%) in a prior study of approximately 1000 prospectively accrued patients that compared ACMG/AMP-based classifications with those from Myriad Genetics [27, 28].

Figure 3.

Summary of per-patient concordance. The only classification discordances we observed were in rare variants, which few patients carry, and most rare variants were completely concordant when observed by multiple laboratories. See supplemental methods for details.

A feature of ClinVar is that it records the date on which each classification was made, thereby allowing us to consider whether the high concordance we observed could be a result of laboratories being overly influenced by each other’s prior classifications. Such influence would be most concerning in the case of Myriad classifications submitted by SCRP, for which the underlying evidence is unavailable for other laboratory directors to evaluate [13]. We saw no evidence of such bias, as 99.4% (503/506) of the classifications that pre-dated a Myriad/SCRP entry were concordant with it compared with 99.1% (2385/2406) that postdated it (see supplemental data). These rates are not significantly different.

Although classifications from the ENIGMA consortium (Evidence-based Network from the Interpretation of Germline Mutant Alleles) [29] were not considered in our comparison, they can provide evidence that clinical laboratories use in their classifications. ENIGMA classifications were available for 250 variants in our study with, in total, 996 laboratory classifications (see supplemental data). Only 2 of 996 were discordant with ENIGMA, both of which were Myriad/SCRP classifications that predate the corresponding ENIGMA submissions.

Among laboratories, the largest class of discordant variants we observed were rare missense changes (18/30) that alter only a single amino acid in the resulting protein. These variants are numerous, composing almost half (913/2006, 45.5%) of our data set, and the vast majority (895/913, 98.0%) had concordant classifications among all submitters (Table 2). Although clearly important, rare missense variants are infrequently observed in patients (6.3% prevalence in our clinical data set). Rare protein truncating and silent variants were also numerous (439/2006 and 173/2006, respectively; 30.5% of the data set together) and were concordantly classified with one exception (see below). Other discordant variants were in canonical RNA splice sites (5/30) or an intron (2/30) or were in-frame deletions (2/30). Relatively few variants of these types were reported; they are of low prevalence, and most are concordant. Finally, two truncating mutations in the last coding exon of BRCA2 had discordant classifications (see below).

To gain insight into the basis for the small number of discordant classifications, we examined all publicly available evidence for and against pathogenicity for each of the 30 discordant variants. We also contacted submitting laboratories regarding specific discordant classifications, particularly those for variants with three or more submitters. We found two common explanations for discordant classifications (see supplemental data for details): (a) In seven variants there was a “historical difference,” meaning that one or more classifications in ClinVar was out of date, and although the variant had been re-classified (thus becoming concordant), updates had not yet been submitted to ClinVar. Two data submission errors were also identified. Including updates to these nine variants raised concordance to 99.0% per variant and 99.8% per patient and resolved all of the discordant truncating mutations. In four additional cases we suspect a historical difference because key evidence (e.g., a publication) that could significantly affect classification postdates a discordant ClinVar entry. (b) Seventeen discordances appeared to be legitimate differences in the judgment of laboratory experts assessing the available evidence of pathogenicity. While proprietary data may have contributed to some discordances—particularly those involving Myriad/SCRP data—most of those cases have an alternative, plausible explanation (i.e., historical discrepancy or difference in expert judgment).

An unusually challenging example of discordance is BRCA1 splicing variant c.594-2A>C, reported by Myriad/SCRP as pathogenic but later downgraded to VUS [30]. Other laboratories in our study also classify this variant as VUS, although one reclassification (to VUS) had not yet propagated into ClinVar (Olga Jarinova, personal communication). We thus considered this a historical discordance. This mutation causes upregulation of an endogenous alternate RNA isoform missing exons 9 and 10 but that appears to provide BRCA1 functionality [31]. This result suggests that laboratory directors should carefully evaluate other mutations in exons 9 and 10 and highlights the complexities of variant classification that laboratory directors must consider in some cases.

Overall however, our analysis suggests that a high level of concordance should perhaps have been expected. Most (83%) of the pathogenic variants were of types that are relatively straightforward to classify (e.g., truncating mutations or large deletions in most regions of BRCA1/2; see supplemental data). Other variant types (notably rare missense and splice-site changes) require additional experimental or genetic data to classify, but relatively few patients (<7%) carry any such variants, and they are often (97.6%) concordant across laboratories (Table 2). When not considered VUS, these variants are usually classified as benign or likely benign, not pathogenic (see supplemental data). Indeed, a comparison of ClinVar releases over the past 2 years shows that most (94.5%) missense VUS, when reclassified, are downgraded to benign or likely benign and thus remain not clinically actionable, consistent with prior studies [32].

Discussion

In this study, we analyzed publicly available data from the ClinVar database and found remarkably few clinically significant discordances in the classifications of over 2000 variants in the two well-characterized cancer risk genes BRCA1 and BRCA2. The observation that all discordant variants were rare, although most rare variants remained concordant, suggests that roughly 1/500 patients would be expected to receive results that would significantly change clinical management from the various laboratories in this study. By comparison, concordance can be far lower among pathologists reading breast biopsies or radiologists reviewing mammograms [33–36]. These genetic test reports would not always be identical, both because our analysis grouped classifications (i.e., we considered potentially actionable pathogenic and likely pathogenic classifications together, and we considered non-actionable benign, likely benign and VUS together) and because laboratories vary on whether benign and likely benign variants are included in reports (VUS may also be excluded from reports in a screening context). Nevertheless the reports’ significance for clinical management decisions remain similar.

Furthermore, we explored the likely cause of the few observed discordances and found that approximately half resulted from out-of-date classifications or submission errors, whereas the remainder were likely expert judgment differences regarding the strength or quality of particular scientific evidence. We were pleased that all but one laboratory responded collaboratively to requests for detailed information about their classifications—despite those requests coming directly from a commercial competitor (Invitae). This process of identifying and reconciling differences was made possible by shared data in a central and unrestricted public database (ClinVar).

Our findings might at first appear to be at odds with other studies comparing variant classifications. A study by Vail et al. [37] compared the interpretation of approximately 2000 BRCA1/2 variants among a number of public databases and found greater discordance than we report. This study’s methodology was significantly different from ours: Most importantly, it incorporated data from research laboratories, older data, and data from curated literature databases that were not classified using modern clinical criteria [7, 38, 39]. Furthermore, it counted differences that would not significantly change management (e.g., VUS versus likely benign). Finally, it measured discordance only on a per-variant basis, not per-patient, which in our analysis was dramatically lower. These methodological differences exaggerate the impact of discordance on clinical application.

Other studies have addressed variant classification concordance under different clinical circumstances. Maxwell et al. [40] studied their own application of the ACMG guidelines to variants in a diverse set of hereditary cancer genes observed in patients, finding an overall per-variant concordance with ClinVar that was high (95%) but lower than our corresponding result (98.5%). We examined all BRCA1/2 variants in their study and found 100% concordance with our ClinVar data. The discordances that Maxwell et al. found were in other cancer genes, only recently incorporated into tests, for which less information is generally available and thus discordance may indeed be higher than it is for BRCA1/2.

Separately, Balmaña et al. [41, 42] examined variants in cancer genes other than BRCA1/2 in the PROMPT registry (http://promptstudy.info). The authors found 19 unique variants, representing 57 of 603 comparable test findings (9.5%), that had two or more significantly different interpretations in ClinVar. The authors concluded that “conflicting interpretation … is frequent and may have implications for medical management”. We examined current (September 30, 2016) ClinVar entries for all 19 of these variants and found that six had discordant interpretations only from a non-clinical source (most commonly, OMIM [43]), while all clinical laboratories in fact agreed with each other [42]. One variant was no longer discordant following a 1-year old (but more recently submitted) reclassification. Two low-risk variants had discordance attributable to the fact that nomenclature and classification criteria for such variants are not standardized under current ACMG guidelines, yet most labs still agreed. We count 10 of Balmaña’s variants representing 2.2% of findings (13/603) having a clinically substantial discordance between clinical testing laboratories, 4.3-fold fewer than the 57/603 they report. Some of this remaining discordance in non-BRCA1/2 genes appears attributable to factors we describe above for BRCA1/2 (e.g. older data) although unfortunately Balmaña et al. did not contact the submitting laboratories to understand the basis of discordance as we did.

In another study, the NIH-funded CSER consortium performed an experiment in which 99 variants in various genes—biased toward relatively challenging cases—were classified by up to nine laboratories [44]. Although many classification differences were observed, only a fraction would change management, and only five variants in BRCA1/2 were included. Importantly, the authors found that sharing classifications among laboratories and thus identifying discordances enabled discussions that resolved many of the differences and contributed to an overall higher quality than any one laboratory could achieve alone.

Our study and those mentioned above highlight important best practices in the use of public databases. Although variant classifications from all sources are valuable and important to centralize and share, database users must apply good judgment and quality control. They must pay attention to dates, as variant classifications can become outdated, for example, when new scientific evidence is published. Moreover, users must consider whether a classification originates from a clinical laboratory that rigorously follows guidelines-based classification procedures or from a submitter who may have applied a different standard. Finally, database users must evaluate the underlying scientific evidence for each classification, just as they do when considering variant classifications in any publication. At present only two of the laboratories included in this study, GeneDx and Invitae, provide the evidence supporting the classification of specific variants in their ClinVar submissions, a situation that we hope will change. Some other laboratories include evidence only in patient reports, but these are not broadly available for both logistical and patient privacy reasons.

Although our analysis shows that clinically significant disagreements in BRCA1/2 variant classification are infrequent, they are, of course, important to patients and clinicians. We believe it is essential for the genetics community to resolve these differences collaboratively, as is standard practice in other areas of oncology, in order to deliver the best possible patient care [18–20, 33–36, 44]. Our study supports others in demonstrating that collaborative interaction among laboratories improves the quality of clinical testing [34–36, 44]. Unlike proprietary databases, ClinVar is freely open to all and makes such collaboration possible on a global scale. Moreover, ClinVar enables independent assessment of variant classification accuracy and consistency, as exemplified by this study. Although laboratories with proprietary databases have made claims of superior accuracy, such claims are not subject to detailed and ongoing independent review [13, 16]. Indeed our observation of high concordance across laboratories calls some of those claims into question. We note that semi-public databases with restrictive licensing terms such a BRCA Share (http://www.umd.be/BRCA1/ and http://www.umd.be/BRCA2/) [45] can present many of the same challenges encountered with proprietary databases (license restrictions prevented the consideration of such data in this study, for example). We also note that patient registries including PROMPT, while highly valuable for other reasons, do not address the needs that ClinVar does [13,18].

For these reasons, the open sharing of de-identified variant classifications is recommended by the AMA, ACMG, NSGC, and other professional societies. In collaboration with international groups, NIH has recently funded initiatives including ClinGen [18–19] and the BRCA Exchange (http://brcaexchange.org) that leverage ClinVar, the literature, and other resources, to share, compare, and reconcile variant classifications, thus continually improving this important aspect of precision oncology. However, at least one major laboratory, Myriad Genetics, has revised its terms of service to prohibit ordering clinicians from sharing de-identified variant classifications (https://www.myriadpro.com/company/terms-of-use/, accessed November 15, 2016), which is how the SCRP data used in this study were obtained. Myriad has also historically resisted requests from patients for their unreported benign variants, prompting legal action by the American Civil Liberties Union [46]. Such restrictions will make ongoing comparative analyses impossible. We hope that our study illustrates the importance of open and unrestricted genetic data sharing via ClinVar and the value of supporting this critical initiative.