Comparative Analysis of Public Knowledge Bases for Precision Oncology

Abstract

PURPOSE

Precision oncology depends on the availability of up-to-date, comprehensive, and accurate information about associations between genetic variants and therapeutic options. Recently, a number of knowledge bases (KBs) have been developed that gather such information on the basis of expert curation of the scientific literature. We performed a quantitative and qualitative comparison of Clinical Interpretations of Variants in Cancer, OncoKB, Cancer Gene Census, Database of Curated Mutations, CGI Biomarkers (the cancer genome interpreter biomarker database), Tumor Alterations Relevant for Genomics-Driven Therapy, and the Precision Medicine Knowledge Base.

METHODS

We downloaded each KB and restructured their content to describe variants, genes, drugs, and gene-drug associations in a common format. We normalized gene names to Entrez Gene IDs and drug names to ChEMBL and DrugBank IDs. For the analysis of clinically relevant gene-drug associations, we obtained lists of genes affected by genetic alterations and putative drug therapies for 113 patients with cancer whose cases were presented at the Molecular Tumor Board (MTB) of the Charité Comprehensive Cancer Center.

RESULTS

Our analysis revealed that the KBs are largely overlapping but also that each source harbors a notable amount of unique information. Although some KBs cover more genes, others contain more data about gene-drug associations. Retrospective comparisons with findings of the Charitè MTB at the gene level showed that use of multiple KBs may considerably improve retrieval results. The relative importance of a KB in terms of cancer genes was assessed in more detail by logistic regression, which revealed that all but one source had a notable impact on result quality. We confirmed these findings using a second data set obtained from an independent MTB.

CONCLUSION

To date, none of the existing publicly available KBs on gene-drug associations in precision oncology fully subsumes the others, but all of them exhibit specific strengths and weaknesses. Consideration of multiple KBs, therefore, is essential to obtain comprehensive results.

INTRODUCTION

Precision oncology (PO) is based on the molecular characterization of tumors and the integration of these data into clinical decision making.¹ Molecular characterizations typically are based on the identification of genomic variants by genomic sequencing, which results in patient-specific variant profiles of single nucleotide variants, small insertions or deletions, copy number variations, and gene fusion events. Depending on the mutational load of the tumors and the sequencing technology used, such profiles may contain between a few to several hundred variants. The identification of clinically relevant biomarkers among these alterations is a central task in PO. This bottleneck of clinical interpretation in PO currently is based mostly on manual work and still lacks standardization.²

Clinical variant interpretation must be based on published information.³ However, such information currently is dispersed across a number of databases, repositories, and Web sites. Therefore, identification, interpretation, and prioritization of biomarkers require manual screening for information in multiple data sources, which leads to heterogeneity between interpreters.⁴ The differences and relative comprehensiveness of individual data sources for this task are currently unknown.

CONTEXT

• Key Objective

• Several knowledge bases that support the clinical evaluation of genetic variants in patients with cancer have emerged during the past years, but their mutual overlaps and extents are unknown.

• Knowledge Generated

• We compared seven knowledge bases and found that they share information only to a moderate extent and that they altogether lack important data. We used discussions about 13 patients within a molecular tumor board of the Charité Comprehensive Cancer Center as reference. Also, none of the knowledge bases is clearly more comprehensive than the others.

• Relevance

• The creation of structured and easy-to-use knowledge bases to support precision oncology has made notable progress in the past years, yet the current systems remain incomplete and inhomogeneous. Our results underline the necessity that, to obtain comprehensive information, oncologists must interrogate multiple knowledge bases and search relevant literature directly.

Here, we report a comparison of seven curated variant knowledge bases (KBs)—Clinical Interpretations of Variants in Cancer (CIViC),⁵ OncoKB,⁶ Database of Curated Mutations (DoCM),⁷ Cancer Gene Census (CGC),⁸ Tumor Alterations Relevant for Genomics-Driven Therapy (TARGET),⁹ Precision Medicine Knowledge Base (PMKB),¹⁰ and CGI Biomarkers.¹¹ We compared these KBs by measuring their mutual overlap of genes, drugs, and gene-drug associations and, if possible, by comparing each KB’s content to actual treatment recommendations for 113 patients with cancer whose cases were discussed at the Molecular Tumor Board (MTB) of the Charité Comprehensive Cancer Center. Results were confirmed using a second, independent set of recommendations for 10 patients with cancer.

METHODS

KBs

We performed PubMed searches and expert interviews to obtain a list of seven publicly available knowledge bases that aim at supporting medical decision making in precision oncology (PO-KB) by providing structured information regarding the relationship between genes, variants, cancer entities, and therapies. Specifically, we used (1) CIViC (Clinical Interpretations of Variants in Cancer), (2) OncoKB, (3) CGC (Cancer Gene Census), (4) DoCM (Database of Curated Mutations), (5) CGI Biomarkers (the cancer genome interpreter biomarker database), (6) TARGET (Tumor Alterations Relevant for Genomics-Driven Therapy), and (7) PMKB (the Precision Medicine Knowledge Base). The databases are described in more details in the Data Supplement; basic statistics are listed in Table 1. The technical data integration procedure and normalization methods we developed for genes, variants, and drugs can be found in the Data Supplement.

TABLE 1.

List of Knowledge Bases Considered

Overlap Comparison

For a quantitative comparison, we computed the mutual overlap of all sources with regard to the set of genes, variants, drugs, and gene-drug associations they contain and visualized them using upset diagrams.¹² For clustering of PO-KBs, we first computed pairwise distances using Jaccard’s formula for set similarity and then applied hierarchical clustering (Data Supplement).

Comparison of Clinically Relevant Gene-Drug Associations

We compared the content of each PO-KB alone and in combinations versus recommendations of the Charité Comprehensive Cancer Center’s MTB (n = 113 patient cases), at which results of comprehensive molecular analysis (whole-exome or panel sequencing, RNA sequencing, immunohistochemical validations) from patients with advanced cancer are discussed on a weekly basis. Clinical interpretation of molecular alterations in the MTB is performed by a trained physician^3,13; the Data Supplement contains details on this process. Results are stored in a structured form. We stripped all information other than patient ID, gene name, and drug name (together called MTB associations). Note that MTB associations are not equivalent to therapeutic recommendations but should be considered as gene-drug associations that are possibly relevant for the given patient. We measured the overlap between this reference set and different combinations of PO-KBs for each patient using the metrics precision (percentage of associations of a [set of] PO-KB that is also contained in the reference), recall (percentage of associations in the reference that is also contained in a [set of] PO-KB) and F1 measure (harmonic mean between precision and recall). These measures are visually displayed later in the Results, and more details can be found in the Data Supplement.

To assess the predictive power of the different PO-KBs, we proceeded as follows: For each association present in any of the PO-KBs, a vector representation was constructed to encode its coverage by CIViC, CGI Biomarkers, TARGET, and OncoKB, respectively. The resulting set of vectors was randomly split into a training set (75%) and a test set (25%). Every gene-drug association that also was listed in the MTB associations was considered positive (association relevant; n = 345 cases), and all others were considered negative (association not relevant; n = 808 cases). We next trained a logistic regression classifier on the training data in a 10-fold cross-validation scheme to compute the out-of-sample error. Eventually, a new model was trained on the entire training set to take this error into account (using R package CARET); we report the results of the second model when applied on the test set.

To validate our findings, we used a second data set, which described treatment recommendations for patients with cancer, as derived at the Molecularly Aided Stratification for Tumor Eradication (MASTERS)¹⁴ program at the National Center for Tumor Diseases in Heidelberg (n = 10) and obtained from Perera-Bel et al.¹⁵ The normalization of gene and drug names for both data sets is described in the Data Supplement. Note that, in this data set, no a priori gene prioritization was performed, which led to a large number of variations per patient. To make results comparable to our first data set, we removed all affected genes for which we did not have a single association in any of the PO-KBs or for which no drug recommendation had been provided. To compensate for the positive bias in this set, we then added twice the number of genes chosen randomly from those we removed before.

RESULTS

Overlap of Studied KBs

We downloaded and normalized seven publicly available databases that aim to support clinical decision making in PO. We found that all databases overlap to a certain degree in the information they provide but also that each provided unique information. Figure 1 shows the overlap at gene (Fig 1A), variant (Fig 1B), and drug (Fig 1C) levels for all seven PO-KBs. CGC stood out because of the much larger list of genes it contains. CIViC (n = 89 genes) and OncoKB (n = 86 genes) have the second largest number of unique genes not mentioned in any other PO-KB; 49 genes were contained in all seven databases. The Data Supplement shows a hierarchical clustering of PO-KBs on the basis of their overlaps in genes, variants, and gene-drug associations.

FIG 1.

No source has it all. Diagrams show the overlap between different precision oncology knowledge bases (PO-KBs; identified by colors) The matrices in the lower panels indicate PO-KBs involved in an intersection, and the bar plots show the number of elements in this intersection. (A) Overlap on the basis of genes. (B) Overlap on the basis of variants. From the PO-KBs analyzed in this work, only four provide variant-level information. (C) Overlap on the basis of drugs. CGI, CGI Biomarkers (the cancer genome interpreter biomarker database); CIViC, Clinical Interpretations of Variants in Cancer; DB, database; DoCM, Database of Curated Mutations; PMKB, Precision Medicine Knowledge Base; TARGET, Tumor Alterations Relevant for Genomics-Driven Therapy.

Although CIViC, OncoKB, and PMKB share a highly similar goal and creation process and also obtain their data from the same origin (scientific publications), each of them contains a substantial amount of unique genes. The most probable reason for this observation is that all PO-KBs are small compared with the huge amounts of published data. The Data Supplement lists all genes contained in one or more of the databases. In addition, the four sources that contain drug information provide both shared and unique information (Fig 1B): 17 drugs are mentioned in all sources, and CIViC has the largest number of unique drugs (n = 223). Similarly, CIViC shows the largest number of unique gene-drug associations (n = 698; Fig 1C).

Comparison of Clinically Relevant Gene-Drug Associations

We next compared the content of each PO-KB and combinations thereof versus expert considerations for the 113 patients discussed at the Charité MTB at the level of gene-drug associations. On average, the experts discussed 6.79 genes (median, five; standard deviation [SD], 5.24) per patient, for whom they discussed 19.42 potentially relevant drugs (median, 18; SD, 12.85). The number of drugs is high, because, for the analysis, drug classes (eg, mammalian target of rapamycin inhibitors) were expanded to all drugs of the respective class to match the granularity of information in different data sets. We analyzed which patients for whom the associations presented to the MTB also were contained in different PO-KBs (Fig 2) and which fractions of gene-drug associations presented in the MTBs also would be found by specific combinations of PO-KBs (Fig 3).

FIG 2.

Enforcing evidences from multiple precision oncology knowledge bases (PO-KBs) leads to fewer gene-drug associations that could be recommended. (A) Percentage of patients for which any gene-drug associations were found when their presence was required in one to four PO-KBs (y-axis). The total number of patients is depicted within the circles. (B) Boxplots show the distribution of gene-drug associations contained in an increasing number of PO-KBs. Each dot represents one patient, and only patients for whom at least one gene-drug association had been found were used.

FIG 3.

Balancing precision and recall to achieve robust results. (A) Visualization of the definition of precision and recall. Colors correspond to the colors used in the other panels. (B) Median precision (blue), recall (red) and F1 (harmonic mean between precision and recall) score (gray) when considering an increasing number of precision oncology knowledge bases (PO-KBs) to support gene-drug associations discussed by the Molecular Tumor Board (MTB) experts. (C) Precision and (D) Recall obtained by using different combination of PO-KBs as boxplots (median, 25th and 75th percentiles; whiskers extend to the maximum values within 1.5 × the interquartile range; dots represent outliers). CGI, CGI Biomarkers (the cancer genome interpreter biomarker database); CIViC, Clinical Interpretations of Variants in Cancer; PR, precision and recall; TARGET, Tumor Alterations Relevant for Genomics-Driven Therapy.

Overall, the MTB discussed 203 different genes and 245 different drugs, of which 152 and 176, respectively, were found in at least one PO-KB. This implies that 25% of genes and 28% of drugs discussed in the MTB could not be found by our software in any of the PO-KBs we considered. Next, we computed the set of prioritized genes per patient and checked for how many of those had associations with drugs in at least one, two, three, or four PO-KBs. Figure 2A shows that 112 patients could receive at least one drug recommendation if support by just one PO-KB is considered as sufficient. When four PO-KBs must support a given gene-drug association, the number of patients decreases to 42. Figure 2B shows that the number of potential suggestions per patient drops sharply with the required number of supporting PO-KBs: When support by any two PO-KBs was required, we obtained a median of five associations per patient. The median increased to 12 if only one PO-KB was required and decreased to two for three PO-KBs and zero for four PO-KBs.

Next, we considered the MTB associations as reference and the contents of the four PO-KBs that contained gene-drug associations as predictors. When the MTB reference gene-drug association was required to be contained in only one PO-KBs—that is, the union of all PO-KBs was used as predictor—the median recall for all patients reached approximately 46% (Fig 3B, blue line), whereas the median precision was low because of the large number of associations found in the PO-KBs that were not discussed in the MTB (Fig 3B, orange line; Data Supplement contains evaluation metrics). When an MTB association was required to be contained in three of the four databases, precision reached 55%, whereas recall decreased to less than 12%, which indicated that gene-drug associations contained in many PO-KBs are often relevant for the MTB but that this applies to only a small number of cases. Figure 3C (precision) and Figure 3D (recall) show the same type of analysis for any combination of one to four PO-KBs. CIViC alone yielded the best F1 score (0.29); however, the best combination without CIViC (OncoKB + CGI Biomarkers) was only marginally worse (bar at position 8; F1 score = 0.26). Using all PO-KBs together offered the highest recall (R = 0.52). Excluding CIViC from the prediction led to a sharp decrease in this metric (best score without CIViC, R = 0.30). This behavior can be explained by the overall highest number of gene-drug associations contained only in this database. TARGET had no notable effect on recall. Using only OncoKB and/or CGI Biomarkers led to the highest precision. Intuitively, this means that searching in more than one PO-KB in our evaluation always led to finding more relevant data but at the cost of also finding more data that are irrelevant for the concrete patient according to the MTB.

Finally, we trained a logistic regression classifier on evidence from all PO-KBs using MTB associations as ground truth. Note that this analysis implicitly learns to weight the different PO-KBs differently, whereas the previous results considered all PO-KBs as equally important. On our withheld test set (n = 288 gene-drug associations), this model reached an F1 score of 43% (precision, 64%; recall, 33%). Inspection of the regression coefficient revealed that the highest impact on performance was achieved by OncoKB (8.99) followed by CIViC (6.48) and CGI Biomarkers (2.78). The impact of TARGET was marginal (0.32).

Validation With Second, Independent Data Set

To validate our findings, we performed the same type of analysis using a second data set originally created in the MTB at the NCT Heidelberg and using the same model as for the Berlin data (ie, the classifier trained on the patients from the Berlin MTB). On the NCT data set, this model reached a precision of 100% at a recall of 38%, which resulted in a higher F1 score of 56%.

The Data Supplement contains an interactive HTML page with the complete list of gene-drug associations of the Berlin and the Heidelberg data sets together with the information about which of the PO-KBs contains which reference associations.

DISCUSSION

We performed a quantitative and qualitative comparison of the current content (February 2019) of seven knowledge bases specialized in PO. Our results indicate that the PO-KBs have partly overlapping and partly unique results. Compared with associations discussed for 113 patients in an MTB of a comprehensive cancer center at a university clinic, we found that PO-KBs contain relevant information not present in any of the others. Our detailed analysis of various quality metrics also shows that a simple quality ranking of PO-KBs is impossible. However, we also want to point out some limitations of our work that warrant additional studies.

Even after careful normalization of gene and drug names in all PO-KBs (Data Supplement), 809 of the 1,154 gene-drug associations of the MTB data set were not found in any of the PO-KBs. This limitation explains the comparably low recall shown in Figure 3 and the low recall of the logistic regression classifier, especially for the first data set. We performed a manual analysis to quantify the potential sources of errors that led to this gap. Reasons for missing associations can be grouped into three partly overlapping categories. Three hundred six gene-drug associations with 51 unique genes were not found because these genes were not listed in any of the PO-KBs. Possible reasons for their absence are that these genes are simply not yet curated, are grossly misspelled, or designate gene families rather than individual genes and so could not be matched by our normalization procedure. Similarly, 157 associations with 69 unique drugs were not found because these drugs are not contained in any of the KBs; 383 associations were missing because none of the PO-KBs reported this particular association, although both the genes and the drugs were found in principle.

Our analysis showed that CIViC has the highest recall and F1 score compared with MTB associations when data from only one PO-KB are considered (ie, no combinations were allowed). This strength can partly be explained by the fact that Charité MTB members tend to submit results of their patient discussions to CIViC. As of January 2019, 746 (approximately 12.8%) of 5,797 CIViC evidence items have been added by Charité staff. These submissions could be the reason for the higher recall of PO-KB combinations contained in CIViC (Fig 3D); however, CIViC also simply contains the highest number of gene-drug associations among all PO-KBs considered, which also positively influences recall. Separation of these two effects is difficult, but we note that regression results on the Heidelberg data set, for which no bias toward CIViC exists, are even better than for the MTB data set. To study the potential bias of CIViC on the Charité MTB data even more, we repeated our regression analysis using all PO-KBs except CIViC. On both data sets, removal of CIViC from the training data left the precision of the learned model unchanged but induced a notable decrease in recall (MTB, 33% decreased to 19%; Heidelberg, 38% decreased to 21%) and, thus, in the F1 score (MTB, 43% decreased to 29%; Heidelberg, 56% decreased to 35%). The fact that the decrease in recall was comparable in both data sets is an argument against a strong bias of the Charité MTB analysis toward CIViC; the overall strong decrease originates from the large number of gene-drug associations only contained in CIViC.

As reported, we trained a classifier on combinations of the different PO-KBs using the MTB associations as ground truth. However, we note that the reported results in terms of prediction accuracy cannot be considered as the ability to predict treatments in a clinical decision support setting for multiple reasons. First, we did not normalize for the impact of gene prioritization but only worked with a prefiltered gene list. Second, we considered all associations in the MTB data as relevant without taking into account the final decision of the MTB, which could be the recommendation of a single drug, could be a combination of drugs, or could disregard all discussed associations. Third, any reasonable recommendation algorithm must take much more information on patients into account than only mutations in genes. Fourth, our study was purely retrospective. Fifth, the patient cases presented in the MTB represent a large variety of different tumor types, such as breast cancer, pancreatic cancer, leiomyosarcoma, ovarian cancer, liposarcoma, and neuroendocrine tumors (Data Supplement). Our analysis ignored the information on cancer types, because this level of detail frequently is missing in the PO-KBs and, when present, uses a highly idiosyncratic nomenclature. The Heidelberg data set was equally diverse in terms of tumor entities. Note that analysis of recommendations derived from genomic information across different cancer types is commonplace in current studies about PO.^16,17

In conclusion, we here report a quantitative and qualitative comparison of precision oncology knowledge databases. Our analysis shows that each of the databases contains, besides a rather small common core, a relevant amount of unique information. When we compared PO-KB contents with gene-drug associations discussed in two MTBs, we found that a considerable fraction of information indeed can be found in some PO-KBs but also that many presumably clinically relevant associations are not yet contained in any of them and that relevant information often is dispersed across different PO-KBs. Therefore, the use of more than one KB currently is strongly advisable and must be complemented by independent literature search, because much more work still is needed to transform all findings that have been published in the biomedical literature into a structured and searchable database format. We also remark that using different PO-KBs as parts of a clinical decision support system calls for a careful consideration and mapping of the evidence levels and codes frequently annotated to gene-drug associations, because these often follow highly heterogeneous definitions.¹⁸

AUTHOR CONTRIBUTIONS

Conception and design: Steffen Pallarz, Manuela Benary, Damian Rieke, Johannes Starlinger, Jurica Ševa, Reinhold Schäfer, Ulf Leser

Collection and assembly of data: Steffen Pallarz, Manuela Benary, Mario Lamping, David Luis Wiegand, Marc Seibert, Ulrich Keilholz, Ulf Leser

Provision of study material or patients: Mario Lamping, Damian Rieke, Ulrich Keilholz

Data analysis and interpretation: Steffen Pallarz, Manuela Benary, Damian Rieke, Christine Sers, Reinhold Schäfer, Ulrich Keilholz, Ulf Leser

Administrative support: Ulrich Keilholz

Manuscript writing: All authors

Final approval of manuscript: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/po/author-center.

Damian Rieke

Honoraria: Bristol-Myers Squibb

Consulting or Advisory Role: Alacris Theranostics

Reinhold Schaefer

Stock and Other Ownership Interests: Cellular Phenomics and Oncology

Ulrich Keilholz

Honoraria: Bristol-Myers Squibb, Merck KGaA, MSD Oncology, AstraZeneca, Novartis, Pfizer, Glycotope GmbH, Roche, Genentech

Consulting or Advisory Role: Bristol-Myers Squibb, Merck Serono, AstraZeneca, MSD Oncology, Pfizer

Speakers' Bureau: MSD Oncology, Bristol-Myers Squibb, Novartis, Merck Serono, Glycotope GmbH, AstraZeneca

Research Funding: Pfizer (Inst), AstraZeneca (Inst), MedImmune (Inst)

Travel, Accommodations, Expenses: AstraZeneca, Merck Serono, MSD Oncology, Ipsen

Ulf Leser

Employment: Roche (I)

Research Funding: Bayer (Inst)

No other potential conflicts of interest were reported.