Table 1. Performance metrics for random forest binary classification models.
Models were trained at IC50 cutoff values of 0.1 μM, 1 μM, and 10 μM, using 145 cellular oncogene mutation statuses among the set of predictors. Reported errors are calculated as standard deviations from 5-fold cross-validation. Baseline values of accuracy, negative predictive value, and Cohen’s kappa statistic at each cutoff value are shown in parentheses. The kappa statistic gauges overall prediction strength, including the tradeoff between specificity and sensitivity, in a single metric. The first baseline value within each set of parentheses is the average baseline value calculated using the tested dataset-based baseline method (dummy classifier) that leads to the highest baseline performance, as measured by Cohen’s kappa statistic; for the GDSC data set, the highest dataset-based baseline performance is yielded by the k-nearest neighbors algorithm (k = 9). The second baseline value within each set of parentheses corresponds to the overall best-performing classification machine learning method (other than random forest) that we tested for the GDSC data set, as evaluated by the kappa statistic (S2 Table). The machine learning method yielding the highest kappa statistic and overall performance other than random forest is the support vector machine.
| IC50 Cutoff | 0.1 μM | 1 μM | 10 μM |
|---|---|---|---|
| Accuracy (%) | 93 ± 2 (93; 93) | 87 ± 1 (82; 87) | 82 ± 1 (80; 82) |
| Sensitivity (%) | 88 ± 4 | 87 ± 2 | 80 ± 1 |
| Specificity (%) | 94 ± 2 | 87 ± 1 | 83 ± 1 |
| False positive rate (%) | 6 ± 2 | 13 ± 1 | 17 ± 1 |
| Negative predictive value (%) | 98 ± 1 (97; 98) | 97 ± 1 (92; 97) | 82 ± 1 (81; 82) |
| Cohen’s kappa statistic (κ) | 0.86 ± 0.04 (0.67; 0.86) |
0.74 ± 0.02 (0.68; 0.73) |
0.64 ± 0.02 (0.60; 0.63) |