Fig. 2 |. Machine learning models and manual reviewers use similar features when making manual review classification decisions.

Features ranked as important by random forest and deep learning models were also ranked highly by experienced manual reviewers (n = 71 features). Human manual reviewer feature importance was determined by asking seven individuals to rank feature importance. Single feature impact for the deep learning model was obtained by training a model on the training set (n = 27,470 variants), shuffling each feature individually, and calculating the mean ROC AUC for all three variant classes. The change in mean ROC AUC for all classes was sorted and plotted. Random forest feature importance was obtained via scikit-learn’s feature importance parameter. All feature importance metrics were ranked normalized. The random forest feature importance is moderately correlated to the deep learning and manual reviewer feature importance (Pearson r = 0.47 and 0.50, respectively). The deep learning importance was weakly correlated with manual reviewer survey results (Pearson r = 0.17). The top 30 (of 71) most important features are shown.