Table 4.
The Accuracy of the Human-Machine Classification System: Implementation of a Strategic Filtera Based on Agreement of Predictions Between Selected Combinations of Different Algorithms (Naïve Bayes Single Word, Naïve Bayes Bi-gram, SVM, Logistic Regression)
| Two Model Agreement | Three Model Agreement | ||||||
|---|---|---|---|---|---|---|---|
|
| |||||||
| Models | SVM= Naïve Bayes Single Word | SVM = Naïve Bayes Bi-gram | SVM= Logistic | Logistic= Naïve Bayes Single Word | Logistic= Naïve Bayes Bi-gram | SVM = Naïve Bayes Single Word =Logistic | SVM=Naïve Bayes Single Word = Naïve Bayes Bi-gram |
| Overall | |||||||
| Sensitivity/PPV | 87% | 89% | 81% | 86% | 88% | 89% | 93% |
| Manual Coded | 28% | 33% | 14% | 24% | 29% | 31% | 43% |
a
A filter is a technique to decide which narratives the computer should classify vs. which should be left for a human to read and classify