Table 1:
CNN performance in classifying DSAs based on their level of reperfusion based on the mTICI scale. Performance is displayed in the form of average accuracies, area under the receiver operating characteristic curves (AUROC), Matthews correlation coefficients (MCC), sensitivities, and specificities along with their standard deviations and 95% confidence intervals (CI). (A) Two-outcome classification (mTICI grade 0,1,2a versus mTICI grade 2b,2c,3) and (B) Three-outcome classification (mTICI grade 0,1,2a versus mTICI grade 2b versus mTICI grade 2c,3). The 3-outcome classification requires a ROC curve for each outcome, thus there is an AUROC for each outcome in (B). The best results are in bold. The results indicate that best performance is achieved when making a 2-outcome classification using an ensembled network.
| Metric | (A) 2 Class: mTICI 0,1,2a versus mTICI 2b,2c,3 | ||
|---|---|---|---|
| Only AP | Only Lateral | AP and Lateral Ensembled | |
| Accuracy (%) | 74.2 ± 3.3 (72.8, 75.7) | 76.9 ± 5.9 (74.4, 79.5) | 81.0 ± 4.5 (79.0, 82.9) |
| AUROC | 0.83 ± 0.04 (0.81, 0.84) | 0.84 ± 0.05 (0.82, 0.87) | 0.86 ± 0.04 (0.84, 0.88) |
| MCC | 0.49 ± 0.07 (0.46, 0.52) | 0.54 ± 0.11 (0.49, 0.59) | 0.62 ± 0.09 (0.58, 0.66) |
| Sensitivity | 0.78 ± 0.1 (0.74, 0.83) | 0.84 ± 0.1 (0.79, 0.88) | 0.88 ± 0.08 (0.85, 0.92) |
| Specificity | 0.69 ± 0.12 (0.64, 0.75) | 0.68 ± 0.14 (0.62, 0.75) | 0.72 ± 0.11 (0.67, 0.76) |
| Metric | (B) 3 Class: mTICI 0,1,2a versus mTICI 2b versus mTICI 2c,3 | ||
| Only AP | Only Lateral | AP and Lateral Ensembled | |
| Accuracy (%) | 56.0 ± 4.9 (53.9, 58.2) | 61.8 ± 4.1 (60, 63.6) | 64.0 ± 5.0 (62.0, 66.0) |
| AUROC - mTICI 0,1,2a | 0.81 ± 0.04 (0.79, 0.83) | 0.84 ± 0.05 (0.82, 0.86) | 0.85 ± 0.04 (0.83, 0.87) |
| AUROC - mTICI 2b | 0.67 ± 0.06 (0.64, 0.70) | 0.74 ± 0.06 (0.72, 0.77) | 0.74 ± 0.06 (0.71, 0.77) |
| AUROC - mTICI 2c,3 | 0.74 ± 0.05 (0.71, 0.76) | 0.75 ± 0.08 (0.72, 0.79) | 0.78 ± 0.06 (0.76, 0.81) |
| MCC | 0.30 ± 0.09 (0.26, 0.34) | 0.40 ± 0.07 (0.37, 0.43) | 0.43 ± 0.08 (0.40, 0.47) |
| Sensitivity | 0.66 ± 0.16 (0.59, 0.73) | 0.74 ± 0.08 (0.70, 0.77) | 0.80 ± 0.1 (0.75, 0.84) |
| Specificity | 0.73 ± 0.08 (0.69, 0.76) | 0.74 ± 0.08 (0.70, 0.77) | 0.76 ± 0.07 (0.73, 0.79) |