Table 2.

Diagnostic performance of the final ensembled model and the 9 basic version convolutional neural network models with test time augmentation

Model	Accuracy	Sensitivity	Specificity	PPV	NPV	AUC	F1 (avg)	κ value
VGG19	0.842	0.835	0.850	0.860	0.823	0.900	0.842	0.684
VGG19 (TTA)	0.851	0.845	0.858	0.868	0.833	0.918	0.851	0.702
ResNet18	0.850	0.846	0.856	0.867	0.833	0.917	0.850	0.700
ResNet18 (TTA)	0.865	0.867	0.862	0.875	0.854	0.928	0.864	0.729
ResNet50	0.860	0.855	0.865	0.875	0.843	0.922	0.859	0.719
ResNet50 (TTA)	0.870	0.867	0.874	0.884	0.856	0.931	0.870	0.740
ResNet152	0.864	0.859	0.868	0.879	0.848	0.926	0.863	0.727
ResNet152 (TTA)	0.874	0.871	0.878	0.888	0.859	0.932	0.874	0.748
DenseNet169	0.860	0.856	0.865	0.875	0.844	0.923	0.860	0.720
DenseNet169 (TTA)	0.871	0.863	0.879	0.888	0.852	0.931	0.870	0.741
DenseNet264	0.866	0.860	0.874	0.883	0.849	0.930	0.866	0.732
DenseNet264 (TTA)	0.876	0.867	0.886	0.894	0.857	0.932	0.876	0.752
EfficientNet-b0	0.864	0.874	0.853	0.868	0.859	0.924	0.864	0.727
EfficientNet-b0 (TTA)	0.874	0.879	0.869	0.882	0.867	0.933	0.874	0.748
EfficientNet-b4	0.870	0.879	0.860	0.875	0.865	0.930	0.870	0.739
EfficientNet-b4 (TTA)	0.878	0.882	0.874	0.886	0.870	0.935	0.878	0.756
EfficientNet-b7	0.874	0.880	0.868	0.881	0.867	0.933	0.874	0.748
EfficientNet-b7 (TTA)	0.881	0.885	0.877	0.889	0.873	0.937	0.881	0.762
Ensemble model	0.889	0.887	0.892	0.901	0.877	0.938	0.889	0.778
Ensemble (TTA) model	0.892	0.890	0.895	0.904	0.880	0.940	0.892	0.784

Abbreviations: DenseNet, Dense Nework; EfficientNet, Efficient Network; PPV, positive predictive value; NPV, negative predictive value; AUC, area under the curve; κ value, the Fleiss’s κ value; ResNet, Residual Network; TTA, test time augmentation; VGG, Visual Geometry Group Network.