Skip to main content
. Author manuscript; available in PMC: 2018 Jan 15.
Published in final edited form as: Neuroimage. 2016 Apr 11;145(Pt B):314–328. doi: 10.1016/j.neuroimage.2016.04.003

Table 2.

Improvement of classification performance at different network depths using the ReLU activation function when the target percentage of non-zero weights was 0.001 at the first layer (i.e., this condition showed the lowest error rate). The percentage improvement in classification error of the one- to three-layer DNNs over the classification performance achieved with the zero-layer DNN (i.e., with the input layer only) are given in parentheses.

Classifier 0-layer Pretraining 1-layer 2-layer 3-layer
Softmax 10.8 without 4.9 (+54.6%) 4.5 (+58.3%) 4.0 (+63.0%)
with 4.4 (+59.3%) 3.6 (+66.7%) 3.4 (+68.5%)
SVM 4.9 without 5.0 (−2.0%) 4.2 (+14.3%) 3.6 (+26.5%)
with 5.9 (−20.0%) 3.3 (+32.7%) 3.2 (+34.7%)

DNN, deep neural network; ReLU, rectified linear unit; SVM, support vector machine.