Table 3.
Model performance metrics and training and validation scores for the predictive models
| Model | Accuracy | AUC | Specificity | Sensitivity | NPV | PPV | Train score | Validation, median (IQR) |
|---|---|---|---|---|---|---|---|---|
| MAL-AOU | ||||||||
| 18 features | ||||||||
| LR | 0.72 | 0.74 | 0.56 | 0.81 | 0.63 | 0.76 | 0.66 | 0.60 (0.19) |
| KNN | 0.56 | 0.48 | 0.33 | 0.69 | 0.38 | 0.65 | 0.75 | 0.60 (0.17) |
| SVM | 0.60 | 0.56 | 0.33 | 0.75 | 0.43 | 0.67 | 0.86 | 0.63 (0.10) |
| RF | 0.68 | 0.76 | 0.44 | 0.88 | 0.67 | 0.74 | 1.00 | 0.65 (0.18) |
| 6 features | ||||||||
| LR | 0.68 | 0.74 | 0.78 | 0.63 | 0.54 | 0.83 | 0.59 | 0.47 (0.28) |
| KNN | 0.52 | 0.66 | 0.44 | 0.56 | 0.36 | 0.64 | 1.00 | 0.65 (0.18) |
| SVM | 0.60 | 0.69 | 0.56 | 0.63 | 0.45 | 0.71 | 0.89 | 0.65 (0.25) |
| RF | 0.72 | 0.80 | 0.67 | 0.75 | 0.60 | 0.80 | 1.00 | 0.70 (0.20) |
| 5 features | ||||||||
| LR | 0.60 | 0.69 | 0.78 | 0.50 | 0.47 | 0.80 | 0.57 | 0.58 (0.25) |
| KNN | 0.60 | 0.56 | 0.44 | 0.69 | 0.44 | 0.69 | 1.00 | 0.60 (0.16) |
| SVM | 0.52 | 0.77 | 0.78 | 0.38 | 0.41 | 0.75 | 0.57 | 0.50 (0.18) |
| RF | 0.64 | 0.69 | 0.33 | 0.81 | 0.50 | 0.68 | 1.00 | 0.70 (0.09) |
| 4 features | ||||||||
| LR | 0.60 | 0.70 | 0.67 | 0.56 | 0.46 | 0.75 | 0.58 | 0.53 (0.28) |
| KNN | 0.56 | 0.60 | 0.33 | 0.69 | 0.38 | 0.65 | 1.00 | 0.65 (0.19) |
| SVM | 0.64 | 0.74 | 0.67 | 0.63 | 0.50 | 0.77 | 0.61 | 0.60 (0.15) |
| RF | 0.64 | 0.64 | 0.44 | 0.75 | 0.50 | 0.71 | 1.00 | 0.74 (0.18) |
| MAL-QOM | ||||||||
| 18 features | ||||||||
| LR | 0.76 | 0.81 | 0.67 | 0.81 | 0.67 | 0.81 | 0.75 | 0.68 (0.10) |
| KNN | 0.72 | 0.78 | 0.56 | 0.81 | 0.63 | 0.76 | 0.78 | 0.50 (0.19) |
| SVM | 0.76 | 0.83 | 0.78 | 0.75 | 0.64 | 0.86 | 0.77 | 0.68 (0.10) |
| RF | 0.76 | 0.83 | 0.67 | 0.81 | 0.67 | 0.81 | 1.00 | 0.50 (0.19) |
| 6 features | ||||||||
| LR | 0.60 | 0.71 | 0.67 | 0.56 | 0.46 | 0.75 | 0.63 | 0.60 (0.26) |
| KNN | 0.52 | 0.57 | 0.56 | 0.50 | 0.38 | 0.67 | 0.73 | 0.65 (0.10) |
| SVM | 0.64 | 0.49 | 0.33 | 0.81 | 0.50 | 0.68 | 0.97 | 0.60 (0.10) |
| RF | 0.52 | 0.67 | 0.67 | 0.50 | 0.43 | 0.73 | 0.81 | 0.60 (0.19) |
| 5 features | ||||||||
| LR | 0.60 | 0.72 | 0.67 | 0.56 | 0.46 | 0.75 | 0.61 | 0.60 (0.15) |
| KNN | 0.76 | 0.75 | 0.56 | 0.88 | 0.71 | 0.78 | 1.00 | 0.60 (0.26) |
| SVM | 0.52 | 0.62 | 0.67 | 0.44 | 0.40 | 0.70 | 0.81 | 0.60 (0.26) |
| RF | 0.60 | 0.71 | 0.56 | 0.63 | 0.45 | 0.71 | 0.85 | 0.70 (0.16) |
| 4 features | ||||||||
| LR | 0.56 | 0.72 | 0.56 | 0.56 | 0.42 | 0.69 | 0.67 | 0.60 (0.19) |
| KNN | 0.60 | 0.62 | 0.33 | 0.75 | 0.43 | 0.67 | 1.00 | 0.70 (0.23) |
| SVM | 0.60 | 0.71 | 0.67 | 0.50 | 0.43 | 0.73 | 0.77 | 0.70 (0.10) |
| RF | 0.72 | 0.75 | 0.67 | 0.75 | 0.60 | 0.80 | 0.99 | 0.70 (0.21) |
| NEADL | ||||||||
| 18 features | ||||||||
| LR | 0.56 | 0.57 | 0.69 | 0.33 | 0.65 | 0.38 | 0.62 | 0.50 (0.26) |
| KNN | 0.52 | 0.41 | 0.63 | 0.33 | 0.63 | 0.33 | 0.97 | 0.60 (0.25) |
| SVM | 0.60 | 0.65 | 0.94 | 0.00 | 0.63 | 0.00 | 0.67 | 0.65 (0.10) |
| RF | 0.76 | 0.81 | 0.75 | 0.78 | 0.86 | 0.64 | 0.81 | 0.70 (0.16) |
| 6 features | ||||||||
| LR | 0.52 | 0.57 | 0.63 | 0.33 | 0.63 | 0.33 | 0.60 | 0.65 (0.20) |
| KNN | 0.56 | 0.48 | 0.63 | 0.44 | 0.67 | 0.40 | 0.95 | 0.60 (0.20) |
| SVM | 0.64 | 0.62 | 0.50 | 0.89 | 0.89 | 0.50 | 0.55 | 0.70 (0.18) |
| RF | 0.72 | 0.85 | 0.75 | 0.67 | 0.80 | 0.60 | 0.80 | 0.70 (0.16) |
| 5 features | ||||||||
| LR | 0.64 | 0.72 | 0.75 | 0.44 | 0.71 | 0.50 | 0.62 | 0.65 (0.20) |
| KNN | 0.64 | 0.63 | 0.69 | 0.56 | 0.73 | 0.50 | 0.94 | 0.60 (0.10) |
| SVM | 0.64 | 0.76 | 1.00 | 0.00 | 0.64 | 0.00 | 0.70 | 0.60 (0.09) |
| RF | 0.68 | 0.82 | 0.75 | 0.56 | 0.75 | 0.56 | 0.86 | 0.68 (0.18) |
| 4 features | ||||||||
| LR | 0.64 | 0.72 | 0.75 | 0.44 | 0.71 | 0.50 | 0.65 | 0.60 (0.20) |
| KNN | 0.68 | 0.71 | 0.75 | 0.56 | 0.75 | 0.56 | 0.93 | 0.68 (0.20) |
| SVM | 0.60 | 0.70 | 0.63 | 0.56 | 0.71 | 0.45 | 0.62 | 0.60 (0.28) |
| RF | 0.76 | 0.87 | 0.75 | 0.78 | 0.86 | 0.64 | 0.80 | 0.70 (0.18) |
| SIS-ADL | ||||||||
| 18 features | ||||||||
| LR | 0.92 | 0.98 | 0.94 | 0.86 | 0.94 | 0.86 | 0.98 | 0.90 (0.08) |
| KNN | 0.80 | 0.75 | 0.94 | 0.43 | 0.81 | 0.75 | 0.96 | 0.68 (0.10) |
| SVM | 0.96 | 0.96 | 1.00 | 0.86 | 0.95 | 1.00 | 0.95 | 0.90 (0.15) |
| RF | 0.68 | 0.76 | 0.83 | 0.29 | 0.75 | 0.40 | 1.00 | 0.70 (0.09) |
| 6 features | ||||||||
| LR | 0.72 | 0.80 | 0.83 | 0.43 | 0.79 | 0.50 | 0.77 | 0.75 (0.27) |
| KNN | 0.72 | 0.77 | 0.72 | 0.71 | 0.87 | 0.50 | 0.78 | 0.70 (0.06) |
| SVM | 0.76 | 0.82 | 0.83 | 0.57 | 0.83 | 0.57 | 0.77 | 0.70 (0.18) |
| RF | 0.68 | 0.72 | 0.72 | 0.57 | 0.81 | 0.44 | 0.87 | 0.65 (0.19) |
| 5 features | ||||||||
| LR | 0.80 | 0.81 | 0.89 | 0.57 | 0.84 | 0.67 | 0.75 | 0.65 (0.19) |
| KNN | 0.76 | 0.76 | 0.83 | 0.57 | 0.83 | 0.57 | 0.73 | 0.65 (0.10) |
| SVM | 0.84 | 0.92 | 0.83 | 0.86 | 0.94 | 0.67 | 0.72 | 0.70 (0.13) |
| RF | 0.68 | 0.74 | 0.78 | 0.43 | 0.78 | 0.43 | 0.80 | 0.65 (0.16) |
| 4 features | ||||||||
| LR | 0.76 | 0.87 | 0.83 | 0.57 | 0.83 | 0.57 | 0.75 | 0.70 (0.09) |
| KNN | 0.68 | 0.69 | 0.78 | 0.43 | 0.78 | 0.43 | 0.82 | 0.70 (0.08) |
| SVM | 0.72 | 0.88 | 0.67 | 0.86 | 0.92 | 0.50 | 0.72 | 0.74 (0.20) |
| RF | 0.64 | 0.72 | 0.72 | 0.43 | 0.76 | 0.38 | 0.90 | 0.70 (0.19) |
IQR interquartile range, MAL Motor Activity Log, AOU Amount of Use, QOM Quality of Movement, NEADL Nottingham Extended Activities of Daily Living, SIS-ADL Stroke Impact Scale Activities of Daily Living domain, LR logistic regression, KNN k-nearest neighbors, SVM support vector machine, RF random forest, AUC area under the receiver operating characteristic curve, NPV negative predictive value, PPV positive predictive value