Table 4.
Baseline experiment results. The results for phrase-based and section header-based methods were obtained from all 50 articles, and the results for linear SVM and BioBERT-based models were obtained using 5-fold cross-validation. Best results for each CONSORT item are in bold. 3a: Trial Design; 3b: Changes to Trial Design; 4a: Eligibility Criteria; 4b: Data Collection Setting; 5: Interventions; 6a: Outcomes; 6b: Changes to Outcomes; 7a: Sample Size Determination; 7b: Interim Analyses/ Stopping Guidelines; 8a: Sequence Generation; 8b: Randomization Type; 9: Allocation Concealment; 10: Randomization Implementation; 11a: Blinding Procedure; 11b: Similarity of Interventions; 12a: Statistical Methods for Outcomes; 12b: Statistical Methods for Other Analyses; Micro: Micro-averaging; Macro: Macro-averaging. AUC: Area Under Receiver Operator Characteristic (ROC) Curve. In the last column, each letter indicates that the results of one method is statistically significantly different from those of another method at 95% confidence level, as measured by McNemar’s test (a: phrase-based vs. section header-based; b: phrase-based vs. linear SVM; c: phrase-based vs. BioBERT; d: section-header based vs. linear SVM; e: section-header based vs. BioBERT; f: linear SVM vs. BioBERT).
| Item | Phrase-based | Section header-based | Linear SVM | BioBERT | Statistical significance (p < 0.05) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Prec. | Recall | F1 | Prec. | Recall | F1 | Prec. | Recall | F1 | Prec. | Recall | F1 | ||
| 3a | 0.71 | 0.55 | 0.62 | 0.14 | 0.75 | 0.23 | 0.70 | 0.58 | 0.62 | 0.93 | 0.49 | 0.63 | abcdef |
| 3b | 1.00 | 0.20 | 0.33 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | |
| 4a | 0.62 | 0.68 | 0.65 | 0.40 | 0.88 | 0.54 | 0.87 | 0.60 | 0.70 | 0.90 | 0.82 | 0.85 | abcef |
| 4b | 0.34 | 0.44 | 0.38 | 0.03 | 0.03 | 0.03 | 0.88 | 0.46 | 0.59 | 0.80 | 0.24 | 0.36 | ef |
| 5 | 0.48 | 0.49 | 0.48 | 0.66 | 0.35 | 0.46 | 0.66 | 0.50 | 0.56 | 0.76 | 0.69 | 0.72 | abcdef |
| 6a | 0.57 | 0.34 | 0.42 | 0.86 | 0.48 | 0.62 | 0.74 | 0.64 | 0.69 | 0.84 | 0.78 | 0.81 | abcde |
| 6b | 0.50 | 0.17 | 0.25 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | |
| 7a | 0.85 | 0.65 | 0.74 | 0.51 | 0.39 | 0.44 | 0.93 | 0.70 | 0.79 | 0.88 | 0.80 | 0.84 | abcdef |
| 7b | 0.18 | 0.88 | 0.29 | 0.00 | 0.00 | 0.00 | 0.80 | 0.64 | 0.70 | 0.00 | 0.00 | 0.00 | abcdf |
| 8a | 0.71 | 0.63 | 0.67 | 0.00 | 0.00 | 0.00 | 0.92 | 0.64 | 0.74 | 0.86 | 0.26 | 0.38 | abcdef |
| 8b | 0.55 | 0.71 | 0.62 | 0.15 | 0.53 | 0.23 | 0.67 | 0.46 | 0.54 | 0.71 | 0.29 | 0.38 | abcdef |
| 9 | 0.86 | 0.27 | 0.41 | 0.00 | 0.00 | 0.00 | 0.28 | 0.19 | 0.22 | 0.00 | 0.00 | 0.00 | bdf |
| 10 | 0.63 | 0.18 | 0.27 | 0.24 | 0.72 | 0.35 | 0.68 | 0.25 | 0.36 | 0.72 | 0.15 | 0.24 | abcde |
| 11a | 0.53 | 0.58 | 0.55 | 0.21 | 0.28 | 0.24 | 0.84 | 0.45 | 0.58 | 0.77 | 0.29 | 0.42 | abcef |
| 11b | 0.36 | 0.56 | 0.43 | 0.00 | 0.00 | 0.00 | 0.20 | 0.13 | 0.16 | 0.00 | 0.00 | 0.00 | |
| 12a | 0.52 | 0.52 | 0.52 | 0.38 | 0.99 | 0.55 | 0.72 | 0.64 | 0.67 | 0.75 | 0.76 | 0.75 | abcf |
| 12b | 0.35 | 0.24 | 0.28 | 0.00 | 0.00 | 0.00 | 0.32 | 0.13 | 0.17 | 0.05 | 0.03 | 0.04 | adf |
| Micro | 0.54 | 0.46 | 0.50 | 0.40 | 0.52 | 0.45 | 0.74 | 0.56 | 0.64 | 0.82 | 0.63 | 0.72 | |
| Macro | 0.57 | 0.47 | 0.47 | 0.21 | 0.32 | 0.22 | 0.60 | 0.41 | 0.48 | 0.52 | 0.33 | 0.38 | |
| AUC | - | - | 0.774 | 0.812 | |||||||||