Table 3.
Agreement Matrix for Mention and Use.
| Metric | Mention (%) | Use (%) |
|---|---|---|
| Concurrence Rate (LLM vs Human Majority) | 76.0 | 56.0 |
| Plausibility Rate (LLM vs Minimum 1 Rater) | 80.0 | 82.0 |
| Pairwise Agreement - Raters 1 and 2 | 98.0 | 68.0 |
| Pairwise Agreement - Raters 1 and 3 | 94.0 | 74.0 |
| Pairwise Agreement - Raters 2 and 3 | 92.0 | 74.0 |
| Overall Human Pairwise Agreement | 94.7 | 72.0 |
Note: Concurrence rate is the proportion of publication-dataset pairs in which the LLM classification matched the human majority vote (yes or no for mention; yes or no for use). Plausibility rate is defined as agreement with at least one of the three reviewers.