Table 5.
ASAP Cross-Prompt Scoring Performance in Terms of QWK (and Comparing Cross-Prompt and Within-Prompt Performance)
| Content | Organization | Word choice | Sentence fluency | Conventions | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Test: ASAP 1 | Test: ASAP 2 | Test: ASAP 1 | Test: ASAP 2 | Test: ASAP 1 | Test: ASAP 2 | Test: ASAP 1 | Test: ASAP 2 | Test: ASAP 1 | Test: ASAP 2 | |
| Training: ASAP 1 | ||||||||||
| Features | 0.69 | 0.56 (-0.13) | 0.66 | 0.52 (-0.14) | 0.69 | 0.55 (-0.14) | 0.65 | 0.56 (-0.09) | 0.64 | 0.54 (-0.10) |
| DistilBERT | 0.71 | 0.60 (-0.11) | 0.67 | .56 (-0.09) | 0.68 | .58 (-0.10) | 0.68 | 0.62 (-0.16) | 0.67 | 0.56 (-0.11) |
| Hybrid | 0.74 | .61 (-0.13) | 0.67 | 0.50 (-0.17) | 0.67 | 0.51 (-0.16) | 0.68 | 0.63 (-0.05) | 0.65 | 0.56 (-0.09) |
| Test: ASAP 2 | Test: ASAP 1 | Test: ASAP 2 | Test: ASAP 1 | Test: ASAP 2 | Test: ASAP 1 | Test: ASAP 2 | Test: ASAP 1 | Test: ASAP 2 | Test: ASAP 1 | |
| Training: ASAP 2 | ||||||||||
| Features | 0.66 | 0.54 (-0.12) | 0.66 | 0.51 (-0.15) | 0.70 | 0.54 (-0.16) | 0.69 | 0.54 (-0.15) | 0.70 | 0.50 (-0.20) |
| DistilBERT | 0.65 | 0.43 (-0.22) | 0.59 | 0.36 (-0.23) | 0.69 | 0.46 (-0.23) | 0.67 | 0.50 (-0.27) | 0.69 | 0.49 (-0.20) |
| Hybrid | 0.69 | 0.43 (-0.26) | 0.69 | 0.45 (-0.24) | 0.72 | 0.48 (-0.24) | 0.74 | 0.51 (-0.23) | 0.69 | .51 (-0.18) |
Differences between cross-prompt and within-prompt performance are represented in brackets (QWK)