Table 3.
Average rating and inter-rater agreement per component of RQS
| No. | RQS item | Range | Average | Median (range) | Inter-rater agreement | |
|---|---|---|---|---|---|---|
| ICC | 95%CI | |||||
| 1 | Image protocol quality | 0–2 | 0.89 | 1 (0–2) | 0.90 | 0.76–1.00 |
| 2 | Multiple segmentations | 0–1 | 0.63 | 1 (0–1) | 1.00 | 1.00–1.00 |
| 3 | Phantom study on all scanners | 0–1 | 0.05 | 0 (0–1) | 1.00 | 1.00–1.00 |
| 4 | Imaging at multiple time points | 0–1 | 0 | 0 (0–0) | 1.00 | 1.00–1.00 |
| 5 | Feature reduction or adjustment for multiple testing | − 3 to 3 | 2.79 | 3 (− 3 to 3) | 1.00 | 1.00–1.00 |
| 6 | Multivariable analysis with non-radiomics features | 0–1 | 0.72 | 1 (0–1) | 0.91 | 0.80–1.00 |
| 7 | Detect and discuss biological correlate | 0–1 | 0.23 | 0 (0–1) | 0.90 | 0.77–1.00 |
| 8 | Cutoff analyses | 0–1 | 0.13 | 0 (0–1) | 0.77 | 0.52–1.00 |
| 9 | Discrimination statistics | 0–2 | 1.42 | 0 (0.5–2) | 0.72 | 0.54–0.90 |
| 10 | Calibration statistics | 0–2 | 0.42 | 0 (0–2) | 1.00 | 1.00–1.00 |
| 11 | Prospective study registered in a trial database | 0–7 | 0.25 | 0 (0–7) | 1.00 | 1.00–1.00 |
| 12 | Validation | − 5 to 5 | 1.39 | 2 (− 5 to 5) | 0.94 | 0.85–1.00 |
| 13 | Comparison to 'gold standard' | 0–2 | 1.26 | 2 (0–2) | 0.93 | 0.82–1.00 |
| 14 | Potential clinical utility | 0–2 | 0.70 | 0 (0–2) | 1.00 | 1.00–1.00 |
| 15 | Cost-effectiveness analysis | 0–1 | 0 | 0 (0–0) | 1.00 | 1.00–1.00 |
| 16 | Open science and data | 0–4 | 0.15 | 0 (0 ≥ 2) | 0.92 | 0.78–1.00 |
| Total points: − 8 to 0 = 0%, 36 = 100% | 0–100% | 11.04 = 30.7% | 12 (− 4 to 22) | 0.80 | 0.69–0.91 | |
CI confidence interval, ICC interclass correlation coefficient, RQS Radiomics Quality Score