Table 1.
Inter-rater scores for validation of the quality of the data.
| Label | FK P1 | FK P2 | IoU P1 | IoU P2 |
|---|---|---|---|---|
| Airway wall thickening (P1) | 0.03 ± 0.20 | 0.28 (n = 4) | ||
| Atelectasis (P1, P2) | 0.34 ± 0.05 | 0.25 ± 0.08 | 0.29 ± 0.03 (n = 33) | 0.37 ± 0.04 (n = 19) |
| Consolidation (P1, P2) | 0.36 ± 0.07 | 0.50 ± 0.07 | 0.36 ± 0.04 (n = 25) | 0.37 ± 0.04 (n = 17) |
| Emphysema (P1) & High lung volume/emphysema (P2) | 0.26 ± 0.32 | 0.10 ± 0.34 | 0.43 (n = 3) | 0.53 (n = 2) |
| Enlarged cardiac silhouette (P1, P2) | 0.56 ± 0.07 | 0.55 ± 0.08 | 0.75 ± 0.02 (n = 20) | 0.75 ± 0.02 (n = 20) |
| Enlarged hilum (P2) | −0.03 ± 0.36 | Undefined | ||
| Fibrosis (P1) & Interstitial lung disease (P2) | 0.29 ± 0.44 | 0.16 ± 0.57 | 0.53 (n = 1) | 0.23 (n = 1) |
| Fracture (P1) & Acute fracture (P2) | 0.25 ± 0.25 | 0.71 ± 0.36 | 0.42 (n = 4) | 0.21 (n = 1) |
| Groundglass opacity (P1, P2) | 0.10 ± 0.17 | 0.21 ± 0.11 | 0.42 (n = 6) | 0.38 ± 0.05 (n = 16) |
| Hiatal hernia (P2) | Undefined | Undefined | ||
| Mass (P1) | −0.01 ± 0.70 | Undefined | ||
| Nodule (P1) | 0.29 ± 0.25 | 0.38 (n = 2) | ||
| Lung nodule or mass (P2) | 0.36 ± 0.49 | 0.83 (n = 1) | ||
| Pleural abnormality (P2) | 0.67 ± 0.07 | 0.27 ± 0.02 (n = 18) | ||
| Pleural effusion (P1) | 0.53 ± 0.06 | 0.37 ± 0.03 (n = 22) | ||
| Pleural thickening (P1) | 0.06 ± 0.40 | 0.29 (n = 1) | ||
| Pneumothorax (P1, P2) | 0.55 ± 0.25 | 0.62 ± 0.28 | 0.41 (n = 3) | 0.27 (n = 3) |
| Pulmonary edema (P1, P2) | 0.22 ± 0.13 | 0.10 ± 0.14 | 0.25 ± 0.03 (n = 11) | 0.36 ± 0.06 (n = 11) |
| Quality issue (P1) | 0.02 ± 0.30 | |||
| Support devices (P1, P2) | 0.77 ± 0.05 | 0.47 ± 0.06 | ||
| Wide mediastinum (P1) & Abnormal mediastinal contour (P2) | 0.23 ± 0.34 | 0.05 ± 0.25 | 0.57 (n = 2) | 0.58 (n = 3) |
For phases 1 (P1) and 2 (P2), we present reliability on image-level labels, calculated using Fleiss’ Kappa (FK), and average IoU of the abnormality ellipses. All scores are paired with standard errors. The number of samples given for the IoU values represents the number of independent CXRs used in the calculation. The phases in which each label was present is listed in parenthesis. Table cells are left blank for labels that were not present in a specific phase.