Table 4.
Manual validation in randomly sampled labeled data (Full Model)
| Randomly Sampled Labeled Data |
||||||
|---|---|---|---|---|---|---|
| Bottom 50% |
Top 50% |
Total |
||||
| Instances (N = 130) | Tests (N = 4 678 607) | Instances (N = 130) | Tests (N = 136 643 970) | Instances (N = 260) | Tests (N = 141 322 577) | |
| Total Correct | 81 (62.3%) | 3 801 382 (81.3%) | 126 (96.9%) | 131 790 613 (96.4%) | 207 (79.6%) | 135 591 995 (95.9%) |
| Concordant Correct | 71 (54.6%) | 3 763 546 (80.4%) | 124 (95.4%) | 129 207 143 (94.6%) | 195 (75%) | 132 970 689 (94.1%) |
| Discordant Predicted Correct | 7 (5.4%) | 37 612 (0.8%) | 1 (0.8%) | 1 565 720 (1.1%) | 8 (3.1%) | 1 603 332 (1.1%) |
| No LOINC Coverage, Code Synonymous | 3 (2.3%) | 224 (<0.1%) | 1 (0.8%) | 1 017 750 (0.7%) | 4 (1.5%) | 1 017 974 (0.7%) |
| Total Incorrect | 31 (23.8%) | 876 859 (18.7%) | 4 (3.1%) | 4 853 357 (3.6%) | 35 (13.5%) | 5 730 216 (4.1%) |
| Concordant Incorrect | 25 (19.2%) | 876 829 (18.7%) | 3 (2.3%) | 2 782 119 (2.0%) | 28 (10.8%) | 3 658 948 (2.6%) |
| Discordant Original Correct | 1 (0.8%) | 1 (<0.1%) | 1 (0.8%) | 2 071 238 (1.5%) | 2 (0.8%) | 2 071 239 (1.5%) |
| Discordant Neither Correct | 1 (0.8%) | 15 (<0.1%) | 0 (0%) | 0 (0%) | 1 (0.4%) | 15 (<0.1%) |
| No LOINC Coverage, Code Incorrect | 4 (3.1%) | 14 (<0.1%) | 0 (0%) | 0 (0%) | 4 (1.5%) | 14 (<0.1%) |
| Insufficient or Conflicting Information | 18 (13.8%) | 366 (<0.1%) | 0 (0%) | 0 (0%) | 18 (6.9%) | 366 (<0.1%) |
Full Model refers to the 1-versus-rest classifier fit to the full labeled dataset.
Label Definitions: Concordant Correct: model-predicted label = original label and is correct; Discordant Predicted Correct: model-predicted label ≠ original label, and model-predicted label is correct; No LOINC Coverage, Code Synonymous: LOINC code does not exist for the combination of test and specimen type in the source data, but the predicted LOINC code is the most reasonable alternative; Concordant Incorrect: model-predicted label = original label and is incorrect; Discordant Original Correct: model-predicted label ≠ original label, and original label is correct; Discordant Neither Correct: model-predicted label ≠ original label, and neither label is correct; No LOINC Coverage, Code Incorrect: LOINC code does not exist for the combination of test and specimen type in the source data, and the predicted LOINC code is not a reasonable alternative; Insufficient or Conflicting Information: either not enough source data to infer code (ie, units missing and would be necessary to assign code), or source data conflicts (ie, test name includes the word “blood” and specimen type is “urine”).