Table 3.
Interrater Reliability and Validity Analysis for the Primary Measure and Sub-measures by Site
| SITE | Kappa values§ | 1Sensitivity | 2Specificity | 3PPV | 4NPV |
|---|---|---|---|---|---|
| Tobacco Screening (Sub-Measure-1) | |||||
| Site 1 308 (0.89)* | k=0.898 | 99.3% (276/278) | 100% (30/30) | 100% (276/276) | 93.8% (30/32) |
| Site 2 203 (0.91)* | k=0.898 | 100% (179/179) | 100% (24/24) | 100% (179/179) | 100% (24/24) |
| Site 3 269 (0.80)* | k=0.730 | 92.6% (213/230) | 100% (39/39) | 100% (213/213) | 69.6% (39/56) |
| Site 4 160 (1.0)* | k=1.000 | 100% (160/160) | - (0/0) | 100% (160/160) | - (0/0) |
| Tobacco Use (Sub-Measure-2) | |||||
| Site 1 235 (0.18)* | k=1.000 | 100% (38/38) | 98.7% (235/238) | 92.7% (38/41) | 100% (235/235) |
| Site 2 127 (0.95)* | k=0.788 | 66.70% (8/12) | 93.4% (156/167) | 42.1% (8/19) | 97.5% (156/160) |
| Site 3 117 (0.09)* | k=0.652 | 94.1% (16/17) | 100% (196/196) | 100% (16/16) | 99.5% (196/197) |
| Site 4 172 (0.19)* | k=0.920 | 94.1% (16/17) | 98.1% (152/155) | 84.2% (16/19) | 99.3% (152/153) |
| Tobacco Cessation Intervention (Sub-Measure-3) | |||||
| Site 1 99 (0.02)* | k=1.000 | 50% (1/2) | 100% (90/90) | 100% (1/1) | 98.9% (90/91) |
| Site 2 149 (0.28)* | k=0.677 | 100% (41/41) | 100% (17/17) | 100% (41/41) | 100% (17/17) |
| Site 3 147 (0.82)* | k=0.657 | 100% (109/109) | 96.9% (31/32) | 99.1% (109/110) | 100% (31/31) |
| Site 4 330 (0.31)* | k=1.000 | 80.3% (110/137) | 100% (182/182) | 100% (110/110) | 87.1% (182/209) |
Sensitivity – when the manual review confirmed that the query correctly identified the patient in the numerator and met the eligibility criteria for each of the sub-measures.
Specificity when the manual review confirmed that the patient, as identified by the query, was not part of the numerator.
Positive Predictive Value (PPV) – when the query confirmed that the patient was part of the numerator as identified by the manual query.
Negative Predictive Value (NPV) – when the query confirmed that the patient was only part of the denominator as defined by the manual query.
Number of charts reviewed manually as determined by sample size calculation and sample proportions (given by the queried measure score)
Kappa values calculated for the first 50 random manual review charts to assess the calibration among paired reviewers within each institution