Table 3.
Problem opioid use classification algorithm performance in the 1,400-patient training set and the 600-patient validation set, for selected values of the algorithm-generated risk score with desired performance characteristics (based on training data), as measured by sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV).
| Row | Desired performance characteristic (based on training data) | Risk score cut-point | Sensitivity† |
Specificity‡ |
PPV§ |
NPV¶ |
Pred. prevalence¥ |
||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Train. | Valid. | Train. | Valid. | Train. | Valid. | Train. | Valid. | Train. | Valid. | ||||
| 1 | Sensitivity | Excellent (0.90) | 0.122 | 0.900 | 0.850 | 0.641 | 0.640 | 0.429 | 0.412 | 0.955 | 0.935 | 56% | 56% |
| 2 | Good (0.80) | 0.229 | 0.800 | 0.729 | 0.827 | 0.786 | 0.581 | 0.503 | 0.933 | 0.907 | 40% | 42% | |
| 3 | Acceptable (0.75) | 0.278 | 0.752 | 0.629 | 0.879 | 0.841 | 0.651 | 0.541 | 0.922 | 0.884 | 35% | 35% | |
| 4 | Specificity | Excellent (0.90) | 0.311 | 0.736 | 0.620 | 0.900 | 0.867 | 0.688 | 0.580 | 0.919 | 0.885 | 32% | 33% |
| 5 | Good (0.80) | 0.202 | 0.821 | 0.738 | 0.800 | 0.764 | 0.551 | 0.481 | 0.937 | 0.907 | 43% | 44% | |
| 6 | Acceptable (0.75) | 0.169 | 0.861 | 0.776 | 0.751 | 0.727 | 0.509 | 0.457 | 0.948 | 0.916 | 47% | 48% | |
| 7 | PPV | Excellent (0.90) | 0.705 | 0.356 | 0.296 | 0.988 | 0.974 | 0.900 | 0.774 | 0.837 | 0.823 | 14% | 13% |
| 8 | Good (0.80) | 0.478 | 0.545 | 0.486 | 0.959 | 0.934 | 0.800 | 0.685 | 0.876 | 0.859 | 22% | 23% | |
| 9 | Acceptable (0.75) | 0.393 | 0.629 | 0.544 | 0.937 | 0.905 | 0.750 | 0.631 | 0.894 | 0.870 | 26% | 28% | |
| 10 | Sensitivity and PPV are balanced | 0.330 | 0.706 | 0.582 | 0.911 | 0.871 | 0.703 | 0.572 | 0.912 | 0.875 | 30% | 31% | |
Sensitivity is the proportion of people correctly classified as having problem opioid use by the algorithm, defined as: Number of people identified with chart review to have problem opioid use and correctly classified by the algorithm to have problem opioid use/the number of people identified with chart review to have problem opioid use.
Specificity is the proportion of people correctly classified as not having problem opioid use by the algorithm, defined as: Number of people identified with chart review to not have problem opioid use and correctly classified by the algorithm to not have problem opioid use/the number of people identified with chart review to not have problem opioid use.
Positive predictive value is the proportion of people the algorithm classifies as having problem opioid use who have problem opioid use identified by chart review, defined as: Number of people identified with chart review to have problem opioid use and classified by the algorithm to have problem opioid use/the number of people identified to have problem opioid use by the algorithm.
Negative predictive value is the proportion of people the algorithm classifies as not having problem opioid use identified by chart review, defined as the number of people identified with chart review to not have problem opioid use and classified by the algorithm to not have problem opioid use/the number of people identified to have problem opioid use by the algorithm.
This is the unadjusted predicted prevalence, defined as the percent of patients in the training sample predicted to be problem opioid use positive using the corresponding risk score cut point. The unadjusted prevalence of problem opioid use positive patients in the training sample was 36.5% (511/1,400).