Table 2:
Full experiment models and results
| Temporal validation | External validation | |||
|---|---|---|---|---|
| AUPRC (95% CI) | AUROC (95% CI) | AUPRC (95% CI) | AUROC (95% CI) | |
| Baseline single classifiers | ||||
| Alcohol* | 0·70 (0·66–0·73) | 0·92 (0·90–0·93) | NA | NA |
| Opioid† | 0·75 (0·71–0·80) | 0·98 (0·98–0·99) | NA | NA |
| Multilabel logistic regression | ||||
| Overall | 0·65 (0·60–0·70) | 0·96 (0·94–0·97) | NA | NA |
| Alcohol | 0·73 (0·70–0·77) | 0·95 (0·94–0·96) | 0·92 (0·90–0·94) | 0·89 (0·87–0·91) |
| Opioid | 0·84 (0·80–0·87) | 0·99 (0·98–0·99) | 0·88 (0·85–0·91) | 0·90 (0·88–0·92) |
| Non-opioid | 0·39 (0·31–0·48) | 0·94 (0·91–0·96) | NA | NA |
| Multilabel deep averaging network | ||||
| Overall | 0·61 (0·56–0·66) | 0·94 (0·92–0·95) | NA | NA |
| Alcohol | 0·72 (0·68–0·76) | 0·93 (0·91–0·94) | 0·87 (0·84–0·89) | 0·79 (0·76–0·82) |
| Opioid | 0·82 (0·78–0·85) | 0·98 (0·97–0·99) | 0·75 (0·71–0·79) | 0·78 (0·75–0·82) |
| Non-opioid | 0·29 (0·22–0·37) | 0·91 (0·88–0·93) | NA | NA |
| Multilabel bag of CUIs | ||||
| Overall | 0·59 (0·55–0·64) | 0·95 (0·94–0·96) | NA | NA |
| Alcohol | 0·72 (0·68–0·76) | 0·94 (0·93–0·95) | 0·87 (0·85–0·90) | 0·83 (0·80–0·85) |
| Opioid | 0·83 (0·80–0·86) | 0·99 (0·98–0·99) | 0·84 (0·81–0·88) | 0·88 (0·86–0·91) |
| Non-opioid | 0·22 (0·17–0·29) | 0·93 (0·91–0·95) | NA | NA |
| Multilabel transformer | ||||
| Overall | 0·66 (0·61–0·71) | 0·97 (0·96–0·97) | NA | NA |
| Alcohol | 0·74 (0·70–0·77) | 0·95 (0·94–0·96) | 0·91 (0·89–0·93) | 0·88 (0·85–0·90) |
| Opioid | 0·84 (0·80–0·88) | 0·99 (0·98–0·99) | 0·87 (0·84–0·90) | 0·90 (0·88–0·92) |
| Non-opioid | 0·40 (0·32–0·49) | 0·96 (0·95–0·97) | NA | NA |
| Multilabel convolutional neural network | ||||
| Overall | 0·69 (0·64–0·74) | 0·97 (0·96–0·98) | NA | NA |
| Alcohol | 0·78 (0·75–0·82) | 0·96 (0·95–0·97) | 0·92 (0·90–0·93) | 0·88 (0·86–0·90) |
| Opioid | 0·87 (0·84–0·91) | 0·99 (0·99–0·99) | 0·91 (0·88–0·93) | 0·94 (0·92–0·95) |
| Non-opioid | 0·41 (0·34–0·50) | 0·96 (0·94–0·98) | NA | NA |
| Auxiliary task multilabel deep averaging network | ||||
| Overall | 0·64 (0·59–0·69) | 0·94 (0·93–0·97) | NA | NA |
| Alcohol | 0·70 (0·66–0·74) | 0·93 (0·92–0·94) | 0·83 (0·80–0·86) | 0·78 (0·75–0·80) |
| Opioid | 0·83 (0·79–0·86) | 0·98 (0·97–0·98) | 0·82 (0·78–0·85) | 0·85 (0·82–0·88) |
| Non-opioid | 0·40 (0·32–0·48) | 0·92 (0·90–0·95) | NA | NA |
| Auxiliary task multilabel bag of CUIs | ||||
| Overall | 0·58 (0·54–0·63) | 0·95 (0·94–0·96) | NA | NA |
| Alcohol | 0·71 (0·67–0·74) | 0·94 (0·92–0·95) | 0·87 (0·84–0·90) | 0·83 (0·81–0·86) |
| Opioid | 0·82 (0·79–0·86) | 0·98 (0·98–0·99) | 0·85 (0·81–0·88) | 0·89 (0·86–0·91) |
| Non-opioid | 0·22 (0·16–0·28) | 0·94 (0·92–0·95) | NA | NA |
| Auxiliary task multilabel transformer | ||||
| Overall | 0·67 (0·62–0·72) | 0·97 (0·95–0·98) | NA | NA |
| Alcohol | 0·75 (0·71–0·78) | 0·96 (0·94–0·97) | 0·91 (0·89–0·93) | 0·87 (0·85–0·90) |
| Opioid | 0·85 (0·81–0·88) | 0·99 (0·98–0·99) | 0·88 (0·85–0·90) | 0·90 (0·87–0·92) |
| Non-opioid | 0·43 (0·35–0·51) | 0·96 (0·94–0·97) | NA | NA |
| Auxiliary task multilabel convolutional neural network | ||||
| Overall | 0·68 (0·63–0·73) | 0·97 (0·97–0·98) | NA | NA |
| Alcohol | 0·79 (0·75–0·82) | 0·96 (0·95–0·97) | 0·92 (0·90–0·94) | 0·89 (0·87–0·91) |
| Opioid | 0·87 (0·83–0·90) | 0·99 (0·99–0·99) | 0·92 (0·90–0·94) | 0·95 (0·93–0·96) |
| Non-opioid | 0·38 (0·31–0·46) | 0·97 (0·96–0·98) | NA | NA |
Temporal validation occurred at Rush University Medical Center. External validation occurred at Loyola University Medical Center. AUPRC=under the precision-recall curve. AUROC=area under the receiver operating characteristic curve. CUI=concept unique identifiers. NA=not applicable.
*
Previously published baseline model for classifying alcohol misuse in the hospital.22
†
Previously published baseline model for classifying opioid misuse in the hospital.17