Table 5:
Results of behavioral testing using CheckList. The example test cases are those in which at least one model failed. Model abbreviations - RF: Random Forest, XGB: XGBoost, BCB: Bio-ClinicalBERT. Test abbreviations - N: Negation, A: Attribution, HP: Historical Phrases, M: (Robustness to) Misspellings
| SBDH | Test | Failure Rate | Example Test Cases, Label | ||
|---|---|---|---|---|---|
| RF | XGB | BCB | |||
| Community | N | 40.5 | 27.2 | 44.5 | Husband died recently. False |
| (Present) | M | 1.9 | 3.8 | 1.9 | Patient ilves with her grandson. True |
| Community | N | 11.0 | 14.5 | 13.3 | Patient lost her child in an accident. True |
| (Absent) | M | 2.5 | 1.7 | 2.5 | Patient is separtaed from his wife. True |
| Education | A | 83.7 | 73.5 | 70.9 | Daughter is a medical student. False |
| M | 6.5 | 7.7 | 13.1 | Son is a honors high school studnet. False | |
| Economics | N | 11.6 | 15.2 | 8.0 | Former ICU nurse. False |
| A | 76.5 | 71.5 | 9.0 | He is not working. Daughter-in-law works as chemist. False | |
| M | 11.2 | 11.2 | 12.2 | He is a retierd dentist. False | |
| Environment | N | 24.7 | 15.6 | 1.3 | Intermittently lives with friends but currently homeless. False |
| A | 31.2 | 11.7 | 1.3 | Homeless last 3 months. His family lives nearby. False | |
| M | 9.1 | 10.4 | 7.8 | Homeelss, searching for apt. False | |
| Alcohol Use | N | 28.0 | 11.1 | 27.1 | Negative for alcohol use. Never |
| A | 48.0 | 53.5 | 43.0 | Negative for ETOH. Son has history of alcohol abuse. Never | |
| HP | 68.3 | 68.3 | 77.8 | Was sober for a few months, but started drinking again. Present | |
| M | 20.0 | 22.0 | 18.0 | He dneies any history of alcohol intake. Never | |
| Tobacco Use | N | 10.0 | 11.7 | 0.6 | Cigs: none ETOH: none. Never |
| A | 27.5 | 39.0 | 17.5 | Denied smoking, EtOH or drugs. Wife is a smoker. Never | |
| HP | 73.9 | 49.7 | 46.4 | Denies recent tobacco (history of abuse but none in years) Past | |
| M | 8.5 | 19.0 | 12.4 | Unclear when he started msoking again. Never | |
| Drug Use | N | 0.6 | 0.6 | 4.3 | Negative for smoking and drug. Never |
| A | 87.0 | 88.0 | 6.0 | Denied smoking, EtOH or drugs. Wife uses marijuana. Never | |
| HP | 58.8 | 68.6 | 98.7 | Had stopped opiates. Started again 1 month ago. Present | |
| M | 9.2 | 17.0 | 3.3 | Hx of IVDU abuse ubt quit. Past | |