Table 1.
OCD identification rate of LLMs and medical and mental health professionals
| Content of OCD vignette | LLM | Group 1. APA membersa (N = 360) | Group 2. Primary care physiciansb (N = 208) | Group 3. Doctoral trainees in psychologyc (N = 130) | Group 4. Medical providers in Guamd (N = 105) | Group 5. Clergy members in Guam (N = 110) | ||
|---|---|---|---|---|---|---|---|---|
| ChatGPT-4 (N = 16) | Gemini Pro (N = 16) | Llama 3 (N = 19) | ||||||
| Across all vignettes | 96.1% (N = 49/51)e | 61.1% | 49.5% | 81.5% | 41.9% | 35.5% | ||
| 100% (n = 16/16) | 93.8% (n = 15/16) | 94.7% (n = 18/19) | ||||||
| Vignette 1. Harm obsessions | 100% (3/3 trials) | 100% (3/3 trials) | 100% (3/3 trials) | 68.5% | 20.0% | 77.8% | 18.2% | 27.8% |
| Vignette 2. Sexual orientation obsessions | 100% (3/3 trials) | 100% (2/2 trials)g | 66.7% (2/3 trials) | 23.0% | 15.4% | 66.7% | 10.0% | 6.7% |
| Vignette 3. Sexual attraction to children obsessions | No responsef | 100% (1/1 trial)h | 100% (3/3 trails) | 57.1% | 29.2% | 77.8% | 15.0% | 11.1% |
| Vignette 4. Religious obsessions | 100% (3/3 trials) | 100% (3/3 trials) | 100% (3/3 trials) | 71.2% | 62.5% | 80.0% | 72.7% | 21.7% |
| Vignette 5. Contamination obsessions | 100% (3/3 trials) | 100% (3/3 trials) | 100% (3/3 trials) | 84.2% | 67.7% | 93.7% | 83.3% | 73.3% |
| Vignette 6. Blurting out offensive language obsessions | 100% (1/1 trial) | 100% (1/1 trial) | 100% (1/1 trial) | N/A | 26.1% | 74.5% | N/A | N/A |
| Vignette 7. Somatic obsessions | 100% (1/1 trial) | 0% (0/1 trial) | 100% (1/1 trial) | N/A | 60.0% | 82.4% | N/A | N/A |
| Vignette 8. Symmetry obsessions | 100% (2/2 trials) | 100% (2/2 trials) | 100% (2/2 trials) | N/A | 96.3% | 100.0% | 85.0% | 71.4% |
aThe top five degrees/licenses of the American Psychological Association (APA) members were PhD (67.6%), MA/MS (31.5%), PsyD (14.2%), EdD/EdS/EdM (6.8%), MSW/LMSW (1.7%). Currently licensed was 81.3%.
bThe areas of specialty included Internal Medicine (35.4%), Pediatrics (32.3%), Family Medicine (22.2%), other specialties (10.6%), and General Medicine (4.5%).
cThe degrees include Clinical Psychology with Health Emphasis PhD, School-Clinical PsyD, School-Clinical PhD, Clinical Psychology PsyD, and Clinical Psychology PhD in 7 APA-accredited doctoral programs in the Greater New York area.
dGroup 4 includes medical doctors, nurse practitioners, physician assistants, and doctors of Osteopathic Medicine. The areas of specialty included Internal Medicine (33.3%), Family Medicine (26.7%), Pediatrics (16.2%), Obstetrics and Gynecology (6.7%), Emergency Medicine (4.8%), and other (12.6%).
eThe sample size differs between LLMs and mental health and health care professionals due to study design (LLM: responses from three LLM models; Human participants: responses from the wide distribution of the vignette studies).
fLLM (ChatGPT-4) did not respond to all three vignette trials due to content violation.
gLLM (Gemini Pro) did not respond to one vignette trial due to content violation.
hLLM (Gemini Pro) did not respond to two vignette trials due to content violation.