Table 4.

Comparison of classifier performance of rule-based and machine learning classifiers (n=600), with machine learning classifier threshold set at 0.5.

Classifier and classification		Features	True positive, n (%)	False positive, n (%)	False negative, n (%)	True negative, n (%)	Sensitivity (95% CI)	Specificity (95% CI)	Accuracy (95% CI)
Rule-based
	ICD-10-CMa or CCb,c	2	220 (36.7)	5 (0.8)	64 (10.7)	311 (51.8)	0.77 (0.73-0.82)	0.98 (0.97-1.00)	0.89 (0.86-0.91)
LASSOd
	Model 1e	34	240 (40)	28 (4.7)	44 (7.3)	288 (48)	0.85 (0.80-0.89)	0.91 (0.88-0.95)	0.88 (0.85-0.91)
	Model 2f	53	239 (39.8)	30 (5)	45 (7.5)	286 (47.7)	0.84 (0.79-0.89)	0.91 (0.87-0.94)	0.87 (0.85-0.90)
	Model 3g	84	242 (40.3)	29 (4.8)	42 (7)	287 (47.8)	0.86 (0.81-0.90)	0.91 (0.88-0.94)	0.88 (0.86-0.97)
Random forest
	Model 1	34	241 (40.2)	28 (4.7)	43 (7.2)	288 (48)	0.85 (0.80-0.89)	0.91 (0.88-0.94)	0.88 (0.86-0.91)
	Model 2	53	242 (40.3)	39 (6.5)	42 (4)	277 (46.1)	0.85 (0.81-0.89)	0.88 (0.85-0.92)	0.86 (0.84-0.89)
	Model 3	84	243 (40.5)	26 (4.3)	41 (6.8)	290 (48.3)	0.86 (0.81-0.90)	0.92 (0.88-0.95)	0.88 (0.86-0.91)

^aICD-10-CM: International Classification of Diseases, Tenth Revision, Clinical Modification.

^bCC: chief complaint.

^cCC refers to suicide-related chief complaints.

^dLASSO: least absolute shrinkage and selection operator–penalized logistic regression.

^eModel 1 considered all mental health–related ICD-10-CM codes organized by Child and Adolescent Mental Health Disorders Classification System categories as well as suicide-related CCs and mental health–related CCs.

^fModel 2 considered suicide-related ICD-10-CM codes and all data elements (eg, child sociodemographics, emergency department disposition, involuntary hold status, medications, and laboratory tests) except mental health–related ICD-10-CM codes.

^gModel 3 considered all structured data elements.