Table 4.

Performance of constraint type algorithm

NLP labels⟍Gold standard	PS	CR	Other	Keywords
Posterior Stabilized (PS)	223	0	0	PS,P/S, stabilized posterior, posterior stabilized, tib ins stab, tib insert stab, posterior stabilized, posterior cruciate was excised, cruciate holes made, posterior cruciate substituted, poly stab, post stab, PCL sacrificed, cruciate substituted, box cuts, post cruciate substituting, PCL excised, cruciate ligaments excised, cruciate ligaments removed, posterior stabilized insert, rotating platform
Cruciate-retaining (CR)	0	138	2	CR, C/R, cruciate retaining, fixed bearing insert, PCL retained, posterior cruciate was intact and retained, cruciate retaining femoral component, cruc ret
Other	0	1	48	CCK (CCK, LCCK, LC/CK, constrained condylar femoral component), MC (Mc, medial congruent liner), UC (UC), and the notes that annotators and NLP didn’t find any clue to determine their categories
Accuracy = 99.2%
Rules: Review all sections of a note:  If finds any of the relative keyword to each PS/CR/Other categories, assigns the relative label.  Prioritize the labels which are found in the “Implant Name” and “Implant Placement”  Prioritize the labels by the order of: CCK, MC, UC, CR, PS

Abbreviations: Ultra-congruent (UC), medial congruent (MC), constrained condylar knee (CCK)