Figure 2. Receiver Operating Characteristic (ROC) Curves for Identifying Clinically Relevant Bleeding From Clinical Notes Using a Rules-Based Approach.

We evaluated 3 natural language processing–derived parameters to identify notes with clinically relevant bleeding in the training data for the total number of mentions (A), the number of bleeding-present mentions (B), and the number of bleeding-absent mentions (C). At least 1 absent or present reference to bleeding, identified by the algorithm, had almost 100% sensitivity for identifying notes with clinically relevant bleeding, but had poor specificity. At least 1 present reference to bleeding was 93% sensitive for identifying notes with clinically relevant bleeding, with greater than 70% specificity.