Table 2.
Summary of studies on natural language processing (NLP) and hypoglycemia.
| Author, year, country | Data source | Definition of hypoglycemia | Method used to identify hypoglycemia | NLP algorithm: rule-based or machine learning | NLP algorithm validation | Outcomes |
| Nunes et al, 2016 [31], United States | Optum Humedica EHRa database, which incorporates EHRs from 35 large medical provider organizations (including >195 hospitals), >25,000 physicians, and >25 million patients, making up the largest EHR database within the United States (January 2009 to March 2014) | Serious: ICD-9b identified events were characterized as serious or nonserious if the diagnosis was identified within a problem list; NLP-identified categories included serious (eg, serious, acute, severe, and profound); mild to moderate: NLP-identified categories included mild to moderate (eg, mild, moderate, slight, and minor) | ICD-9 algorithm (structured diagnostic codes only); NLP algorithm (NLP of clinical notes); combined algorithm (either ICD-9 diagnostic codes or NLP of clinical notes) | Rule-based | The final algorithm was validated by manual review: precision (PPVc)=0.77, recall (sensitivity)=0.67 | Period prevalence (%): any conditions: ICD-9: 12.37, NLP: 25.11, combined: 32.19; serious: ICD-9: 11.93, NLP: 10.71, combined: 18.72; mild to moderate: ICD-9: 0.00, NLP: 0.76, combined: 0.78. Incidence rate (per 100 person-years): any conditions: ICD-9: 2.25, NLP: 4.78, combined: 6.28. Serious: ICD-9: 2.12, NLP: 1.72, combined: 3.19; mild to moderate: ICD-9: 0.00, NLP: 0.09, combined: 0.08. Event rate (per 100 person-years): any conditions: ICD-9: 6.92, NLP: 10.03, combined: 16.12; serious: ICD-9: 6.63, NLP: 3.06, combined: 8.90; mild to moderate: ICD-9: 0.00, NLP: 0.20, combined: 0.19 |
| Nunes et al, 2017 [29], United States | Optum EHR database (January 2009 to December 2014) | Serious: ICDd and CPTe evidence of medical intervention or abstracted descriptors suggestive of serious event; nonserious, mild to moderate: No ICD or CPT evidence of medical intervention but with abstracted descriptors suggestive of mild to moderate event; nonserious, unspecified: no ICD or CPT evidence of medical intervention and no descriptors of event seriousness | ICD codes and NLP | Rule-based | The final algorithm was validated by manual review: precision (PPV)=0.77, recall (sensitivity)=0.67 | Incidence rate (per 100 person-years; 95% CI): any conditions: overall: 11.76 (11.49-12.04), sulfonylureas use: 12.77 (12.40-13.15), sulfonylureas nonuse: 10.39 (10.00-10.79). Serious: overall: 5.06 (4.88-5.24), sulfonylureas use: 5.77 (5.52-6.03), sulfonylureas nonuse: 4.09 (3.84-4.34). Nonserious, mild to moderate: overall: 0.14 (0.11-0.17), sulfonylureas use: 0.17 (0.13-0.22), sulfonylureas nonuse: 0.09 (0.06-0.13). Nonserious, unspecified: overall: 6.57 (6.37-6.78), sulfonylureas use: 6.83 (6.56-7.11), sulfonylureas nonuse: 6.21 (5.91-6.52) |
| Loughlin et al, 2018 [32], United States | Optum EHR database (January 2012 to January 2015) | Documented blood glucose level <3.9 mmol/L or emergency physician–charted diagnosis of hypoglycemia | Hypoglycemia and gastrointestinal symptoms (vomiting, nausea, diarrhea, or constipation) were identified by using both ICD-9 Clinical Modification diagnostic codes within structured fields and NLP clinical notes; hypoglycemia was identified using an algorithm developed by Optum, incorporated diagnostic codes, and NLP of clinical notes | Rule-based | The final algorithm was validated by manual review: precision (PPV)=0.77, recall (sensitivity)=0.67 | Incidence rate (per 1000 person-years; 95% CI): EQWf cohort: 52.5 (44.4-61.6), BIg cohort: 65.7 (59.1-72.7). Any gastrointestinal symptoms: EQW cohort: 225.5 (206.8-245.5), BI cohort: 191.0 (179.1-203.6). Participants with at least one event (n/N [%]): EQW cohort: 149/2008 (7.42), BI cohort: 368/4016 (9.16). Any gastrointestinal symptoms (n/N [%]): EQW cohort: 534/2008 (26.60), BI cohort: 946/4016 (23.56) |
| Pettus et al, 2019 [33], United States | Optum Humedica EHR database (January 1, 2007, to March 31, 2017) | Hypoglycemia: ICD-9 and ICD-10h codes for hypoglycemia; plasma glucose level measures ≤70 mg/dL; IMi glucagon administration; NLP: mention of hypoglycemia; severe hypoglycemia: ICD-9 and ICD-10 codes for hypoglycemia that is severe by default or ICD-9 and ICD-10 codes for hypoglycemia and hypoglycemia is reason for care on discharge or admission or hypoglycemia index date on same day as emergency department visit or inpatient diagnosis on admission (all related to hypoglycemic coma); plasma glucose level measures <54 mg/dL; IM glucagon administration; NLP: mention of hypoglycemia with either a descriptor of hypoglycemia severity, including severity terms (eg, severe) and attributes (eg, emergency), or emergency department visit or inpatient admission on same day as medical record was written | ICD-9 and ICD-10 codes; plasma glucose measures ≤70 mg/dL; IM glucagon administration; NLP | Rule-based | The final algorithm was validated by manual review: precision (PPV)=0.77, recall (sensitivity)=0.67 | Any hypoglycemia (%): BI switchers: 42.2-46.2. Insulin naïve: 22.8-28.8. Severe hypoglycemia: BI switchers: 8.2-17.4, insulin naïve: 2.7-8.6 |
| Li et al, 2019 [34], United States | Regenstrief Medical Record System, which is an urban safety-net medical institution in Indianapolis, Indiana, United States. In 2012, Eskenazi Health had 1081 physicians on staff and serviced 950,592 outpatient visits, including 234,637 community health center visits (January 1, 2004, to December 31, 2013) | Plasma or point-of-care glucose value of at least 5 mg/dL and <70 mg/dL, documented in the medical record; ICD-9 code: 251.1 or 251.2; ICD code 250.8 without any of the following codes: 259.8, 272.7, 681.xx, 682.x, 686.9, 707.1x, 707.2x, 707.8, 707.9, 709.3, 730.0x, 730.1x, 730.2x, 731.8; text note indicating hypoglycemia, including a blood glucose value | Laboratory tests; diagnostic codes; NLP | Rule-based | — | A 1-year window for prior episodes of hypoglycemia: overall prevalence (n/N [%]): 8182/38,780 (21); non-LAIj and sulfonylureas within 90 days (%): 42.92; sulfonylureas without insulin (%): 23.82; no insulin, no sulfonylureas (%): 17.85%; blood glucose value between 5 mg/dL and 70 mg/dL (n/N [%]): 7070/38,780 (18.23); blood glucose value<54 mg/dL (n/N [%]): 4784/38,780 (12.34); NLP (n/N [%]): 3751/38,780 (9.67), with 539/38,780 (1.39), identified only by NLP |
| Misra-Hebert et al, 2020 [35], United States | Cleveland Clinic Health System patient records (2005 to 2017) | Hypoglycemia: blood glucose level <70 mg/dL; severe hypoglycemia: patients with T2Dk require hospitalization or emergency department visit; nonsevere hypoglycemia: does not require assistance for recovery | NLP; ICD-9 codes: 251.0, 251.1, 251.2; ICD-10 codes: E08.641, E11.641, E11.649, E13.64, E13.641, E13.649, E16.0, E16.1, E15, E16.2 | Rule-based | Compared with clinician chart review manually, PPV=93% | Prevalence: among 204,517 patients with no codes for nonsevere hypoglycemia, evidence of nonsevere hypoglycemia was found in 7035 (3.4%) using NLP. Number of nonsevere hypoglycemia events: ICD codes (n/N [%]): 10,205/204,517 (4.99), NLP: 14,763/204,517 (7.22), with overlap of only 5 events. Incidence proportion of patients from 2005 to 2017 ICD codes (%): severe hypoglycemia: 0.3 to 1.7, nonsevere hypoglycemia: 0.4 to 1.3; NLP+ICD (%): nonsevere hypoglycemia: 0.8 to 2.6 |
| Uzoigw et al, 2020 [36], United States | Amplity Insights database, unstructured health records, generated from provider notes as transcribed from verbal to written form (January 1, 2016, to April 30, 2018) | Nonsymptom-based: mention of hypoglycemia, low blood glucose level or blood glucose value≤70 mg/dL; symptom-based: keywords identified by endocrinologists, used by patients to describe hypoglycemia | ICD codes; NLP | Rule-based | — | Prevalence during 2 years (%): T2D: ICD: 52 (<0.1); combined symptom and nonsymptom-based: 11.4; nonsymptom-based: 7.59; symptom-based: irritable or anxious: 14.50; cognitive issues: 12.14; elevated or irregular heart rate: 10.21. T1Dl: ICD codes: 30 (0.1); combined symptom and nonsymptom-based: 20.4; nonsymptom-based: 18.12; symptom-based: irritable or anxious: 16.00; cognitive issues: 8.17; elevated or irregular heart rate: 8.17 |
| Ganz et al, 2014 [37], United States | Humedica real-time longitudinal clinical data patient-level EHR database (January 2008 to December 2011) | Severe hypoglycemia: blood glucose level≤40 mg/dL | ICD-9 codes 251.0x, 251.1x, 251.2x, or 250.3x on different days; NLP | Rule-based | The final algorithm was validated by manual review: precision (PPV)=0.77, recall (sensitivity)=0.67 | Posttitration follow-up period (1.8 years): incidence rate (per 100 patient-years; 95% CI)=4.63 (4.59-4.67); total severe hypoglycemia rate (per 100 patient-years)=9.69 (9.64-9.75). Incidence rate for patients with history of severe hypoglycemia events (95% CI)=5.91 (5.76-6.06). Total severe hypoglycemia rate for patients with history of severe hypoglycemia events (95% CI)=9.00 (8.87-9.12) |
aEHR: electronic health record.
bICD-9: International Classification of Diseases, Ninth Revision.
cPPV: positive predictive value.
dICD: International Classification of Diseases.
eCPT: Current Procedures Terminology.
fEQW: exenatide once weekly.
gBI: basal insulin.
hICD-10: International Classification of Diseases, Tenth Revision.
iIM: intramuscular.
jLAI: long-acting insulin.
kT2D: type 2 diabetes.
lT1D: type 1 diabetes.