Abstract
Background:
Clinician’s asthma guideline adherence in asthma care is suboptimal. The effort for improving adherence can be enhanced by assessing and monitoring clinician’s adherence to guidelines reflected in electronic health records (EHRs) which yet requires costly manual chart review since many care elements cannot be identified by structured data.
Objective:
This study was designed to demonstrate the feasibility of an artificial intelligence (AI) tool using natural language processing (NLP) leveraging free text of EHRs of pediatric patients to extract key components of the 2007 National Asthma Education and Prevention Program guidelines.
Methods:
This is a retrospective cross-sectional study using a birth cohort with asthma diagnosis at Mayo Clinic between 2003 and 2016. We used 1,039 clinical notes with an asthma diagnosis from a random sample of 300 patients. Rule-based NLP algorithms were developed to identify asthma guideline congruent elements by examining care description in EHR free text.
Results:
NLP algorithms demonstrated a sensitivity (0.82 – 1.0), specificity (0.95 – 1.0), positive predictive value (0.86 −1.0), and negative protective value (0.92 – 1.0) against manual chart review for asthma guideline-congruent elements. Assessing medication compliance and inhaler technique assessment were the most challenging elements to assess due to the complexity and wide variety of descriptions.
Conclusion:
NLP technologies may enable automated assessment of clinician’s documentation in EHRs regarding adherence to asthma guidelines and can be a useful population management and research tool for assessing and monitoring asthma care quality. Multi-site studies with a larger sample size are needed for assessing the generalizability of our NLP algorithms.
Keywords: adherence to asthma guidelines, natural language processing, documentation variation, automated chart review, national asthma education and prevention program
Introduction
Pediatric asthma, the most common chronic illness among children, still causes significant morbidity and cost.(1–3) For example, among children ages 5 – 11, 50.3% are reported to have missed ≥ 1 school day, while 11.7% missed ≥ 9 school days due to asthma.(4) This increased burden of childhood asthma is true for our community in Olmsted County, Minnesota, where Mayo Clinic is located; For example, in our setting, the prevalence of asthma in a population of school-age children has been reported to be 17.6%.(5) and asthma is, indeed, the most common chronic condition and third most costly health care condition.(6) Implementation of and adherence to the asthma guidelines have been reported to improve asthma care and outcomes.(7–11)
The 2007 National Asthma Education and Prevention Program (NAEPP) on Asthma guidelines updated by an expert panel report-3 (EPR-3) have been an important basis for evidence-based asthma management strategies. In 2020, targeted updates of NAEPP guidelines on asthma management have been released.(12)
While clinician’s adherence to asthma guidelines is suboptimal and variable, given the positive impact of adherence to asthma guidelines on improved asthma outcomes and its implication on training, health disparities (13–15) and quality measure during a pay-for-performance era,(16) improving adherence to asthma guidelines will be crucial. Such effort for improving adherence to asthma guidelines in both asthma care and research hinges upon efficient and effective approaches for assessing clinician’s adherence to asthma guidelines based on their actual asthma management reflected in electronic health records (EHRs) instead of self-report.
However, assessing clinician’s adherence to asthma guidelines based on EHRs is challenging as it requires labor-intensive, time consuming, and costly manual chart review.(17, 18) Otherwise, many studies rely on self-reported survey or structured data such as asthma medications and current procedural terminology (CPT) codes, which have limited applications in assessing adherence to asthma guidelines (e.g., inhaled corticosteroids (ICS) or short-acting beta agonist use or performing spirometry) and ICD codes are unsuitable for assessing/monitoring asthma symptoms, activity limitation, inhaler technique, and control and severity status.
Alternatively, natural language processing (NLP), a subfield of artificial intelligence (AI) to analyze large amounts of human language data, has demonstrated great potentials in the areas of clinical practice and translational research. Previously we have demonstrated successful application of NLP to assessing asthma care and research (19–23) and other areas.(24–28) NLP can be an efficient, effective and practical way to mitigate the challenges of manual chart review in asthma care and research via enabling automated assessment of clinician’s adherence to guidelines from EHRs as a key step for improving asthma care quality and outcome. In this study we examined whether NLP algorithms can automatically identify asthma guideline elements from EHRs of pediatric patients and determined clinician’s guideline adherence and documentation.
Methods
Study Setting
This study utilized the 2003–2016 Mayo Clinic Birth Cohort (see Study Subjects section) enrolled in the Primary Care Pediatric practices at Mayo Clinic. Mayo Clinic is a tertiary care center, and the clinicians in our study include primary care physicians (pediatricians, family medicine doctors, and residents) and subspecialists (allergists and pulmonologists).
Study Design
This is a cross-sectional retrospective study nested in a birth cohort study designed to develop NLP algorithms to identify clinician’s documentation of asthma guideline elements. A high-level NLP algorithm development process is depicted in Figure 1. We previously reported the methods for NLP algorithm development for various asthma-related variables in detail (19–23) (see the NLP development section below). The NLP algorithms were developed using expert rules based on textual markers (i.e., keywords relevant to asthma guideline elements) and evaluated against manual chart review as a gold standard.
Figure 1.
A process of asthma guideline element extraction and evaluation
Study Subjects
The study data comes from 300 patients randomly selected from the 2003 – 2016 Mayo Clinic birth cohort who had an asthma diagnosis based on ICD-10 codes (20% mild intermittent [J45.2X], 30% mild persistent [J45.3X], and 50% moderate to severe asthma [J45.4X-J45.5X]) available in the institution during May 2016 – April 2018). From these patients, we selected clinical notes that contained an asthma diagnosis under the problem list (n=1,039); we used clinical notes from 200 patients as a training set (n=724 clinical notes) and clinical notes from the other 100 patients (n=315 clinical notes) as a test set. Figure 2 depicts the overall data selection process. The exclusion criteria include: 1) non-Olmsted County birth, 2) those born at non-Mayo Clinic facilities, 3) subjects with clinical conditions making asthma ascertainment difficult, 4) subjects who had received medical care other than at Mayo Clinic for potential asthma-related care (e.g., pneumonia, bronchiolitis), 5) subjects with insufficient medical records making asthma ascertainment difficult, and 6) subjects without research authorization as required by Minnesota statues.(29)
Figure 2.
A flow diagram of the process of data selection
Annotation of Asthma Guideline Elements
Operationalization of Guideline Elements
As the study period was prior to the availability of the 2020 asthma guideline updates (which primarily focused on asthma medications [especially Step 3 and 4 therapy], FeNO use, and allergy intervention), our assessment on guidelines elements (e.g., option for maintenance medications such as ICS) was based on the 2007 NAEPP guidelines, which were not significantly affected by the new 2020 asthma guidelines. Elements identifiable through EHR free text and their definitions based on chart review were adapted from our previous study,(17) where evidence-based elements were selected from 2007 NAEPP stepped care flow diagrams.(7, 30) The definitions and examples of documentation required for each element are described in Table I; these elements are grouped into 1) asthma control (day or night time asthma symptoms and activity limitation), 2) factors affecting control (medication compliance, inhaler technique education, and evaluation for triggers or smoking), and 3) medication therapy (short acting bronchodilator [rescue medications] and daily maintenance medications). These definitions of guideline elements were operationalized to be applicable to develop an NLP algorithm through a discussion and consensus process with experts (asthma expert [BY], pulmonologist [HYS], and allergists [JY and MP]); we compiled relevant concepts and keywords and determined logics how to judge whether a given guideline element was recorded or not in EHR documents (Table II). For example, one of the asthma control elements, “activity limitation” was defined as an explicit mention of activity limitation (e.g., “limitation on activity”), combination of indication keywords (e.g., “difficulty in the gym class”), or implied activity limitation phrases (e.g., “missed school”). To avoid potential false positives of activity limitation not related to asthma, we parsed the element from the history of present illness section that includes asthma concepts.
Table I.
Guideline elements extracted from clinical notes by NLP
| Guideline element | Required documentation | Example |
|---|---|---|
| Asthma Control | Daytime asthma symptoms (with frequency) Nighttime asthma symptoms (with frequency) Activity limitations |
He had no difficulty of breathing for a month. She can awaken 1–2 times during the night due to coughing. He missed school due to asthma symptoms. |
|
| ||
| Factors associated with control | ||
| Medication compliance Inhaler technique Allergy/irritant assessment Smoking |
Medication compliance discussed Review of patient’s inhaler technique Allergy testing or assessment Household member smoking status |
Patient is able to use Flovent about 60% of time. Discussed correct inhaler technique. Exercise can also be a trigger. Non-smoker |
|
| ||
| Medications | ||
| Rescue medication Maintenance medication |
Recorded Recorded |
Albuterol (under medication section) Flovent (under medication section) |
Adapted from Yawn’s work (17).
Table II.
Keywords and rules to identify data elements in NAEPP guidelines
| Data Element | Keywords | Rules |
|---|---|---|
| Asthma symptoms | 1) wheez*, cough*, dyspnea*, tachypnea*, sob, short* of breath*, increased breath, (chest/ throat) … (tight*/squeeze*/stuck/congestion), chest … pain/discomfort, respiratory … (distress/illness/difficult*/rate), (short*/heavy/rapid/difficult/labor/hard/struggle/worsen/loud/raspy/retraction) … breath*, (asthma*/allerg*/respiratory/daytime) … symptoms, chest pain/discomfort 2) night*, nighttime, nocturnal, evening*, tonight, wake*, woke, get* up, sleep*, slept, awoke, awak*, morning 3) day*, yesterday, noon, week*, today |
I. Nighttime symptom: combination of keyword 1 and 2 II. Daytime symptom: combination of keyword 1 and 3, or keyword 1 with absence of keywords 2 |
| Activity limitation | 1) exertional/normal activity, exercise/activity tolerance, miss/leave/left/day off/restricts/trouble… work/school 2) limit*, athletic*, impacts, difficult*, trouble*, problems, impar*, decreas*, stop*, down, able, can, inhaler, restrict, normal, active, participat*, play 3) basketball, hockey, softball, gym, run*, walk*, soccer, football, ski*, dance*, jump*, swim*, play*, active*, exercise*, performs, performance, daily living, exert |
I. Keyword 1 only II. Combination of keyword 2 and 3 III. Combination of keyword 3 and asthma symptom keyword 1 |
| Medication compliance | 1) adherent, adherence, discontinued, (he/she/they/patient/they) … not initiate/using/taking/tries/been doing/, has been … taking/off/utilizing/missed/forgot/stopped, asthma…well controlled, he/she/they/patient/they … trouble/difficulty remembering, has been … inconsistent/consistent, adhering, compliance, deny regular use, nonadherence, noncompliance 2) medication*, medicine, dose*, med, dosing, dosage, dose, inhaled corticosteroid, controller, ICS, puff*, inhaler* medication |
I. Combination of keyword 1 and 2 II. Combination of keyword 1 and maintenance medications |
| Inhaler technique | 1) observ*, reassess*, review*, demonstrat*, check*, educat*, teach*, taught, explain*, reinforce*, discuss*, instruction, constraints, how to use 2) inhaler, MDI, neb, nebulizer, optichamber, spacer 3) techniques, administrations, dosing, guidance 4) asthma/rescue/daily/preventive/control medication, ICS, list of maintenance & rescue medications |
I. Combination of keyword 1 and 2 II. Combination of keyword 1, 3, and 4 |
| Allergy/irritant | 1) worsen, trigger*, cause, flare, occur, exacerbate, irritant, associated with, problem when, aggravate, induced, manifest 2) smoke, airborne, allergens*, activit*, cold air, common cold, cold weather, cold … URI, mold, cat, rabbit, hay, air quality/pollutant, influenza, vaccine, immunization, pets, exposure 3) trees, grass, dogs, dust, mite, elm, oak, weed 4) (as/is a/ be a) trigger, symptoms … with exercise, symptoms/ flare of asthma … (with/by/when/cause) |
I. Combination of keyword 1 and 2 II. Combination of keyword 3 and “allergy” III. Keyword 4 only |
| Smoking | smok*, tobacco, cigarette*, nonsmoker*
(exclusion indicator: smoke detector, fire, trigger) |
Smoking keywords (disregard if appears with exclusion indicators) |
| Rescue medications | Short-acting beta-agonist (SABAs) (exclusion indicator: topical, nasal, ointment, eye, drop, ophthalm*) |
medications belong to this (disregard if appears with exclusion indicators) |
| Maintenance medication | Asthma maintenance medications (exclusion indicator: topical, nasal, ointment, eye, drop, ophthalm*) |
medications belong to this (disregard if appears with exclusion indicators) |
denotes any characters to handle spelling variations and some potential misspells
Manual Annotation (Gold Standard Generation)
Based on the operationalized definitions of guideline elements, we developed annotation guidelines for annotators to perform manual chart review and mark elements in EHR free text. Two physicians (HYS [pulmonologist] and JY [allergist]) performed chart review and annotated guideline elements using an open-source portable annotation tool, multi-document annotation environment (MAE).(31) This manual annotation was served as the gold standard to develop and validate an NLP algorithm to identify guideline elements.
NLP Algorithm Development
The development and evaluation process went through the following steps: 1) retrieving clinical notes from EHRs leveraging the unified data platform, 2) text processing for generic NLP (i.e., sentence segmentation, sectionization, assertion [e.g., positive, negated, hypothetical]) as a preprocessing step (a left side in a upper dotted box, Figure 1) and temporal information extraction (a lower dotted box, Figure 1), 3) implementing rules to identify specific guideline elements (a right side in a upper dotted box, Figure 1), and 4) statistical analysis to evaluate the performance of NLP algorithms. Clinical notes consist of multiple sections (e.g., diagnosis, history of present illness [HPI], impression/report/plan [IRP], current medication). Different sections contain specific information relevant to patient conditions. We identified a set of sections that potentially contain specific guideline elements and used them to assess adherence to guidelines (i.e., documentation of guideline elements). Then, we extracted keywords related to each guideline element based on the description patterns in clinical notes and applied expert rules to determine presence or absence of a given concept of guideline elements in EHR documents (summarized in Table II). The keywords and rules were initially provided by expert clinicians and updated and then refined iteratively through the discussion of both clinicians and informaticians as we developed the NLP algorithms based on the training set. The final NLP algorithms were evaluated on the independent test set.
We implemented NLP algorithms under the framework of MedTaggerIE, a clinical NLP pipeline developed by Mayo Clinic.(32) The temporal information (i.e., frequency and duration) associated with guideline elements of daytime and nighttime asthma-related symptoms were identified by using a MedTime, an open-source temporal information extraction tool developed by Mayo Clinic.(33) MedTime uses pre-defined temporal patterns and variations and maps them to the standard forms, enabling further customization.
The performance of the NLP algorithm extracting guideline elements was evaluated using manual chart review as a gold standard in a document level (i.e., whether a guideline element is recorded or not in the clinical note) by calculating sensitivity, specificity, predictive positive value (PPV), and negative predictive value (NPV). To further assess the validity of NLP algorithms in automated assessment of clinician’s adherence to asthma guidelines, we also compared guideline adherence (i.e., the percentage of clinical notes that recorded a given asthma guideline element out of the total clinical notes) between NLP-based computation and manual chart review.
Results
Study Cohort Characteristics
Table III contains basic characteristics of the study cohort. For the Test cohort (n=100), median age (interquartile range) was 8.6 years (4.3–13.1) with 45% female, 77% White and 5% Hispanic or Latino, which are similar to those of the Training cohort. There are total 280 clinicians who generated the clinical notes in the study period with median (IQR) number of clinical notes per patient 2 (1, 5) and 2 (1, 4) for the training and test cohort, respectively. Figure 3 shows prevalence of asthma guideline elements (7.0% – 74.5%; i.e., percentage of clinical notes that contain a given asthma guideline element) in manual chart review (gold standard) of the study cohort (N=300). Daytime symptom (74.5%), rescue medication (66.4%), and maintenance medication (52.2%) are most frequent elements assessed and documented in more than half of clinical notes. Smoking status (7.0%) was the least frequently assessed and documented element in the notes.
Table III.
Basic characteristics of the study cohort
| Characteristics | Training cohort (N=200) | Test cohort (N=100) |
|---|---|---|
| Median (IQR) of the number of clinical notes per patient | 2 (1, 5) | 2 (1, 4) |
|
| ||
| Age as of 5/1/2016, median (IQR) | 9.5 (6.3, 13.2) | 8.6 (4.3, 13.1) |
|
| ||
| Female, n (%) | 96 (48) | 45 (45) |
|
| ||
| Race, n (%) | ||
| White | 152 (76) | 77 (77) |
| Black | 20 (10) | 4 (4) |
| Asian | 6 (3) | 3 (3) |
| Others/Unknown | 22 (11) | 16 (16) |
|
| ||
| Ethnicity, n (%) | ||
| Hispanic or Latino | 9 (5) | 5 (5) |
| Non-Hispanic | 189 (94) | 94 (94) |
| Unknown | 2(1) | 1 (1) |
|
| ||
| Asthma severity by NAEPP, n (%) | ||
| Mild intermittent | 40 (20) | 20 (20) |
| Mild persistent | 60 (30) | 30 (30) |
| Moderate to Severe asthma | 100 (50) | 50 (50) |
Figure 3.
Prevalence of adherence to asthma guideline elements in the clinical note level (N=300 patients).
NLP Algorithm Performance
Identification of Asthma Guideline Elements
The performance of NLP algorithm’s ability to identify documentation of guideline elements (i.e., whether a clinical note contained a given guideline element or not) is summarized in Table IV. Overall, NLP algorithms had high sensitivity (0.82 – 1.0), specificity (0.95 – 1.0), PPV (0.89 – 1.0), and NPV (0.92 – 1.0) depending on the guideline elements assessed. Specifically, medication therapy (denoted as rescue and maintenance medications) was almost perfectly identified (0.99 to 1.0 for all measures) by NLP algorithms and asthma control elements (day/nighttime symptoms and activity limitation) such as presence of wheezing or coughing were accurately identified (≥0.92 for all measures without regard to frequency). Some factors that might impact asthma control status, such as medication compliance and inhaler technique adequacy, had relatively lower sensitivity (= 0.82) compared to other guideline elements. Assessment of daytime and nighttime symptom frequency was often linked to medication use making challenging to distinguish frequency of daytime and nighttime symptoms from frequency of medications. Medication compliance and inhaler technique adequacy were often described using various language and word patterns that are difficult to capture by handcrafted rules. The concept-level performance of NLP algorithms (i.e., measured on all guideline elements in clinical notes) is presented in Table E1 in Online Repository Materials. This performance reflects more granular NLP algorithm performance and the overall performance was similar to the one in a document-level.
Table IV.
Performance of asthma guideline element extraction on the test set (document-level, N= 315 documents).
| Guideline element | Sensitivity | Specificity | PPV | NPV | Number of documents |
|---|---|---|---|---|---|
| Asthma Control | |||||
| Daytime asthma symptoms (with frequency) | 0.96 (0.89) | 0.99 (0.99) | 0.99 (0.95) | 0.96 (0.97) | 154 (64) |
| Nighttime asthma symptoms (with frequency) | 0.98 (0.85) | 0.99 (0.99) | 0.99 (0.94) | 0.99 (0.99) | 85 (20) |
| Activity limitations | 0.97 | 0.98 | 0.93 | 0.99 | 70 |
|
| |||||
| Factors associated with level of control | |||||
| Medication compliance | 0.82 | 0.95 | 0.89 | 0.92 | 102 |
| Inhaler technique | 0.82 | 0.98 | 0.86 | 0.97 | 38 |
| Allergy/irritant assessment | 0.98 | 0.97 | 0.94 | 0.99 | 111 |
| Smoking status | 1.00 | 1.00 | 1.00 | 1.00 | 21 |
|
| |||||
| Medications | |||||
| Rescue medication | 1.00 | 1.00 | 1.00 | 1.00 | 213 |
| Maintenance medication | 0.99 | 1.00 | 1.00 | 0.99 | 173 |
Automated Assessment of Clinician’s Adherence to Asthma Guidelines
NLP-based clinician’s adherence to asthma guidelines (i.e., use of NLP-detected guideline elements to generate adherence, percentage of clinical notes that record an asthma guideline element) was compared to the one with manual chart review (i.e., use of chart reviewed guideline elements to generate adherence) in Table V. The NLP-based adherence assessment was highly correlated with that of manual chart review (rate difference ranges only from 0 to 2.6%), demonstrating the feasibility of NLP-based automated chart review to assess clinician’s adherence to asthma guidelines; factors associated with asthma control (medication compliance, inhaler technique, allergy/irritant assessment) and activity limitation had 0.7% - 2.6% differences, and the rest were very close (≤0.5% differences).
Table V.
Comparison of clinician’s adherence assessment between NLP and manual chart review on the test set (N=315 documents).
| Guideline element | Adherence, % |
|
|---|---|---|
| NLP | Manual | |
| Asthma Control | ||
| Daytime asthma symptoms (with frequency) | 74.4 (29.6) | 76.4 (31.5) |
| Nighttime asthma symptoms (with frequency) | 41.4 (8.9) | 41.9 (9.9) |
| Activity limitations | 36.0 | 34.5 |
|
| ||
| Factors associated with level of control | ||
| Medication compliance | 29.8 | 32.4 |
| Inhaler technique | 11.4 | 12.1 |
| Allergy/irritant assessment | 36.8 | 35.2 |
| Smoking status | 6.7 | 6.7 |
|
| ||
| Medications | ||
| Rescue medication | 67.6 | 67.6 |
| Maintenance medication | 54.6 | 54.9 |
Discussion
To our knowledge, this is the first study which assessed NLP algorithms to identify asthma guideline elements from EHRs and determine clinician’s guideline adherence and documentation. Our NLP algorithms demonstrated high sensitivity (0.82 – 1.0), specificity (0.95 – 1.0), PPV (0.86 −1.0), and NPV (0.92 – 1.0) in asthma guideline element identification. Collectively, all elements achieved >0.9 for all four performance measures except medication compliance and inhaler technique. These two guideline elements were difficult to be accurately identified by hand-crafted rules due to the complexity of their descriptions, rather requiring contextual understanding. For example, “Discussed 3rd neb treatment here versus one upon home.” This statement contains some indication words relevant to inhaler techniques, such as “discussed” and “neb,” but is not regarding teaching or reviewing inhaler techniques, rather discussing the efficacy of nebulizer treatment that needs to be differentiated from true inhaler technique in asthma guidelines. A recent innovation of deep learning using contextual embedding such as Bidirectional Encoder Representations from Transformers (BERT) (34) has a capability to better understand context (i.e., semantic meaning of words or phrases) and may allow capturing complex cases of guideline elements missed by handcrafted rules.
The high concordance between NLP-based and manual chart review assessment of guideline adherence (Table V) demonstrates the potential of NLP to automatically process large amount of EHR documents efficiently and effectively to provide evidence-based asthma management practice reflected in EHRs to achieve improvement of adherence to asthma guidelines and outcomes. Also, NLP algorithms can help to identify individual clinicians or their care teams that have low adherence to specific guideline elements with evidence (their documentations in EHRs) so as to help them to improve asthma care quality via training, education, and guidance monitoring changes of asthma care quality over time.(35) Also, our study results can be extended to adult asthma care as it only requires minimal revision of our current NLP algorithms.
Clinicians are a primary creator of EHR documents, but their clinical practices vary substantially (36) affecting the quality of clinical documentation and cause potential bias in downstream applications of EHRs. Inconsistent approach to patient care may also result in higher costs requiring to reduce clinical variations at the point of care.(37, 38) Yawn et al previously demonstrated that adherence to asthma guidelines in primary care practices remains low (e.g., 3.1% for asthma action plan to 32.5% for allergy evaluation), missing many opportunities for improvement in asthma care that could lead to improved outcomes.(17) Recently, Akinbami et al reported a primary care clinician’s (family/general medicine practitioners, internists, and pediatricians) adherence with asthma guidelines (EPR-3 recommendation) based on the 2012 National Asthma Survey of Physicians.(39) While all primary care clinicians prescribed inhaled corticosteroids for daily control (84%−91%), adherence to guidelines for performing spirometry was significantly low (7%−17%) and that for assessing/monitoring asthma symptoms significantly varied. For asthma specialists’ adherence to asthma guidelines, Cloutier el al used the same survey data and showed that while agreement in asthma guidelines (EPR-3) between allergists and pulmonologists did not differ, both groups of specialists, however, reported relatively low adherence in asthma action plan use and assessment of the inhaler technique.(40) The variability in clinician’s asthma guideline adherence and documentation (17, 41, 42) extends to medical trainees (e.g., residents),(35) indicating the need for systematic efficient and effective assessment of asthma guidelines adherence to better guide and target education for clinicians and medical trainees to improve asthma care. The literature reported that improvement of medical trainees’ competence in asthma diagnosis and management requires clinical effectiveness data-driven report or feedback derived from their clinical documentation.(43) As medical trainees’ documentation often contains inaccurate information in progress notes,(44) the potential use of a point-of-care personal digital assistant has been examined as a means of improving accuracy and quality of their clinical documentation.(45) A requirement of the Accreditation Council of Graduate Medical Education (ACGME) is the provision of data-driven clinical competence assessment (e.g., actual clinical practice derived from clinical notes).(46) However, this has proven challenging for many programs due to the lack of tools available to medical educators because manual chart review of trainees’ clinical notes by the teaching faculty on a regular basis is challenging. Despite the needs of improving clinical competence in asthma care and clinical documentation through “individualized clinical effectiveness data” derived from clinical documentation, such data or tool is not available in most clinical settings.
The NAEPP guidelines outline asthma-related concepts for evidence-based asthma management that can be used as the basis for asthma assessment. However, most guideline elements of NAEPP are often embedded in EHR free text requiring manual chart review. This study addressed these unmet needs using NLP to perform automated chart review to identify NAEPP asthma guideline adherence elements from clinical notes in the EHR. This capability enables mining large scale EHRs to support effective asthma management at a population level, offers personalized learning to clinicians in asthma care and documentation, and ultimately improves asthma care quality and outcomes (7–11) potentially resulting in accountability application (e.g., public report, payment etc.).
Despite these advantages of NLP algorithms, there are several limitations in this study. We operationalized the definitions of NAEPP guideline elements to be applicable for NLP algorithms by a discussion and consensus process with experts (i.e., knowledge engineering). However, some elements were challenging to clearly define. For example, activity limitation, the extent of its definition is not straightforward due to a broad range of activities and difficulty to make a clear association with asthma. Although we defined activity limitation through rigorous discussion and consensus (e.g., sport/physical activity limitation, missed school), it might be still subjective and may require modification if needed for specific applications. Some standard template for documenting asthma assessment and management might help to address this challenge.
The NLP algorithms were developed using clinical notes in a single institution tailored to a specific EHR system at Mayo Clinic (GE-based EHR system). The number of clinicians who generated clinical notes (n=280) was reasonably large to reflect clinical documentation variations to support generalizability and validity of NLP algorithms. Despite this diversity of clinicians at Mayo Clinic and the high NLP performance, the algorithms may not perform similarly in other EHR systems or institutions due to EHR variabilities. However, our NLP algorithms can be adapted efficiently since we separate domain-specific knowledge (i.e., keywords and rules) from the main NLP programs, allowing easy customization supporting portability of our NLP algorithms, as demonstrated in our previous studies.(21, 47) Also, our NLP algorithms need to be assessed for clinicians managing adults with asthma. As the primary scope of our present study is to develop NLP algorithms, we plan to validate our NLP algorithms on EHRs from different institutions and adult cohort as our future studies since it needs additional manual chart review to create the gold standard that requires significant time, effort, and cost. The NLP algorithm generalization to external EHR data requires to address clinical documentation variations: process variation due to different clinical practice and workflows and reporting schemes for generating EHR data, syntactic variation due to heterogeneous clinical language,(48, 49) and semantic variation due to potential different interpretations of the event.(50, 51) Considering these facts, the process of NLP algorithm generalization includes two main steps: (1) adjust the algorithm on different EHR data to be technically operable (i.e., address process variation; for example, adjusting text format) and (2) refine the algorithm to achieve acceptable performance (i.e., address syntactic and semantic variation; for example, refining negation syntax).(21, 47) Lastly, although NLP algorithms can extract and compile asthma guideline-congruent elements from clinician’s documentation, promoting evidence-based assessment using EHRs, it may not perfectly reflect actual clinician’s asthma care in practice. Future studies need to assess accuracy of our NLP-based assessment of clinician’s adherence to guidelines based on documentation in medical records.
This study focused on demonstrating the feasibility of an NLP tool leveraging free text of EHRs to extract key components of asthma care congruent with the NAEPP. Our planned research is to assess if clinician’s adherence to asthma guidelines using the NLP tool mining large-scale EHR documents is associated with clinical outcomes and patients’ health disparities.
Conclusion
Clinician’s adherence to asthma guidelines in primary care practices is low and also, this caveat poses many opportunities for improving asthma care quality through innovative approaches. In achieving this goal, an AI approach using NLP technologies is a useful tool enabling an automated evidence-based assessment of clinician’s adherence to asthma guidelines via mining large scale EHRs in near real time and providing personalized learning to clinicians in asthma care and documentation while reducing the burden of identifying asthma care with low adherence. The NLP tool may be useful for both specialty and primary care clinicians, especially in those training, supporting evidence-based clinical effectiveness data assessments to enhance adherence to guidelines. Multi-site studies with large and diverse EHR data are needed for assessing the generalizability of our NLP algorithms.
Supplementary Material
Highlights Box:
What is already known about this topic?
Asthma guideline adherence in primary care is low, missing opportunities for improving asthma care and outcome. However, assessing clinician’s guideline adherence requires manual chart review of electronic health records, not readily applicable in asthma care.
What does this article add to our knowledge?
Natural language processing (NLP) enables automated assessment of clinician’s adherence to asthma guidelines based on electronic health records (EHRs) and can be a useful population management and research tool for assessing and monitoring asthma care quality.
How does this study impact current management guidelines?
An NLP-empowered assessment tool alleviates costly and challenging manual chart review of EHRs and provides an innovative approach to assess and improve clinician’s adherence to evidence-based asthma management guidelines.
Acknowledgement:
We thank Mrs. Kelly Okeson, for her administrative assistance.
Sources of Support/Funding: This study was supported by National Institute of Health (NIH)-funded R21 grant (R21 AI142702) and R01 grant (R01 HL126667).
Abbreviations:
- ACGME
Accreditation Council of Graduate Medical Education
- AI
Artificial Intelligence
- CPT
Current Procedural Terminology
- ICS
Inhaled Corticosteroids
- NAEPP
National Asthma Education and Prevention Program
- NLP
Natural Language Processing
- EHR
Electronic Health Record
- EPR-3
Expert Panel Report-3
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Financial Disclosure: Young J. Juhn is Principal Investigator (PI) of the Respiratory Syncytial Virus incidence study supported by GlaxoSmithKline. The remaining authors have no financial relationships relevant to this article to disclose.
Conflicts of interest: The authors declare no conflict of interest.
References
- 1.Control CfD, Prevention. Vital signs: asthma prevalence, disease characteristics, and self-management education: United States, 2001−-2009. MMWR Morbidity and mortality weekly report 2011;60(17):547. [PubMed] [Google Scholar]
- 2.Lethbridge-Çejku M, Vickerie JL. Summary health statistics for US adults; National Health Interview Survey, 2003 2004. [PubMed] [Google Scholar]
- 3.Stanton MW, Rutherford M. The high concentration of US health care expenditures: Agency for Healthcare Research and Quality; Rockville (MD); 2006. [Google Scholar]
- 4.Kim CH, Gee KA, Byrd RS. Excessive absenteeism due to asthma in California elementary schoolchildren. Academic pediatrics 2020;20(7):950–7. [DOI] [PubMed] [Google Scholar]
- 5.Yawn BP, Wollan P, Kurland M, Scanlon P. A longitudinal study of the prevalence of asthma in a community population of school-age children. The Journal of pediatrics 2002;140(5):576–81. [DOI] [PubMed] [Google Scholar]
- 6.Zhong W, Finnie DM, Shah ND, Wagie AE, St. Sauver JL, Jacobson DJ, et al. Effect of multiple chronic diseases on health care expenditures in childhood. Journal of primary care & community health 2015;6(1):2–9. [DOI] [PubMed] [Google Scholar]
- 7.National Heart L, Institute B. Expert panel report III: guidelines for the diagnosis and management of asthma 2007. [Google Scholar]
- 8.Piecoro LT, Potoski M, Talbert JC, Doherty DE. Asthma prevalence, cost, and adherence with expert guidelines on the utilization of health care services and costs in a state Medicaid population. Health services research 2001;36(2):357. [PMC free article] [PubMed] [Google Scholar]
- 9.Senthilselvan A, Lawson JA, Rennie DC, Dosman JA. Regular use of corticosteroids and low use of short-acting β2-agonists can reduce asthma hospitalization. Chest 2005;127(4):1242–51. [DOI] [PubMed] [Google Scholar]
- 10.Cloutier MM, Hall CB, Wakefield DB, Bailit H. Use of asthma guidelines by primary care providers to reduce hospitalizations and emergency department visits in poor, minority, urban children. The Journal of pediatrics 2005;146(5):591–7. [DOI] [PubMed] [Google Scholar]
- 11.Fox P, Porter PG, Lob SH, Boer JH, Rocha DA, Adelson JW. Improving asthma-related health outcomes among low-income, multiethnic, school-aged children: results of a demonstration project that combined continuous quality improvement and community health worker strategies. Pediatrics 2007;120(4):e902–e11. [DOI] [PubMed] [Google Scholar]
- 12.Cloutier MM, Dixon AE, Krishnan JA, Lemanske RF, Pace W, Schatz M. Managing asthma in adolescents and adults: 2020 asthma guideline update from the National Asthma Education and Prevention Program. JAMA 2020;324(22):2301–17. [DOI] [PubMed] [Google Scholar]
- 13.Chin MH, Alexander-Young M, Burnet DL. Health care quality-improvement approaches to reducing child health disparities. Pediatrics 2009;124(Supplement 3):S224–S36. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Berdahl TA, Friedman BS, McCormick MC, Simpson L. Annual report on health care for children and youth in the United States: trends in racial/ethnic, income, and insurance disparities over time, 2002–2009. Academic pediatrics 2013;13(3):191–203. [DOI] [PubMed] [Google Scholar]
- 15.Ogbogu PU, Capers IV Q, Apter AJ. Disparities in Asthma and Allergy Care: What Can We Do? The Journal of Allergy and Clinical Immunology: In Practice 2021;9(2):663–9. [DOI] [PubMed] [Google Scholar]
- 16.Lindenauer PK, Remus D, Roman S, Rothberg MB, Benjamin EM, Ma A, et al. Public reporting and pay for performance in hospital quality improvement. New England Journal of Medicine 2007;356(5):486–96. [DOI] [PubMed] [Google Scholar]
- 17.Yawn BP, Rank MA, Cabana MD, Wollan PC, Juhn YJ, editors. Adherence to asthma guidelines in children, tweens, and adults in primary care settings: a practice-based network assessment. Mayo Clin Proc; 2016: Elsevier. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Shapiro A, Gracy D, Quinones W, Applebaum J, Sarmiento A. Putting guidelines into practice: improving documentation of pediatric asthma management using a decision-making tool. Archives of pediatrics & adolescent medicine 2011;165(5):412–8. [DOI] [PubMed] [Google Scholar]
- 19.Kaur H, Sohn S, Wi C-I, Ryu E, Park MA, Bachman K, et al. Automated chart review utilizing natural language processing algorithm for asthma predictive index. BMC pulmonary medicine 2018;18(1):1–9. *These authors contributed equally. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Sohn S, Wi C-I, Wu S, Liu H, Ryu E, Krusemark E, et al. Ascertainment of Asthma Prognosis Using Natural Language Processing from Electronic Medical Records. Journal of Allergy and Clinical Immunology 2018;141(6):2292–4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Wi C-I, Sohn S, Ali M, Krusemark E, Ryu E, Liu H, et al. Natural Language Processing for Asthma Ascertainment in Different Practice Settings. The Journal of Allergy and Clinical Immunology: In Practice 2018;6(1):126–31. *These authors contributed equally. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Wi C-I, Sohn S, Rolfes MC, Seabright A, Ryu E, Voge G, et al. Application of a Natural Language Processing Algorithm to Asthma Ascertainment: An Automated Chart Review. American Journal of Respiratory And Critical Care Medicine 2017;196(4): *These authors contributed equally. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Wu ST, Sohn S, Ravikumar K, Wagholikar K, Jonnalagadda SR, Liu H, et al. Automated chart review for asthma cohort identification using natural language processing: an exploratory study. Annals of Allergy, Asthma & Immunology 2013;111(5):364–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Sohn S, Clark C, Halgrim S, Murphy S, Chute C, Liu H. MedXN: an Open Source Medication Extraction and Normalization Tool for Clinical Text. Journal of the American Medical Informatics Association 2014;21(5):858–65. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Sohn S, Kocher J-PA, Chute CG, Savova GK. Drug side effect extraction from clinical narratives of psychiatry and psychology patients. J Am Med Inform Assoc 2011;18(Suppl 1):144–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Sohn S, Savova GK. Mayo Clinic Smoking Status Classification System: Extensions and Improvements AMIA Annual Symposium; San Francisco, CA: 619–623; 2009. [PMC free article] [PubMed] [Google Scholar]
- 27.Sohn S, Ye Z, Liu H, Chute C, Kullo I. Identifying Abdominal Aortic Aneurysm Cases and Controls using Natural Language Processing of Radiology Reports. AMIA Summits Transl Sci Proc; Mar 2013; San Francisco2013. [PMC free article] [PubMed] [Google Scholar]
- 28.Sohn S, Torii M, Li D, Wagholikar K, Wu S, Liu H. A Hybrid Approach to Sentiment Sentence Classification in Suicide Notes. Biomedical informatics insights 2012(Suppl. 1):43–50. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. 2020 Minnesota Statutes: DISCLOSURE OF HEALTH RECORDS FOR EXTERNAL RESEARCH. 2020 [Google Scholar]
- 30.Yawn BP, Enright PL, Lemanske RF Jr, Israel E, Pace W, Wollan P, et al. Spirometry can be done in family physicians’ offices and alters clinical decisions in management of asthma and COPD. Chest 2007;132(4):1162–8. [DOI] [PubMed] [Google Scholar]
- 31.Rim K, editor Mae2: Portable annotation tool for general natural language use Proceedings of 12th Joint ACL-ISO Workshop on Interoperable Semantic Annotation; 2016. [Google Scholar]
- 32.Liu H, Bielinski S, Sohn S, Murphy S, Wagholikar K, Jonnalagadda S, et al. An information extraction framework for cohort identification using electronic health records. AMIA Summits Transl Sci Proc; 2013. Mar; San Francisco, CA2013. p. 149–53. [PMC free article] [PubMed] [Google Scholar]
- 33.Sohn S, Wagholikar KB, Li D, Jonnalagadda SR, Tao C, Elayavilli RK, et al. Comprehensive temporal information detection from clinical text: medical events, time, and TLINK identification. J Am Med Inform Assoc 2013. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Devlin J, Chang M-W, Lee K, Toutanova K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:181004805 2018. [Google Scholar]
- 35.Sohn S, Wi C, Juhn Y, Liu H. Analysis of Clinical Variations in Asthma Care Documented in Electronic Health Records Between Staff and Resident Physicians. Studies in health technology and informatics 2017;245:1170–4. [PMC free article] [PubMed] [Google Scholar]
- 36.Weber GM, Kohane IS. Extracting physician group intelligence from electronic health records to support evidence based medicine. PloS one 2013;8(5):e64933. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Schuur JD, Baugh CW, Hess EP, Hilton JA, Pines JM, Asplin BR. Critical Pathways for Post–Emergency Outpatient Diagnosis and Treatment: Tools to Improve the Value of Emergency Care. Academic Emergency Medicine 2011;18(6):e52–e63. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Wennberg JE. Unwarranted variations in healthcare delivery: implications for academic medical centres. British Medical Journal 2002;325(7370):961. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Akinbami LJ, Salo PM, Cloutier MM, Wilkerson JC, Elward KS, Mazurek JM, et al. Primary care clinician adherence with asthma guidelines: the National Asthma Survey of Physicians. Journal of Asthma 2020;57(5):543–55. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Cloutier MM, Akinbami LJ, Salo PM, Schatz M, Simoneau T, Wilkerson JC, et al. Use of National Asthma Guidelines by Allergists and Pulmonologists: A National Survey. The Journal of Allergy and Clinical Immunology: In Practice 2020;8(9):3011–20. e2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Mold JW, Fox C, Wisniewski A, Lipman PD, Krauss MR, Harris DR, et al. Implementing asthma guidelines using practice facilitation and local learning collaboratives: a randomized controlled trial. The Annals of Family Medicine 2014;12(3):233–40. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Yee AB, Fagnano M, Halterman JS. Preventive asthma care delivery in the primary care office: missed opportunities for children with persistent asthma symptoms. Academic pediatrics 2013;13(2):98–104. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Hemnes AR, Bertram A, Sisson SD. Impact of medical residency on knowledge of asthma. Journal of Asthma 2009;46(1):36–40. [DOI] [PubMed] [Google Scholar]
- 44.Carroll AE, Tarczy-Hornoch P, O’Reilly E, Christakis DA. Resident documentation discrepancies in a neonatal intensive care unit. Pediatrics 2003;111(5):976–80. [DOI] [PubMed] [Google Scholar]
- 45.Carroll AE, Tarczy-Hornoch P, O’Reilly E, Christakis DA. The effect of point-of-care personal digital assistant use on resident documentation discrepancies. Pediatrics 2004;113(3):450–4. [DOI] [PubMed] [Google Scholar]
- 46.(ACGME) ACfGME. ACGME Resident Survey Content Areas 2014. [Available from: http://www.acgme.org/Portals/0/ResidentSurvey_ContentAreas.pdf.
- 47.Sohn S, Wang Y, Wi C-I, Krusemark EA, Ryu E, Ali MH, et al. Clinical documentation variations and NLP system portability: a case study in asthma birth cohorts across institutions. Journal of the American Medical Informatics Association 2018;25(3):353–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Stetson PD, Johnson SB, Scotch M, Hripcsak G. The sublanguage of cross-coverage. American Medical Informatics Association 2002. p. 742–6. [PMC free article] [PubMed] [Google Scholar]
- 49.Friedman C, Kra P, Rzhetsky A. Two biomedical sublanguages: a description based on the theories of Zellig Harris. Journal of Biomedical Informatics 2002;35(4):222–35. [DOI] [PubMed] [Google Scholar]
- 50.Wu Y, Denny JC, Rosenbloom ST, Miller RA, Giuse DA, Wang L, et al. A long journey to short abbreviations: developing an open-source framework for clinical abbreviation recognition and disambiguation (CARD). Journal of the American Medical Informatics Association 2016:ocw109. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Xu H, Stetson PD, Friedman C. Methods for building sense inventories of abbreviations in clinical notes. Journal of the American Medical Informatics Association 2009;16(1):103–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.