Application of a Natural Language Processing Algorithm to Asthma Ascertainment. An Automated Chart Review

Abstract

Rationale: Difficulty of asthma ascertainment and its associated methodologic heterogeneity have created significant barriers to asthma care and research.

Objectives: We evaluated the validity of an existing natural language processing (NLP) algorithm for asthma criteria to enable an automated chart review using electronic medical records (EMRs).

Methods: The study was designed as a retrospective birth cohort study using a random sample of 500 subjects from the 1997–2007 Mayo Birth Cohort who were born at Mayo Clinic and enrolled in primary pediatric care at Mayo Clinic Rochester. Performance of NLP-based asthma ascertainment using predetermined asthma criteria was assessed by determining both criterion validity (chart review of EMRs by abstractor as a gold standard) and construct validity (association with known risk factors for asthma, such as allergic rhinitis).

Measurements and Main Results: After excluding three subjects whose respiratory symptoms could be attributed to other conditions (e.g., tracheomalacia), among the remaining eligible 497 subjects, 51% were male, 77% white persons, and the median age at last follow-up date was 11.5 years. The asthma prevalence was 31% in the study cohort. Sensitivity, specificity, positive predictive value, and negative predictive value for NLP algorithm in predicting asthma status were 97%, 95%, 90%, and 98%, respectively. The risk factors for asthma (e.g., allergic rhinitis) that were identified either by NLP or the abstractor were the same.

Conclusions: Asthma ascertainment through NLP should be considered in the era of EMRs because it can enable large-scale clinical studies in a more time-efficient manner and improve the recognition and care of childhood asthma in practice.

At a Glance Commentary

Scientific Knowledge on the Subject

Automated chart review using natural language processing for asthma ascertainment through electronic medical records reduces intrarater and interrater variability and inconsistency (i.e., methodologic heterogeneity) in asthma ascertainment.

What This Study Adds to the Field

In the era of electronic medical records, automated chart review using natural language processing helps clinicians and researchers identify children with recurrent asthma symptoms in a timely and consistent manner.

Asthma is the most common chronic illness in childhood, affecting 4–17% of children in the United States (1–5) and 2.8–37% of children worldwide, depending on the countries included in the estimate (6). Despite the availability of evidence-based guidelines for asthma management and effective asthma therapies, asthma continues to cause a significant morbidity and burden to our society (7).

Important concerns in current asthma care and research regarding identifying patients with asthma include the use of inconsistent asthma criteria (i.e., no standardized diagnostic criteria for asthma), asthma ascertainment process (e.g., physician-diagnosed asthma, symptom-based questionnaire, or billing codes), and sampling frame (e.g., different source population for cases [allergy clinic] and control subjects [primary care]). These challenges have led to inconsistent results of studies including genome-wide association studies, clinical trials, and biomarker studies (8–16). To more accurately assess clinically relevant information, such as symptoms and risk factors for asthma, manual chart review must be used because such information is typically embedded in text in electronic medical records (EMRs). To overcome the limitations of structured data (e.g., poor sensitivity [31%] of International Classification of Diseases [ICD] codes), manual chart review based on existing predetermined criteria has been widely used for asthma epidemiologic studies (17–21). However, this method is challenging, if not infeasible, especially for large-scale studies because applying asthma criteria to great volumes of medical records requires extraordinary effort and costs.

Growing deployments of EMR systems have established large practice-based longitudinal datasets, which allow for the identification of patient cohorts for epidemiologic investigations and population management. Natural language processing (NLP), which can extract information from narrative text automatically, has received a great deal of attention and has played a critical role in secondary use of the EMR for clinical and translational research (22–31). Previously, we developed an NLP algorithm, NLP-PAC, to automatically apply the Predetermined Asthma Criteria (PAC) for asthma ascertainment with significantly improved sensitivity (85%) when compared with structured data, such as ICD-9 codes (31%) (26). Using NLP-PAC has the potential to address some of the current challenges in asthma research and care because applying the NLP algorithm to large EMR datasets will enable large-scale clinical studies in asthma.

Herein, we applied NLP-PAC to an independent birth cohort with the purpose of further assessing both criterion and construct validity of the algorithm. Some of the results of our study have been previously reported in the form of an abstract (32).

Methods

Study Setting

Rochester, Minnesota, is centrally located in Olmsted County, and health care here is virtually self-contained within the community. Under the auspices of the Rochester Epidemiology Project, which links all inpatient and outpatient clinical diagnoses and information from every episode of care to each patient and health care provider, approximately 95% of Olmsted County residents’ medical records have been available for research purposes since 1966, allowing high fidelity of longitudinal studies (33). This resource has been electronically available at the Mayo Clinic since 1997 (i.e., the inception of EMR at Mayo Clinic).

Study Design and Subjects

This was a retrospective birth cohort study. The main aim of this study was to validate two aspects of our present NLP-PAC: criterion validity and construct validity (see Statistical Analysis). The accuracy of NLP-PAC (26) has been refined on the training cohort (n = 430), which is a random sample of the 2002–2006 birth cohort, Rochester, Minnesota, who had been enrolled in a previous asthma study (34). In this study, we evaluated the validity of the current version of NLP-PAC using an independent test cohort, by randomly sampling 500 subjects from the 1997–2007 Mayo Birth Cohort (n = 8,525) who were born at Mayo Clinic, enrolled in primary pediatric care at Mayo Clinic Rochester, and had EMR data available for applying the algorithm. The inclusion criteria were (1) a member of the Mayo Birth Cohort, (2) presence of research authorization for using medical record for research, (3) Olmsted County residency during the study period, (4) children who did not potentially have asthma-related medical records (e.g., diagnosis of asthma, bronchiolitis, pneumonia, wheezing) outside Mayo Clinic, and (5) children without any medical conditions fulfilling the exclusion criteria in Table 1. There was no requirement for a minimum follow-up to be included in the study.

Table 1.

Predetermined Asthma Criteria

Patients were considered to have definite asthma if a physician had made a diagnosis of asthma and/or if each of the following three conditions were present, and they were considered to have probable asthma if only the first two conditions were present:

1. History of cough with wheezing, and/or dyspnea; or history of cough and/or dyspnea plus wheezing on examination

2. Substantial variability in symptoms from time to time or periods of weeks or more when symptoms were absent

3. Two or more of the following:

• Sleep disturbance by nocturnal cough and wheeze

• Nonsmoker (14 yr or older)

• Nasal polyps

• Blood eosinophilia higher than 300/μl

• Positive wheal and flare skin tests; or elevated serum IgE

• History of hay fever or infantile eczema; or cough, dyspnea, and wheezing regularly on exposure to an antigen

• Pulmonary function tests showing one FEV1 or FVC less than 70% predicted and another with at least 20% improvement to an FEV1 of higher than 70% predicted; or methacholine challenge test showing 20% or greater decrease in FEV1

• Favorable clinical response to bronchodilator

Patients were excluded from our previous study if any of these conditions were present:

• Pulmonary function tests that showed FEV1 to be consistently below 50% predicted or diminished diffusion capacity

• Tracheobronchial foreign body at or about the incidence date

• Hypogammaglobulinemia (IgG <2.0 mg/ml) or other immunodeficiency disorder

• Wheezing occurring only in response to anesthesia or medications

• Bullous emphysema or pulmonary fibrosis on chest radiograph

• PiZZ alpha-1 antitrypsin

• Cystic fibrosis

• Other major chest disease, such as juvenile kyphoscoliosis or bronchiectasis

Predetermined Asthma Criteria

Drs. John Yunginger and Charles Reed, renowned researchers and clinicians for asthma, developed and validated the original PAC for retrospective studies among children and adults based on chart review (Table 1) (35). To our knowledge, these are the only existing predetermined criteria for asthma that determines asthma status and the index date of incident asthma retrospectively based on medical records. As defined by PAC, most cases of probable asthma (85%) became definite asthma over time (35, 36). PAC was found to have high reliability, and extensive epidemiologic work for asthma has used PAC showing the excellent construct validity in identifying known risk factors for asthma and asthma-related adverse outcomes (e.g., microbial infections) (35–45). Index date was defined as the date when the PAC was met for the first time.

NLP-PAC for PAC

The development of NLP algorithm for PAC was previously described in detail (26, 27), and a high-level diagram of the system is depicted in Figure 1. Briefly, there are two components in NLP-PAC: the text processing component, which extracts concepts in PAC (delineated in Table 1) from medical records; and the asthma status ascertainment component, which classifies asthma status at a patient level using pattern-based rules, assertion status (e.g., nonnegated [had wheezing vs. denied wheezing], associated with patient [not family history]), and section constraints (e.g., diagnosis). Some primary concepts were combined into secondary concepts to meet the criteria (e.g., “wheezing” and “coughing”). The algorithm was implemented using the open-source NLP pipeline MedTagger (http://ohnlp.org/index.php/MedTagger) developed by Mayo Clinic (46).

Figure 1.

A high-level diagram of the system (i.e., NLP-PAC). There are two components in NLP-PAC: the text processing component extracts concepts in PAC delineated in Table 1 from medical records, and the asthma status ascertainment component classifies asthma status at a patient level using pattern-based rules, assertion status (e.g., nonnegated, associated with patient), and section constraints (e.g., diagnosis). EMR = electronic medical records; NLP = natural language processing; PAC = predetermined asthma criteria.

Asthma Ascertainment by NLP-PAC and Chart Review of EMR by Abstractor

For each patient, we retrieved medical records that were available up to September 1, 2015. Only EMR narratives were used by both NLP-PAC and abstractor to ascertain asthma status. After completing asthma ascertainment by NLP-PAC and the abstractor using the training cohort (34), we performed an error analysis for false positives (i.e., NLP indicates “yes” for asthma, but abstractor indicates “no”) and false negatives (i.e., vice versa) to revise and refine NLP-PAC through a reiterative process. Any discrepancies (i.e., false positives and false negatives) were adjudicated by an independent reviewer (M.A.P., an allergy specialist). We applied this NLP-PAC to the test cohort for validation. Chart review of EMR by an abstractor was performed independently. For chart review of the test cohort’s EMR, only one abstractor (M.C.R.) ascertained asthma status by PAC after checking interrater agreement with an independent rater (physician, C.-I.W.) showing 100% agreement using five random samples. The data abstractor was blinded to the asthma status by the NLP algorithm throughout the data collection phase. Once they finished asthma ascertainment, an independent reviewer (C.-I.W.) reviewed charts of subjects with discrepancy between gold standard (i.e., abstractor’s review of EMR) and NLP-PAC, and reconciled the discrepancy and confirmed it with an allergist (M.A.P.).

Other Variables

To assess construct validity, one abstractor (A.S.) collected all pertinent variables known to be risk factors for asthma. After checking interrater agreement with another rater (C.-I.W.), both raters showed 100% agreement for each variable for a random sample of five subjects such as a family history of asthma; a history of other atopic diseases, such as allergic rhinitis or eczema; maternal smoking during pregnancy and household smoking exposure after birth; cesarean section; breastfeeding; and birth weight. The data sources for these variables included birth certificates, well-child visit notes, and the clinical note sections of family history and final diagnosis.

Statistical Analysis

In our present study, we summarized the characteristics of each study subject. Performance of NLP-PAC was assessed for criterion and construct validity. For criterion validity, performance of the algorithm was assessed by using agreement rate, kappa index, sensitivity, specificity, positive predictive value, and negative predictive value for concordance in asthma status between NLP-PAC and manual chart review as a gold standard. Construct validity was tested using logistic regression models by assessing the association of NLP-PAC results with the known risk factors for asthma, because asthma status ascertained by NLP-PAC is expected to be correlated with the known risk factors for asthma if it captures the underlying construct (i.e., asthma). The construct validity of NLP-PAC was compared with that of EMR chart review by an abstractor. Odds ratios and their corresponding 95% confidence intervals were presented. All statistical analyses were performed using JMP statistical software package version 10 (SAS Institute, Inc., Cary, NC).

Results

Characteristics of Study Subjects

The characteristics of study subjects are summarized in Table 2. Out of 500 children in the test cohort, three were excluded because of conditions that may make it hard to differentiate asthma-related wheezing from others, such as mild pectus excavatum with respiratory symptoms, tracheomalacia, or paradoxical vocal cord motion. Among the 497 eligible study subjects, 255 (51%) were males, 383 (77%) white persons, and the median age at last follow-up date was 11.5 years (interquartile range, 9.0–14; range, 4.7–17.9). All study subjects had more than 4 years of follow-up, and each subject had multiple EMR notes documented by multiple health care providers (e.g., nurse note, physician note) during study follow-up period. The median (interquartile range) numbers of EMR notes per subject were 100 (73–140) and 63 (44–92) among subjects with and without asthma, respectively (P < 0.001).

Table 2.

Characteristics of Study Subjects

Variables	n = 497
Age at last follow-up date, yr, median (IQR)	11.5 (9.0–14.5)
Male	255 (51%)
White	383 (77%)
Allergic rhinitis	84 (16%)
Eczema	147 (29%)
Family history of asthma	131 (26%)
Smoking during pregnancy	31 (6%)
Smoking exposure after birth	32 (6%)
Breastfeeding	361 (78%)
Cesarean section	113 (22%)
Birth weight, kg, median (IQR)	3.3 (3.0–3.7)

Definition of abbreviation: IQR = interquartile range.

Concordance in Asthma Status between NLP-PAC and Chart Review of EMR by Abstractor (Criterion Validity)

The results are summarized in Table 3. NLP-PAC identified 158 subjects meeting asthma criteria, whereas manual chart review identified 147 subjects with 131 by both approaches. Kappa index and agreement for asthma status between NLP-PAC and chart review by abstractor were 0.91 and 0.96, respectively, suggesting excellent agreement. Sensitivity, specificity, positive predictive value, and negative predictive value for NLP-PAC in asthma ascertainment using chart review by abstractor as the gold standard showed high predictive values (97%, 95%, 90%, and 98%, respectively). These results were similar between children with and without lung function test among those aged 6 years or older (Table 3).

Table 3.

Agreement for Asthma Status between NLP Algorithm and Manual Chart Review (Gold Standard) (n = 497)^*

Criterion Validity	Unweighted Cohen’s Kappa	Overall Agreement	Sensitivity	Specificity	Positive Predictive Value	Negative Predictive Value
All subjects (n = 497)	0.91	0.96	143/147 (97%)	335/350 (95%)	143/158 (90%)	335/339 (98%)
Children ≥6 yr (n = 491)	0.90	0.96	143/147 (97%)	329/344 (95%)	143/158 (90%)	329/333 (98%)
Lung function test, yes (n = 51)	0.92	0.98	43/43 (100%)	7/8 (87%)	43/44 (97%)	7/7 (100%)
Lung function test, no (n = 440)	0.89	0.95	100/104 (96%)	322/336 (95%)	100/114 (87%)	322/326 (98%)

Definition of abbreviation: NLP = natural language processing.

Sensitivity, specificity, positive predictive value, and negative predictive value were calculated as performance of NLP with manual chart review as a gold standard. For example, for the sensitivity among all subjects, 147 was the number of patients whose asthma status was ascertained as “yes” by manual chart review, and 143 was the number of patients whose asthma status was ascertained as “yes” by the NLP among 147 subjects ascertained by manual chart review.

^*Asthma prevalence of 31%.

Association of Asthma Status of NLP-PAC and Chart Review of EMR by Abstractor with the Known Risk Factors (Construct Validity)

The known risk factors for asthma identified by NLP were the same as the ones identified by the abstractor. Children with asthma determined by NLP-PAC had higher odds of having a family history of asthma, a history of allergic rhinitis and eczema, maternal smoking during pregnancy, smoking exposure after birth, and no breastfeeding history compared with those without asthma (P < 0.05 in each), but not for cesarean section, or low birth weight (Table 4). Asthma status by manual chart review showed similar results in terms of the association with known risk factors for asthma, except eczema with marginal significance.

Table 4.

Associations of Asthma Status Determined by NLP and Abstractor with Known Risk Factors for Asthma (n = 497)

	By NLP				By Manual Chart Review
	No Asthma (n = 339)	Asthma (n = 158)	OR (95% CI)	P Value	No asthma (n = 350)	Asthma (n = 147)	OR (95% CI)	P Value
Age, yr, median (IQR)*	11.2 (8.6–13.9)	12.4 (9.5–15.4)	1.0 (1.0–1.1)	0.01	11.1 (8.6–13.9)	12.6 (9.8–15.5)	1.0 (1.0–1.1)	0.001
Male, n (%)	165 (48)	90 (56)	1.3 (0.9–2.0)	0.08	172 (49)	83 (56)	1.3 (0.9–1.9)	0.13
White, n (%)	263 (77)	120 (75)	0.9 (0.5–1.4)	0.68	271 (77)	112 (76)	0.9 (0.5–1.4)	0.76
Allergic rhinitis, n (%)	38 (11)	46 (29)	3.2 (2.0–5.2)	<0.001	40 (11)	44 (29)	3.3 (2.0–5.3)	<0.001
Eczema, n (%)	90 (26)	57 (36)	1.5 (1.0–2.3)	0.03	96 (27)	51 (34)	1.4 (0.9–2.1)	0.07
Family history of asthma, n (%)	69 (20)	62 (39)	2.5 (1.6–3.8)	<0.001	72 (20)	59 (40)	2.5 (1.7–3.9)	<0.001
Smoking during pregnancy (missing: 10),† n (%)	13 (3)	18 (11)	3.1 (1.5–6.6)	0.001	15 (4)	16 (11)	2.7 (1.2–5.6)	0.006
Smoking exposure after birth (missing: 36),‡ n (%)	15 (4)	17 (10)	2.3 (1.1–4.8)	0.01	14 (4)	18 (12)	3.0 (1.4–6.2)	0.001
Breastfeeding (missing: 38), n (%)	255 (81)	106 (72)	0.5 (0.3–0.9)	0.01	263 (81)	98 (72)	0.5 (0.3–0.9)	0.02
Cesarean section, n (%)	78 (23)	35 (22)	0.9 (0.6–1.4)	0.83	81 (23)	32 (21)	0.9 (0.5–1.4)	0.73
Birth weight, kg, median (IQR)	3.3 (3.0–3.7)	3.3 (3.0–3.7)	0.9 (0.7–1.3)	0.89	3.3 (2.9–3.7)	3.3 (3.0–3.7)	1.0 (0.7–1.3)	0.92

Definition of abbreviations: CI = confidence interval; IQR = interquartile range; NLP = natural language processing; OR = odds ratio.

^*Age at the last follow-up date.

^†Maternal smoking status during pregnancy.

^‡Household smoking exposure status between ages 0 and 6 years.

Comparison of Time Efficiency between NLP-PAC and Abstractor’s Chart Review–based Asthma Ascertainment

We compared time spent ascertaining asthma status between NLP-PAC versus chart review of EMR by abstractor to assess time efficiency. It took 384 hours for data abstractors to complete chart review of EMR for asthma ascertainment of 430 study subjects, whereas it took 22 minutes (2.3-GHz single laboratory top) to run NLP-PAC for the same subjects. The time spent running NLP-PAC was calculated only for running the algorithm, not including time spent developing NLP-PAC. These findings demonstrate that after development, the NLP algorithm can enable large scale clinical studies in a highly time-efficient manner.

Discussion

To our knowledge, this is the first study that demonstrates that asthma status ascertained by NLP algorithm using EMR has excellent concordance with chart review of EMR by abstractor (i.e., criterion validity) and is associated with the known risk factors for asthma (i.e., construct validity). Our study results suggest feasibility of determining asthma status by an NLP algorithm. Although our findings need to be replicated by future studies with a larger sample size using different EMR systems, they suggest the huge potential of leveraging NLP for asthma care and research in the EMR and big data era.

Because literature on determination of asthma status by NLP does not currently exist, it is difficult to compare our study findings with others. We previously reported the performance of the original NLP algorithm (26), in which the results showed that sensitivity, specificity, positive predictive value, and negative predictive value were 81%, 95%, 84%, and 94%, respectively (asthma prevalence is 31% in this cohort). Although the original study showed feasibility and reasonable performance of the NLP algorithm, the original NLP algorithm for asthma ascertainment was based on a small convenience sample (n = 112) and did not assess construct validity. The typical approach for assessing performance of NLP in the medical informatics literature was determining criterion validity against chart review of EMR by abstractor (e.g., sensitivity, specificity, positive and negative predictive value, area under the curve, and F-measure). This approach, however, is used for outcomes with clear-cut laboratory definitions or imaging/biopsy definitions, such as diabetes, hypercholesterolemia, or cancer (29–31). In this present study, we examined construct validity by assessing the association of asthma status determined by NLP versus chart review of EMR by abstractor for those known risk factors for asthma. We not only significantly improved criterion validity, but we also successfully demonstrated the association of asthma status determined by our NLP algorithm with the known risk factors for asthma, such as allergic rhinitis or eczema. Some risk factors for asthma were not associated with asthma status by NLP in this study, such as cesarean section and low birth weight, which showed inconsistent association with asthma in the literature, likely caused by the weaker effect of these risk factors on asthma or measurement error of these variables rather than poor performance of NLP (44, 47, 48).

In the present validation study, we improved the way that the NLP algorithm captured and interpreted key words and sentences of PAC in a way to reduce false positives of NLP. For example, one of the main sources for false positives of NLP is a negated sentence, such as “Patient had wheezing” and “Patient denied wheezing,” which cannot be differentiated with a simple key search function, whereas NLP was designed and improved to rule out those negated sentences. Furthermore, hypothetical sentences (e.g., as part of patient instruction, such as “cold medicine in case of cough”) and nonpatient issues (e.g., sister had wheezing) have been trained using abstractors’ annotations, and thus, NLP performance was improved.

There is one noteworthy finding in our study. When we performed an error analysis and disconcordant cases between the NLP algorithm and data abstractor were adjudicated by an independent reviewer, we discovered that the NLP algorithm (computer) was correct for some false-positive cases. It suggests that data abstractors might potentially miss some important asthma-related symptom events embedded in texts of medical records. Although this error by data abstractors is likely to be nondifferential, it could be escalated if the volume of medical records to be reviewed was larger. Another advantage of NLP algorithm for asthma ascertainment is its capability of identifying the asthma index date (the date when PAC was met for the first time), which allows temporality assessment in epidemiologic studies. In the present study, among the 147 subjects with asthma ascertained by chart review of EMR by abstractor, 118 asthma index dates by NLP (80%) were within 4 weeks of those defined by chart review of EMR by abstractor, including 102 (69%) with exact same index date, supporting the utility of the NLP algorithm for determining a time component of events of interest, which is essential to determine the association between exposures and outcomes in observational studies (17–19).

There have been prior attempts at applying NLP in asthma research. Himes and coworkers (22) identified risk factors for asthma exacerbation, such as race and body mass index, using a logistic regression model, and Zeng and coworkers (23) developed an algorithm to extract asthma as a principal diagnosis from clinical notes (22, 23). Even 20 years ago, there were studies using NLP for monitoring patient care by extracting asthma-related information from discharge summaries or outpatient progress notes (24, 25). However, these studies focused on text searching capabilities of NLP (i.e., record level) but did not apply the text processing and classification components of NLP at a patient level. Therefore, our presented work is the first report for applying an NLP approach to text extraction, processing, and classification of patients for asthma status based on a predetermined criteria. This NLP approach will help clinicians and researchers address the limitations of using structured data (e.g., poor sensitivity of ICD codes in identifying asthma), labor-intensive manual chart review, self-reported data or symptoms (recall bias), and biomarkers or laboratory tests (e.g., impracticality for large-scale studies). This NLP-based approach for identifying patients with asthma will help maximize clinical benefits of NLP for asthma care in practice along with other NLP algorithms for monitoring asthma care, minimizing the chance of a delayed diagnosis of asthma (23–25, 49).

The main strength of our study is the epidemiologic advantages of our study setting and design in conducting retrospective studies, which enabled us to capture all inpatient and outpatient asthma-related events for this present study. Also, our study is a birth cohort study, which allowed us to follow subjects longitudinally since birth. Additional strengths of this study include evolution of NLP algorithm through incorporating free text data (e.g., asthma symptom). Our NLP algorithm has the capability to determine a time component of events of interest, such as index date of asthma, which are important for assessing relationships of asthma and other events or for determining temporal trends of asthma. In addition, our study highlights the potential use of NLP algorithms for ascertaining asthma and other chronic conditions, such as rheumatoid arthritis, as a clinical decision support tool (50, 51).

The main limitation was that we limited our analysis to study subjects who were born at and received medical care from Mayo Clinic for validation purposes. Future studies need to address whether our NLP algorithm for PAC can be applied to different EMR systems with similar validity, which our group is currently working on. Another limitation of this study is a potential misclassification of asthma status by PAC, because the presence of prebronchodilator airflow obstruction or airway hyperresponsiveness is not required to be considered asthmatic by PAC (which are not routinely performed in asthma care among children) (52). When we examined the criterion validity among children with and without lung function tests, it did not significantly affect the results (98% with lung function tests vs. 95% without). Although we acknowledge the potential misclassification of asthma because of the lack of lung function tests in our study, NLP algorithm for PAC may still be more suitable for large-scale clinical research for asthma than commonly used methods for asthma ascertainment in the current literature, such as self-report and ICD codes. Despite the small number of subjects, currently, the identification of children meeting the exclusion criteria in Table 1 relies solely on abstractors. Developing an NLP algorithm for exclusion criteria might be necessary for consistency in the future. Finally, our study results need to be replicated in a different study setting to ascertain the accuracy of the NLP algorithm. After completing development, validation, and optimization, we intend to explore various options to share and disseminate this algorithm with the research community.

In conclusion, the NLP approach for asthma ascertainment is a useful tool for asthma research and care in the era of the EMR and big data because it enables large-scale clinical studies and population management. It significantly improves cost and time efficiency in identifying children with asthma symptoms, while also improving methodologic heterogeneity in asthma ascertainment.