Evaluating early-life asthma definitions as a marker for subsequent asthma in an electronic medical record setting

Abstract

Background

Case definitions for asthma incidence in early life vary between studies using medical records to define disease. This study assessed the impact of different approaches to using medical records on estimates of asthma incidence by age 3 and determined the validity of early-life asthma case definitions in predicting school-age asthma.

Methods

Asthma diagnoses and medications by age 3 were used to classify 7,103 children enrolled in Kaiser Permanente Georgia according to 14 definitions of asthma. School-age asthma was defined as an asthma diagnosis between ages 5 and 8. Sensitivity (probability of asthma by 3 given school-age asthma), specificity (probability of no asthma by 3 given no school-age asthma), positive and negative predictive value (probability of (no) school-age asthma given (no) asthma by 3), and likelihood ratios (combining sensitivity and specificity) were used to determine predictive ability.

Results

9.0% to 35.2% of children were classified as asthmatic by age 3 depending on asthma case definition. Early-life asthma classifications were more specific than sensitive and were better at identifying children who would not have school-age asthma (negative predictive values: 80.7% to 86.6%) than at predicting children who would have school-age asthma (positive predictive values: 43.5% to 71.5%).

Conclusions

Choice of case definition had a large impact on the estimate of asthma incidence. While ability to predict school-age asthma was limited, several case definitions performed similarly to clinical asthma prediction tools used in previous asthma research (e.g., the Asthma Predictive Index).

INTRODUCTION

Asthma development often begins early in life with an estimated 50-80 percent of children who have asthma experiencing symptoms before age five (1). It is difficult to diagnose asthma in young children due to variable and non-specific symptoms and lack of reliable objective testing. It is also challenging to distinguish children who will experience persistent asthma throughout childhood from those with transient wheeze. Despite these complications, extensive research focuses on asthma in early childhood, requiring investigators to develop case definitions for incident asthma in early life.

There is both clinical and research interest in using early-life respiratory symptoms to identify children who will experience persistent asthma in later childhood. Previous studies have created and evaluated the performance of clinical asthma prediction tools to identify children in early life who are at high risk of having persistent asthma or wheeze at school age. The Asthma Prediction Index (API), which consists of both loose and stringent indices, was developed in 2000 using the Tucson Children's Respiratory Cohort and is a popular clinical prediction tool (2). Additional prediction indices developed using birth cohorts include the Isle of Wright score and the PIAMA (Prevention and Incidence of Asthma and Mite Allergy) risk score (3, 4). These predictive indices require information that can be prospectively collected by clinicians in the interest of patient care such as results of blood work and skin prick tests. The inclusion of detailed clinical parameters make these indices not well-suited for use in large retrospective studies that rarely have access to such information on all individuals. It is unknown whether retrospective studies that lack detailed clinical information can also predict who will experience persistent asthma at school age. This ability would be valuable since large studies have the potential to shed light on causes of asthma that may be missed by smaller clinical studies with less statistical power.

There is tremendous variability between case definitions for early-life incident asthma among studies using medical records or administrative claims data to define disease. Case definitions differ in the quantity and types of diagnoses and medications required to classify a child as asthmatic. In 2005, Dombkowski and colleagues used Medicaid data to assess differences between prevalent asthma case definitions for use in surveillance among children ages 18 years and younger (5). Similar to earlier research in a different Medicaid population (6), they found that childhood asthma prevalence was highly dependent on the definition used. It is uncertain whether the same variability in estimated disease proportion would be observed for case definitions intended to classify incident asthma in non-Medicaid pediatric populations, such as among children enrolled in health maintenance organizations.

The goals of the present study are to fill some of this knowledge gap by comparing different cumulative incident asthma case definitions in the first three years of life and assessing their ability to predict asthma at school age. Specifically, this study addresses the following objectives in a birth cohort of children enrolled in Kaiser Permanente Georgia: 1) Assess the impact of different approaches to using medical records to estimate the cumulative incidence of asthma by age three. 2) Determine the validity of these early-life asthma case definitions, which exclusively use information available in medical records, in predicting school-age asthma. This analysis seeks to identify a case definition for asthma in early life that minimizes disease misclassification when used as a proxy for asthma at school age.

METHODS

The Kaiser Air Pollution and Pediatric Asthma (KAPPA) Study is a retrospective birth cohort of children born between 2000 and 2010 enrolled in Kaiser Permanente Georgia (KPGA) Health Maintenance Organization (HMO) for at least the first year of life. Ethics approval for the study was obtained from the Emory University and KPGA Institutional Review Boards. KPGA is an integrated health care system that provides medical care services to approximately 240,000 members in the metropolitan Atlanta area. The KAPPA study followed children from birth until September 2013 and was developed to assess the effects of air pollution exposure in infancy on childhood asthma incidence. This analysis was restricted to KAPPA children who were followed until at least age six (born between 2000 and 2007). Among the 18,488 children who met this requirement, this analysis was completed using the subgroup of 7,103 children enrolled in KPGA continuously (allowing up to 90 day enrollment gaps). We used information from KPGA electronic medical records and administrative databases to examine 14 different case definitions for early-life incident asthma. Table 1 contains definitions of terms used in the case definitions and Table 2 includes the case definitions assessed. Several of these case definitions are used either exactly or with slight variations (e.g., modified medication list, specific timing of events) in previous studies (7-13).

TABLE 1.

Diagnosis and medication definitions

Outcome	Definition
Asthma diagnosis	ICD-9 code 493.XX
Wheeze diagnosis	ICD-9 code 786.07
Acute asthma diagnosis	a) emergency department or inpatient asthma diagnosis or b) asthma diagnosis with status asthmaticus or acute exacerbation (ICD-9 codes 493.01, 493.02, 493.11, 493.12, 493.21, 493.22, 493.91, 493.92)
Asthma controllera	Aminophylline, beclomethasone diproprionate, budesonide, budesonide/formoterol fumarate, cromolyn sodium, fluticasone propionate, fluticasone/sameterol, mometasone furoate, montelukast sodium, salmeterol xinafoate, theophylline anhydrous, tiotropium bromide, triamcinolone acetonide
Asthma reliever	Albuterol, albuterol sulfate, ipratropium bromide, ipratropium/albuterol sulfate, levalbuterol, metaproterenol sulfate
Asthma-related medication	Dispensing of any asthma controller or reliever

^aUnderlined medications contain a steroid

TABLE 2.

Early-life asthma case definitions

Case Definition	Criteria Needed	Asthma diagnosis (n)	Other diagnoses	Asthma-related medication dispensings required (n)
1	Any	1	1 wheeze diagnosis
2	All	1
3	All	2
4	All	3
5	Any	2	1 acute asthma diagnosis
6	Any	1		2
7	All	1		1
8	All	1		2
9	Any	1		2 (at least 1 steroid)
10	All	1		2 (at least 1 steroid)
11	Any	1		1 controller
12	All	1		1 controller
13	All	1		2 reliever or 1 controller
14	Any	3	1 asthma-related ED visit or hospitalization	1 if in same year as 1 asthma diagnosis

These are the minimum required events for each case definition using events by age 3. Only 1 diagnosis per day counted. ED=emergency department. Definitions of all terms are included in Table 1.

For example, in order to meet case definition 1 a child had to have at least 1 asthma diagnosis or 1 wheeze diagnosis, in order to meet case definition 3 a child had to have at least 2 asthma diagnoses, and in order to meet case definition 7 a child had to have at least 1 asthma diagnosis and 1 asthma-related medication dispensing.

Incident asthma in early life was classified for each child using events from the medical record between birth and age three. We then individually assessed the ability of each of the 14 definitions of early-life incident asthma to predict school-age asthma status, defined as at least 1 asthma diagnosis (ICD-9 code 493.XX) between ages five and eight. Although asthma diagnoses at school age are subject to measurement error, they are more reliable than earlier diagnoses and indicate evidence of continued asthma morbidity (1). Predictive ability was measured using sensitivity (probability of incident asthma by age 3 given school-age asthma), specificity (probability of no incident asthma by age 3 given no school-age asthma), positive predictive value (PPV) (probability of school-age asthma given incident asthma by age 3), negative predictive value (NPV) (probability of no school-age asthma given no incident asthma by age 3) and kappa statistics ((observed agreement between early-life and school-age classifications minus expected agreement) divided by 1 minus expected agreement) (14). Likelihood ratio tests, which combine sensitivity and specificity to assess overall prediction accuracy, were also calculated: positive likelihood ratio (sensitivity divided by one minus specificity) and negative likelihood ratio (one minus sensitivity divided by specificity) (15). All analyses were completed in SAS 9.3 (Cary, NC).

RESULTS

In this cohort of 7,103 children (Table 3), 1,705 children (24.0%) had an asthma diagnosis between ages five and eight (“school-age”). Using diagnoses and medication dispensings in the first three years of life, 2,719 children (38.3%) were classified as asthmatic by at least one case definition. Cumulative asthma incidence by age three ranged from 9.0% (definition 4) to 35.2% (definition 6) depending on the case definition used (Table 4).

TABLE 3.

Cohort Characteristics (n=7,103)

Characteristic	N (%)
Sex
Female	3,474 (48.9)
Male	3,629 (51.1)
Race/Ethnicity
Black	3,004 (42.3)
White	2,847 (40.1)
Other Racea	691 (9.7)
Missing Race	561 (7.9)
Hispanic Ethnicity	359 (5.1)
Maternal Education
<12th grade	91 (1.3)
High School/GED	737 (10.4)
Some College or more	4,330 (61.0)
Missing Education	1,945 (27.4)
Kaiser Permanente Enrollment Duration b
Enrolled through age 6	7,103 (100.0)
Enrolled through age 8	4,075 (57.4)
Year of Birth
2000 – 2001	2,273 (32.0)
2002 – 2003	2,130 (30.0)
2004 – 2005	1,482 (20.9)
2006 – 2007	1,218 (17.1)

^aIncludes the following racial groups: Asian, American Indian, Alaska Native, Native Hawaiian or other Pacific Islander, children identifying with more than one racial group

^bEnrollment through age 6 part of inclusion criteria. Children enrolled through age 8 are a subset of children enrolled through age 6. Reduction in sample size across follow-up reflects shorter follow-up time available for children born in later years of the study (e.g., a child born in 2005 could be at most 8 years old at the time of medical record data abstraction) as well as HMO enrollment attrition over time.

TABLE 4.

Early asthma classifications (using events by age 3) and prediction of school-age asthma (at least 1 asthma diagnosis between ages 5 and 8) among children enrolled in Kaiser Permanente Georgia (n=7,103)

Asthma definition applied to age 0-3 years	% Meeting definition by age 3	Sensitivity	Specificity	Positive Predictive Value	Negative Predictive Value	Positive Likelihood Ratio	Negative Likelihood Ratio	Kappa Statistic
1. 1 asthma or wheeze diagnosis	29.4%	57.6%	79.5%	47.0%	85.6%	2.8	0.5	0.34
2. 1 asthma diagnosis	22.5%	49.9%	86.2%	53.2%	84.5%	3.6	0.6	0.37
3. 2 asthma diagnoses	13.2%	35.4%	93.7%	64.1%	82.1%	5.6	0.7	0.34
4. 3 asthma diagnoses	9.0%	26.8%	96.6%	71.5%	80.7%	7.9	0.8	0.30
5. 2 asthma diagnoses OR 1 acute asthma diagnosis	14.2%	36.8%	93.0%	62.3%	82.3%	5.2	0.7	0.35
6. 1 asthma diagnosis OR 2 medication dispensings	35.2%	63.8%	73.8%	43.5%	86.6%	2.4	0.5	0.32
7. 1 asthma diagnosis AND 1 medication dispensing	21.7%	49.2%	87.0%	54.4%	84.4%	3.8	0.6	0.37
8. 1 asthma diagnosis AND 2 medication dispensings	19.8%	46.7%	88.7%	56.7%	84.0%	4.1	0.6	0.38
9. 1 asthma diagnosis OR 2 medication dispensings 1 of which must be a steroid	24.0%	52.2%	84.8%	52.1%	84.9%	3.4	0.6	0.37
10. 1 asthma diagnosis AND 2 medication dispensings 1 of which must be a steroid	11.7%	31.8%	94.6%	65.1%	81.5%	5.9	0.7	0.32
11. 1 asthma diagnosis OR 1 controller dispensing	24.4%	52.8%	84.6%	52.0%	85.0%	3.4	0.6	0.37
12. 1 asthma diagnosis AND 1 controller dispensing	12.1%	32.8%	94.5%	65.2%	81.7%	5.9	0.7	0.33
13. 1 asthma diagnosis AND (2 reliever dispensings OR 1 controller dispensing)	19.9%	47.0%	88.7%	56.8%	84.1%	4.2	0.6	0.38
14. Any of the following: a) 1 asthma diagnosis AND 1 medication dispensing in the same year, b) 1 asthma-related ED visit or hospitalization, c) 3 asthma diagnoses	21.6%	49.3%	87.1%	54.7%	84.5%	3.8	0.6	0.38

These are the minimum required events for each case definition. Only 1 diagnosis per day counted. ED=emergency department. 1,705 children in cohort (24%) have an asthma diagnosis between ages 5 and 8. Sensitivity: probability of incident asthma by age 3 given school-age asthma. Specificity: probability of no incident asthma by age 3 given no school-age asthma. Positive predictive value: probability of school-age asthma given incident asthma by age 3. Negative predictive value: probability of no school-age asthma given no incident asthma by age 3. Positive likelihood ratio: sensitivity divided by one minus specificity. Negative likelihood ratio: one minus sensitivity divided by specificity. Positive likelihood ratios >10 and negative likelihood ratios <0.1 are considered to be indicative of case definitions with high predictive value. Positive likelihood ratios between 2 and 10, and negative likelihood ratios between 0.5 and 0.1 indicate case definitions that may have some predictive value (15). Kappa Statistic: (observed agreement between early-life and school-age classifications – expected agreement)/(1-expected agreement) (14).

Across case definitions, the extent to which the asthma cases or non-cases at school age were misclassified varied. Overall, the tests were more specific than sensitive. In this population, with a school-age asthma prevalence of 24%, the early-life asthma classifications were far superior at ruling out school-age asthma (NPVs ranged from 80.7% to 86.6%) than they were at predicting school-age asthma (PPVs ranged from 43.5% to 71.5%). Across definitions, the positive and negative likelihood ratios would generally be considered as having poor to moderate predictive ability for a clinical test (15). We saw no evidence that prediction ability was dependent on child race or gender (Table S1). Kappa statistics ranged from 0.30 to 0.38 which can be considered fair agreement (14).

The impact of adding additional information to the case definition was mixed. Consider for example definition 7, at least 1 asthma diagnosis and 1 medication dispensing. Making the definition more complex by additionally classifying a child as asthmatic if they had 1 asthma-related emergency department visit or hospitalization or 3 asthma diagnoses (definition 14) resulted in little predictive benefit by any examined metric. However, changing definition 7 by specifying that the medication had to be a controller (definition 12), sharply decreased the percent of children classified as asthmatic by age 3 and resulted in an increase in specificity and PPV. Similar results were found specifying medication type in other definitions.

DISCUSSION

Using electronic medical records from a large HMO, we systematically examined different ways to classify asthma in early life and evaluated which case definitions were best able to predict children who will have evidence of school-age asthma. In this population, choice of case definition had a large impact on the estimate of asthma incidence in early life. Dombkowski and colleagues reached a similar conclusion when examining case definitions for prevalent asthma in a cohort of children enrolled in Medicaid (5). For example, kappa statistics between asthma classifications using events before age five and an asthma diagnosis in the subsequent year ranged from 0.28 to 0.40. We examined different asthma classifications than the Dombkowski study and had more time between initial classification and later disease, but observed similar kappa values (Table 4).

While none of the case definitions we examined consistently identified children who would be diagnosed with school-age asthma, their performance using limited medical record data was comparable to that of clinical asthma prediction tools which use more detailed health information. When using events by age three to predict active asthma at age six, the loose Asthma Predictive Index (API) has a sensitivity of 56.6%, and a specificity of 80.8%, which are very similar to the sensitivity and specificity of our definitions 1, 9, and 11. When validated at the same age, the stringent API has an almost identical sensitivity and specificity as our case definition 4 (stringent API sensitivity 27.5%, specificity 96.3%) (2). Similarities in performance also exist between our definitions and other clinical prediction tools. For example, when using a cut point of a severity score of 6, the Environmental and Childhood Asthma (ECA) severity index has almost identical prediction metrics to our case definition 13 (ECA sensitivity 51.5%, specificity 88.1%, PPV 54.3%, NPV 86.8%) (16). The similar performance between our medical records based predictors and clinical prediction tools may not be surprising given that clinicians use elements of clinical prediction tools to inform diagnoses. The generally poor predictive ability of our case definitions and of clinical prediction tools reflects the complex and often transient nature of early-life respiratory symptoms (17, 18). It may also result from asthma remission due to effective therapies and avoidance of exacerbating exposures. Despite their limitations, clinical asthma prediction tools have proved useful in research, for example in studies to identify lung function biomarkers and develop asthma therapies (19).

There is a high prevalence of school-age asthma in this cohort, with almost a quarter of children receiving at least one asthma diagnosis between ages five and eight. This prevalence is higher than Georgia state estimates; in 2010 it was estimated that among children ages five to nine years in Georgia 13.7% had current asthma and 20.4% had ever been diagnosed with asthma (20). The higher prevalence in our population can likely be explained partially by the use of medical records for classification which in comparison to parental report yields higher prevalence estimates for childhood asthma (21). Additionally, prevalence of asthma diagnoses has been found to be higher among insured than uninsured children (22). Over-diagnosis of asthma may also contribute to this high prevalence.

The 14 case definitions for early-life asthma that we examined are a subset of the many potential definitions one could choose. We did not examine definitions that use only information on medications, and not diagnoses, to determine whether a child has asthma. These definitions were excluded because medications used to treat asthma are also used to treat other conditions. We also did not examine incident asthma case definitions that considered whether a diagnosis was classified as primary in the medical record, because in our dataset we were unable to determine primary status for 83.9% of asthma diagnoses given to children in our cohort. While we are referring to this outcome as early-life asthma given the use of asthma ICD-9 diagnoses, we are cognizant that respiratory conditions before age six are not typically called asthma and continued wheezing may be a more appropriate term for these outcomes.

This analysis has several strengths and limitations. The KAPPA study is uniquely positioned to examine early-life asthma case definitions due to access to medical records on over 7,000 children insured by KPGA from birth until at least age six. The record-based classification used in this study, instead of the commonly used parental report, prevents recall bias from impacting results. The use of medication dispensings, rather than medication prescriptions, is a strength of this analysis since dispensings are expected to align more closely with actual medication intake. Limitations of using medical record data are the inability to account for variations in provider practices and lack of information on indication for medications. There is undoubtedly some misclassification of asthma status among children between ages five and eight. Even though reliability of asthma diagnoses increases as children age, this outcome is not perfect in determining school-age asthma status, particularly since it was determined using ICD-9 codes. Our analyses were restricted to children in the KAPPA cohort who were followed until age six. Results were comparable if we restricted the cohort to children enrolled through age eight. While prevalence of an early-life asthma diagnosis was similar between children in our analysis and children lost to follow-up (22.5% vs. 21.2%), it is possible that loss to follow-up impacted our findings.

This study was conducted in the KPGA population, an insured, primarily urban population in the southeastern U.S. Positive and negative predictive values are directly dependent on asthma prevalence; one would expect these values to differ when examining the performance of these case definitions in a population with a different school-age asthma prevalence. Other prediction metrics may also vary in different populations, particularly outside of an HMO setting. While we do not anticipate all of our results will generalize well to markedly different populations, we expect that the dependence of incidence estimates on case definition and the limited ability of early-life asthma to predict school-age asthma are likely generalizable to any pediatric population.

There is no perfect way to classify asthma status using medical records, particularly in early childhood. Given the challenges of asthma diagnosis in early life, misclassification in asthma research is unavoidable. Our analysis indicated that choice of case definition had a large impact on the estimate of asthma incidence in early life. This dependence has implications for the comparability of findings between studies that use different case definitions for childhood asthma. The results of this analysis emphasize the importance of completing sensitivity analyses to assess the impact of case definition choice on research results and to facilitate better comparisons across studies. Among the early-life asthma case definitions we examined, there was not an obvious choice as to which was best at predicting school-age asthma. Which definition is best-suited for future research may depend on the purposes of a given study. Several of our case definitions performed similarly to clinical asthma prediction tools, showing that asthma diagnoses and medications in early life can be used to predict asthma at school age with as much accuracy as can be obtained with some detailed clinical tools. The comparable predictive ability of our early-life asthma definitions combined with the unique advantages of large record-based studies highlight the potential for record-based studies to continue to advance our knowledge about asthma etiology.