Comparative ability of comorbidity classification methods for administrative data to predict outcomes in patients with chronic obstructive pulmonary disease

Abstract

Purpose

Administrative healthcare databases are used for health services research, comparative effectiveness studies, and measuring quality of care. Adjustment for comorbid illnesses is essential to such studies. Validation of methods to account for comorbid illnesses in administrative data for patients with chronic obstructive pulmonary disease (COPD) has been limited. Our objective was to compare the ability of the Charlson index, the Elixhauser method and the Johns Hopkins’ Aggregated Diagnosis Groups (ADGs) to predict outcomes in patients with COPD.

Methods

Retrospective cohorts constructed using population-based administrative data of patients with incident (n=216,735) and prevalent (n=638,926) COPD in Ontario, Canada, were divided into derivation and validation datasets. The primary outcome was all-cause death within one year. Secondary outcomes included all-cause hospitalization, COPD-specific hospitalization, non-COPD hospitalization, and COPD exacerbations.

Results

In both the incident and prevalent COPD cohorts, the three methods had comparable discrimination for predicting mortality (c-statistics in the validation sample of incident patients: 0.819 for the Charlson method vs. 0.822 for the Elixhauser method vs. 0.830 for the ADG method). All three methods had lower predictive accuracy for predicting non-fatal outcomes.

Conclusions

In a disease-specific cohort of COPD patients, all 3 methods allowed for accurate prediction of mortality, with the Johns Hopkins ADGs having marginally higher discrimination.

INTRODUCTION

Population-based cohorts are invaluable resources for health services research, comparative effectiveness studies, and describing the relative quality of care delivered by health care providers, hospitals or regions. Health administrative databases often provide coverage of entire populations or disease groups and contain comprehensive data on outcomes. However, data on risk factors in these cohorts are often limited due to the lack of detailed clinical information. Consequently, other methods to account for confounding and to reduce bias are necessary. However, it is important that these methods be validated prior to their widespread adoption. In observational comparative effectiveness research, it is particularly important to be able to distinguish the risk of the occurrence of outcomes, such as mortality, that is attributable to the exposure of interest (e.g. drug exposure or medical intervention) from the risk attributable to comorbidities, which often confound attempts to estimate the effect of disease-specific interventions on outcomes.

Several methods to classify comorbidities using administrative health care data exist. Charlson et al. developed a weighted index of comorbidities for predicting mortality, originally derived in hospitalized general medical patients and initially validated in female oncology patients¹¹. This index was subsequently adapted by Deyo et al. for use with the International Classification of Diseases (ICD-9-CM) diagnosis and procedure codes that are frequently used in electronic health care administrative databases and is ubiquitous in health services research¹². Similarly, Elixhauser et al. developed a method to classify comorbidities in hospitalized patients using diagnoses coded using the ICD-9-CM diagnosis codes in administrative data¹³. Both the Charlson comorbidity index and the Elixhauser coding scheme have been updated for use with the ICD-10 diagnosis classification scheme^26;30.

The Johns Hopkins Adjusted Clinical Group (ACG)® system is a person-focused diagnosis-based method of categorizing persons’ illnesses. The ACG system assigns each ICD code (-9 version, -9-CM version, or -10 version) to one of 32 diagnosis clusters known as Aggregated Diagnosis Groups (ADGs). Individual diseases or conditions are placed into a single ADG based on five clinical dimensions: duration of the condition, severity of the condition, diagnostic certainty, etiology of the condition, and specialty care involvement^21;28;31;33. Unlike the Charlson and Elixhauser comorbidity classification schemes, the ADG and ACG definitions do not rely only on the use of inpatient health administrative data, but also use ambulatory health care data. ICD codes within the same ADG are similar in both clinical criteria and expected need for healthcare resource. While each individual may have diagnoses belonging to between zero and 32 ADGs, each person is assigned to exactly one ACG. Persons within the same ACG are expected to have similar healthcare resource utilization. The term ACG refers to the overall comorbidity classification system, while the term ADG refers to determining the presence or absence of comorbidities in each of the 32 ADG categories.

While each of the above comorbidity classification schemes has been shown to perform acceptably in the general adult population⁶ and has been used in health services research, it is not clear which provides optimal classification of comorbidities in subpopulations with chronic diseases. Performance in disease-specific populations cannot be assumed from performance in the general population. For instance, a regression model developed to predict mortality in the general adult population was not well calibrated for predicting mortality in patients with schizophrenia³. Therefore, validation of comorbidity classification methods in disease-specific populations is necessary and important.

Chronic obstructive pulmonary disease (COPD) is a common chronic respiratory disease, affecting over 10% of the adult population^9;17, , with the one in four adults over the age of 35 years developing COPD during their lifetime¹⁸. Furthermore, it is the fourth leading cause of death in North America²⁰ and a major cause of hospitalization⁷. Comorbid illnesses are particularly common in COPD²² and cause a majority of COPD deaths^2;23, likely because COPD’s primary risk factor (smoking) is common to many other diseases and possibly also because systemic inflammation, a pathophysiologic feature of COPD¹⁴, contributes to extrapulmonary manifestations of other diseases. Because of its large burden in individuals and health care systems, it is crucial that we have evidenced-based strategies to help those affected. Health services research can provide some of this evidence if methods to reduce bias and confounding in this population are available. Thus COPD is a particularly salient disease within which to evaluate the accuracy of comorbidity classification and identification of baseline mortality risk. Moreover, administrative databases usually do not contain many important predictors of mortality and markers of COPD severity, including smoking history, lung function, body-mass index, and functional impairment.

Prior studies have compared the performance of these three methods to predict mortality in a general adult population⁶ and in two different populations of patients with chronic conditions (diabetes and schizophrenia)^3;4. The objective of the current study was to compare the relative performance of these comorbidity classification schemes for predicting clinical outcomes in patients with COPD.

METHODS

Data sources

We used five population-based administrative health care databases from the Canadian province of Ontario, deterministically linked at the patient level using an encrypted version of the health insurance number. The Ontario Chronic Obstructive Pulmonary Disease database contains information on all residents of Ontario age 35 years or older with physician-diagnosed COPD, identified by having one or more physician billing claims or one or more hospital discharges with COPD diagnostic codes 491(chronic bronchitis), 492 (emphysema), or 496 (chronic airway obstruction, not elsewhere classified) (ICD-9) or J41 (simple chronic bronchitis), J42 (unspecified chronic bronchitis), J43 (emphysema), or J44 (chronic obstructive pulmonary disease) (ICD-10). In a population-based case verification study, this COPD case definition algorithm was found to have a sensitivity of 85.0% and a specificity of 78.4%¹⁶. Details and examples of the use of this case definition can be found elsewhere^16;18. The Registered Persons Database (RPDB) contains basic demographic information on all Ontario residents who were ever eligible for Ontario’s universal health care insurance program, including date of birth, sex, date of death (if applicable). The Canadian Institute for Health Information (CIHI) Discharge Abstract Database (DAD) contains information on all hospital admissions in the province of Ontario. For each hospitalization record, there are 25 fields for recording diagnoses made on the patient during the course of the hospitalization. Since 2002, diagnoses have been coded using the ICD-10 coding scheme. The Ontario Health Insurance Plan (OHIP) physician billing database contains billing claims submitted by Ontario physicians to the provincial universal health insurance program, including a fee code describing the type of service provided and a single diagnosis code (a truncated ICD-9 code) denoting a reason for the service. The Ontario Mental Health Reporting System (OMHRS) contains data on patients in adult-designated inpatient mental health beds in general, provincial psychiatric, and specialty psychiatric facilities, including reasons for admission and discharge and psychiatric and non-psychiatric diagnoses.

Patients

We constructed separate cohorts of patients with incident and prevalent COPD. The incident cohort consisted of all patients with an initial diagnosis of COPD between April 1, 2004 and March 31, 2008. For each patient in the incident cohort, the index date was the date of the initial diagnosis of COPD. Patients were only considered to have incident COPD if they did not have any claims for COPD in the 5 years prior to their diagnosis date¹⁷. The prevalent cohort consisted of all residents of Ontario who had been diagnosed with COPD prior to January 1, 2007. January 1, 2007 served as the index date for each patient in the prevalent cohort.

For each patient, we identified all diagnoses associated with all hospital admissions from the CIHI DAD and all physician billing claims in the OHIP database for physician services provided in the two years prior to the index date. We calculated the Charlson comorbidity index^12;26 and the Elixhauser comorbidities^13;26 using data from the CIHI DAD for hospitalizations occurring in the two years prior to the index date. For the Elixhauser comorbidities, the OMHRS database was also used to identify mental health and addiction diagnoses. Following common practice, patients who had not been hospitalized in the previous two years had their Charlson score to zero and their values of each of the 30 Elixhauser comorbidities to absent. We used the Johns Hopkins ACG® software program to determine the presence or absence of each of the 32 ADGs using diagnoses recorded in either the CIHI DAD or in OHIP physician claims.

The study was approved by the Research Ethics Board of Sunnybrook Health Sciences Centre.

Statistical Methods

All of our analyses were conducted separately in the incident cohort and the prevalent cohort. SAS version 9.2 (SAS Institute, Cary NC) and R version 2.11.1 (The R Foundation for Statistical Computing, Vienna, Austria) were used for statistical analyses. In order to evaluate the performance of different comorbidity classification schemes to predict outcomes in a sample that was independent of the sample in which regression models were derived, we used a random number generator to divide each of the incident and prevalent cohorts into approximately equally sized derivation and validation samples. In the derivation sample, we created a separate logistic regression model for each comorbidity measure, containing age (assuming a linear relationship between age and the log-odds of the outcome), sex, and either the Charlson comorbidity score, 30 indicator variables denoting the presence of absence of the 30 Elixhauser comorbidities or 32 indicator variables representing the presence or absence of the 32 ADGs.

The primary outcome was the occurrence of death within 365 days of the index date. We examined four secondary outcomes: (i) any hospitalization within one year of the index date; (ii) any COPD or related disease hospitalization within one year of the index date; (iii) any non-COPD hospitalization within one year of the index date; (iv) any COPD exacerbation within 180 days of the index date. A COPD-specific hospitalization was defined to be any hospitalization in which the CIHI DAD recorded a diagnosis of COPD (ICD10 J41-J44), pneumonia or influenza (ICD10 J10-J18), acute bronchitis (ICD10 J20) or bronchitis (ICD10 J22 or J40). A COPD exacerbation was defined to have occurred if the patient filled a prescription for oral corticosteriods or a respiratory antibiotic within seven days of a physician visit that was associated with a diagnosis of bronchitis, pneumonia, influenza, emphysema, asthma, or other chronic obstructive pulmonary disease (ICD9 466, 486, 487, 491, 492, 493, or 496). Since data on prescription medication use from the Ontario Drug Benefits database is only universally available for those over the age of 65 years, this final secondary outcome was only examined in those patients over the age of 65 years.

From each model, we obtained the predicted probability of mortality for each patient in the validation sample. The discrimination of each model developed in the derivation sample was assessed in the derivation and validation samples using the c-statistic¹⁹. Model calibration, which assesses the concordance between predicted and observed mortality, was assessed in two ways. First, we divided the validation sample into approximately equally sized groups according to the predicted probability of mortality. Since the prevalent sample was substantially larger than the incident sample, we used 100 strata defined by the centiles of risk for the prevalent sample and 50 strata for the incident sample. Within each stratum, we determined both the mean predicted probability of mortality based on the logistic regression model and the observed probability of mortality. The relationship between observed and predicted mortality was then examined graphically. Second, we determined the calibration slope ²⁹. To do so, we took the regression coefficients from the final logistic regression model estimated in the derivation sample and applied them to the validation sample to obtain the log-odds of predicted mortality. We then used logistic regression to regress the observed mortality within one year of the index date in the validation sample on the log-odds of predicted mortality. The calibration slope can be conceptualized as the slope of the line relating observed to predicted mortality as the number of strata becomes arbitrarily large. Deviation of the calibration slope from unity denotes miscalibration.

RESULTS

The incident cohort consisted of 216,735 individuals between the ages of 35 and 99 years. The median age was 63 (quartiles: 51, 75) and 50% were women. Overall, 18,426 (8.5%) died within 365 days of their index date. The prevalent cohort consisted of 638,926 individuals between the ages of 35 and 100 years. The median age was 66 (quartiles: 55, 76) and 51% were women. Overall, 14,124 (2.2%) died within 365 days of their index date.

In the validation sample in the incident population cohort, the ADG model for predicting mortality had greater discrimination than the Elixhauser comorbidity model, which in turn had greater discrimination than the Charlson score model (c-statistics: 0.819 for the Charlson method vs. 0.822 for the Elixhauser method vs. 0.830 for the ADG method) (Table 1 and Figure 1(a)). However, it should be noted that these differences in discrimination are likely not qualitatively important. Similarly, all 3 models for predicting mortality displayed very good, and approximately similar, concordance between observed and predicted mortality, with calibration slopes of 1.0044 (95% CI: 0.9839, 1.0250) for the Charlson Score model, 0.9954 (95% CI: 0.9753, 1.0156) for the Elixhauser comorbidities model, and 0.9915 (95% CI: 0.9715, 1.0112) for the ADG model (Figure 2). In the prevalent population cohort, the 3 models showed similar discrimination (c-statistics: 0.807 for the Charlson model vs. 0.810 for the Elixhauser model vs. 0.816 for the ADG model) (Table 1 and Figure 1 (b)) and calibration slopes [0.9845 (95% CI: 0.9636, 1.0054) v. 0.9837 (95% CI: 0.9628, 1.0047) v. 0.9822 (95% CI: 0.9613, 1.0028)] (Figure 3). Lack of calibration was only suggested in those patients in the highest strata of predicted probability of mortality. In the incident cohort, the ADG method appeared to have modestly better calibration than the other 2 methods for higher-risk strata, while in the prevalent cohort, the ADG method appeared to have better calibration than the Charlson score for lower-risk strata.

Table 1.

C-statistics of different models for predicting 1-year mortality in COPD patients.

Outcome	Charlson model		Elixhauser model		ADG model
	Derivation sample	Validation sample	Derivation sample	Validation sample	Derivation sample	Validation sample
Incident cohort
One-year mortality	0.816	0.819	0.820	0.822	0.829	0.830
All-cause hospitalization	0.678	0.682	0.686	0.690	0.700	0.705
COPD hospitalization	0.717	0.721	0.725	0.729	0.727	0.730
Non-COPD hospitalization	0.630	0.632	0.639	0.640	0.669	0.671
COPD exacerbation	0.514	0.514	0.530	0.526	0.607	0.600
Prevalent cohort
One-year mortality	0.809	0.807	0.813	0.810	0.820	0.816
All-cause hospitalization	0.698	0.699	0.706	0.708	0.724	0.728
COPD hospitalization	0.754	0.752	0.779	0.778	0.774	0.777
Non-COPD hospitalization	0.644	0.646	0.653	0.654	0.687	0.691
COPD exacerbation	0.533	0.550	0.563	0.562	0.676	0.676

Each cell contains the c-statistic for assessing model discrimination between patients who did and did not experience the outcome of interest. The c-statistic ranges from 0.5 to 1, with higher scores indicating better discriminative ability.

Figure 1.

Receiver Operating Characteristic (ROC) curves of the three regression models in the incident and prevalent populations: Incident population (left panel) and prevalent population (right panel).

Figure 2.

Calibration plots in the incident COPD population: Comparison of observed vs. predicted mortality in 50 strata of risk.

Figure 3.

Calibration plots in the prevalent COPD population: Comparison of observed vs. predicted mortality in the centiles of risk.

In both cohorts, the three different comorbidity classification schemes displayed substantially lower predictive accuracy for predicting non-mortality outcomes (Table 1). For each outcome, the Charlson model displayed the lowest discrimination, while the ADG model displayed the highest discrimination. The largest differences between these models were seen for predicting COPD exacerbations. Discrimination for each of the four secondary outcomes was substantially lower than for the primary outcome.

DISCUSSION

All three of the comorbidity classification methods we examined were able to predict mortality accurately in both incident and prevalent cohorts. While differences in discrimination between the methods were negligible, the ADG method had slightly better discrimination and modestly better concordance between observed and predicted probabilities of mortality across the strata of risk than the other methods. Based on our findings, while all three methods may be acceptable for predicting mortality, the use of the ADG method may be preferable for use for risk-adjustment in observational studies of chronic diseases like COPD due to its modestly better concordance. A summary of the differences between the three methods is provided in Table 2.

Table 2.

Summary of comparison of different comorbidity classification schemes for predicting outcomes in patients with COPD.

Criterion	Summary of comparison
Discrimination for predicting mortality	All three methods had similar discrimination for predicting one-year mortality
Calibration for predicting mortality	All three methods displayed good calibration (concordance between predicted and observed risk of mortality). The ADG method had modestly better calibration in the higher risk strata in the incident cohort.
Predicting non-fatal outcomes	All three methods displayed poorer discrimination was observed for predicting non-fatal outcomes in COPD patients. The Charlson method performed worse than the other two methods for predicting COPD exacerbations.
Definition of comorbidities	ICD9/10 codes are explicitly defined in publications for the different comorbidities in the Charlson and Elixhauser systems. The ADGs are a proprietary system that requires specialized software to define the presence or absence of the different ADGs.
Cost of classification system	The use of ADGs requires a user license which typically requires a fee (although this may be nominal when used for research purposes). The Charlson and Elixhauser coding algorithms are non-proprietary and can be used without payment.

The ability of each method to predict hospitalizations and COPD exacerbations was substantially poorer than the ability to predict mortality. This is not surprising as these outcomes are likely more sensitive than all-cause mortality to acute disease factors (e.g. viral infections, seasonal variations) and non-disease factors (e.g. social situation, availability of hospital beds, access to health care), which chronic comorbidity models are not designed to capture. Nevertheless, the greatest difference between the ADG model and other models was seen for COPD exacerbations, which are commonly identified and managed in the outpatient setting. This validates the expectation that the incorporation of ambulatory care data in the ADG model makes this model better for comorbidity adjustment when evaluating ambulatory-sensitive outcomes.

Because the Charlson comorbidity index relies on diagnosis codes from hospital discharge abstracts, in order to reflect its initial derivation and validation, its use may not be optimal in populations in which not all of the patients have been hospitalized. Similarly, the Elixhauser comoribidities were developed for use with electronic hospital discharge abstracts, and were not designed to employ diagnoses from ambulatory records. In contrast, the ADG method permits comorbidity adjustment to be conducted in ambulatory as well as hospitalized populations. However, the assignment of ICD-9/10 diagnosis codes to ADGs is via a proprietary algorithm. Thus, this method may lack transparency and face validity compared to the other methods, for which the assignment of ICD-9/10 codes is explicitly described ^12;13. Furthermore, the use of the ADGs requires a user license which typically requires a fee (although this may be nominal when used for research or academic purposes), whereas the Charlson and Elixhauser coding algorithms are non-proprietary and can be used without payment. We have elected to use each comorbidity classification scheme in a manner that reflected its development: the Charlson and Elixhauser classification schemes were intended for use with in-patient hospital records, while the ADG method was intended for use with both in-patient and out-patient (ambulatory) patient records. Our intent was not to compare the impact of the choice of data source on the accuracy of outcome prediction. Rather, our objective was to compare the choice of comorbidity classification scheme on outcome prediction.

The Johns Hopkins ADGs and ACGs were developed for predicting health care resource utilization. However, there is a paucity of research into the ability of these comorbidity classification schemes to predict mortality and nothing that looks specifically at its performance in patients with COPD. A recent study using similar methodology as in the current study found that ADGs allowed for accurate prediction of mortality in a general population cohort from Ontario, Canada^5;6. Similarly, regression models incorporating age, sex, and the ADGs predicted mortality in patients with diabetes and in those with schizophrenia^3;4.

A few studies have examined the ability of the Johns Hopkins ACGs to predict mortality in different patient populations. Regression models using the ACGs (some, but not all, combined with age and sex) had c-statistics ranging from 0.701 to 0.768 for predicting mortality outcomes in a variety of settings ^8;15;24;25.The Charlson comorbidity index has been used previously for risk-adjustment in studies of COPD¹. However, our study, which examined patients from both hospital and ambulatory care setting, provides a more comprehensive evaluation of the Charlson index’s ability to predict mortality in this patient population.

We did not have access to detailed clinical data on our populations of patients with COPD. Thus, we were unable to adjust for important predictors of mortality in COPD patients such as the components of the BODE index ¹⁰. Also, we did not consider all comorbidity classification schemes that have been proposed in the health services literature. Schneeweiss et al. proposed that the number of unique drugs prescribed be used for risk-adjustment purposes ²⁷, while the Chronic Disease Score uses outpatient pharmacy records ³². A limitation of these two methods is their use of prescription records, as in many jurisdictions, data on drug prescribing are not available for the entire population. For example, in Ontario, data on prescription medication use are only available for patients over the age of 65 years.

Our results will be of interest to health services researchers working with administrative databases to study chronic diseases and to the health care providers and decision makers that rely on their findings. Further studies should confirm our findings in other chronic disease-specific populations.