A Simple Method for Evaluating Within Sample Prognostic Balance Achieved by Published Comorbidity Summary Measures

Abstract

Objective

To demonstrate how a researcher can investigate the appropriateness of a published comorbidity summary measure for use with a given sample.

Data Source

Surveillance, Epidemiology, and End Results linked to Medicare claims data.

Study Design

We examined Kaplan–Meier estimated survival curves for four diseases within strata of a comorbidity summary measure, the Charlson Comorbidity Index.

Data Collection

We identified individuals with early-stage kidney cancer diagnosed from 1995 to 2009. We recorded comorbidities present in the year before diagnosis.

Principal Findings

The use of many comorbidity summary measures is valid under appropriate conditions. One condition is that the relationships of the comorbidities with the outcome of interest in a researcher’s own population are comparable to the relationships in a published algorithm’s population. The original comorbidity weights from the Charlson Comorbidity Index seemed adequate for three of the diseases in our sample. We found evidence that the Charlson Comorbidity Index might underestimate the impact of one disease in our sample.

Conclusion

Examination of survival curves within strata defined by a comorbidity summary measure can be a useful tool for determining whether a published method appropriately accounts for comorbidities. A comorbidity score is only as good as those variables included.

When a patient has several diseases, it may be difficult for clinicians to determine how the diseases affect an individual’s health in unison. To assist in summarizing the effect of multiple conditions, researchers have developed summary measures such as the Charlson Comorbidity Index (Charlson et al. 1987), Elixhauser Score (Elixhauser et al. 1998; van Walraven et al. 2009), or ACE-27 score (Piccirillo et al. 2004) to condense information into easy-to-use metrics. These summary measures attach weights to each comorbid condition and then sum the weights of those conditions that are present in an individual. We refer to these summary measures as comorbidity scores.

Comorbidity scores can be useful for prognostic and health services research purposes, including comparative effectiveness studies. For example, having multiple diseases (i.e., comorbidities) can complicate research on treatment effects for specific diseases. Consider a case when researchers are interested in assessing the effect of surgery on overall survival for the treatment of kidney cancer. Comorbidities may have a direct effect on survival independent of any treatment effects, and they may have an indirect effect by affecting treatment choices. A patient with cardiovascular disease, for example, and regional (stage T3) kidney cancer may be at high risk of non-cancer-specific mortality based on her cardiovascular disease alone. However, she may also be at high risk of cancer-specific mortality if her cardiovascular disease precludes her from receiving effective treatment for her cancer (i.e., radical nephrectomy). In addition, competing comorbidities may also be exacerbated as a result of treatment for the disease of interest, also compromising survival (i.e., nephrectomy may result in worsened cardiovascular disease by worsening hypertension).

Conversely, cardiovascular disease may not affect treatment choices for other diseases in a similar fashion. For example, lumpectomy for breast cancer is associated with a much lower operative risk than nephrectomy. Therefore, it is less likely that a patient will not be able to undergo definitive surgery for breast cancer due to cardiovascular disease. In addition, it is less likely that treatment for breast cancer will have a serious impact on a patient’s cardiovascular disease.

In the example above, we used cardiovascular disease as a single-comorbidity example. When several comorbid conditions are present in unison, assessing their combined impact on treatment effects becomes more difficult. Summary measures such as comorbidity scores can be useful in reducing a high dimensional problem, such as the examination of multiple disease indicators, into a single variable problem.

Austin et al. (2013) recently provided a mathematical proof as to why comorbidity scores are effective. This expanded the work of others who wrote about prognostic scores (Hansen 2008; Stuart, Lee, and Leacy 2013). Austin et al.’s (2013) work examined the utility of using previously published comorbidity score algorithms, with a focus on those algorithms developed using survival models. Essentially, comorbidity scores provide a form of balance such that within levels of the comorbidity scores, additional information about the diseases used to create the comorbidity scores does not provide additional prognostic value. That is, average outcomes between those with and without an individual disease do not differ within levels of the comorbidity score. One caveat is that a comorbidity score is only as good as the variables that are used to create the comorbidity score.

Intuitively, comorbidity scores provide a form of balance because they are similar to propensity scores (Rosenbaum and Rubin 1983). In Figure1a, we provide a heuristic that demonstrates the similarity with respect to prognostic scores (Hansen 2008), which are more general forms of comorbidity scores. Prognostic scores can include variables other than comorbidities such as age and sex. As shown in Figure1a, there are two ways to block the confounding relationship between a binary exposure (or treatment) and an outcome (Pearl 2009). One can either block the pathway between the confounder and the exposure (or treatment) or block the pathway between the confounder and the outcome. Propensity scores can block the pathway between the confounders and the exposure, whereas prognostic scores can block the pathway between the confounder and the outcome. More technically, conditional on the propensity scores or the prognostic scores, the relationship between the exposure and the outcome is no longer confounded by the variables that were used to create the scores. In practice, propensity scores model the treatment, while prognostic scores model the outcome.

Figure 1.

(a) Confounder Relationship with a Binary Exposure or Treatment. (b) Prognostic Pathway Often of Interest in the Development of Comorbidity Scores

Comorbidity scores are a narrower version of prognostic scores in that they typically are developed without regard to an exposure or intervention of interest. Many also do not include nondisease components. Figure1b recasts Figure1a but places it in the more limited scope of comorbidity score development.

To frame this in more concrete terms, we consider the dilemma facing physicians and patients in the presence of a localized kidney cancer diagnosis. Early stage, that is, localized T1/T2, kidney cancers are often slow growing tumors that have a low risk of metastasis (Smaldone et al. 2012). At the same time, treatment for kidney cancer often consists of partial or complete removal of the kidney, which can result in substantial complications such as worsening renal function. The benefits of treatment for localized kidney cancer are hence unclear in an older population which has a high prevalence of competing comorbidities which may be exacerbated by nephrectomy (e.g., hypertension, diabetes mellitus) (Kunkle, Egleston, and Uzzo 2008; Sun et al. 2014). This suggests that active surveillance, whereby the patient undergoes routine imaging and only undergoes surgery when the tumor grows large enough, may be a viable option for many patients. To date, however, there are no large-scale randomized trials of active surveillance versus surgical treatment among those with localized kidney cancer. Thus, inferences about the benefit of active surveillance are generally derived from observational data. Kutikov et al. (2010, 2012) developed a prognostic model of postsurgery outcomes for those with early-stage kidney cancer. The goal of the paper was not to provide evidence of the effectiveness of surgery but to give patients information as to their likelihood of death following treatment. If patients are predicted to have short life expectancy even with definitive treatment for their kidney cancer, they may choose to forego treatment as treatment might only have a modest improvement in life expectancy at best. A comorbidity score that summarizes a patient’s comorbidities may help a physician and patient synthesize the burden of several coexistent conditions.

In this paper, we demonstrate how to evaluate whether a comorbidity score is appropriate for one’s own prognostic research using the Charlson Comorbidity Index (a.k.a. Charlson Score, Charlson et al. 1987). We chose to use the Charlson Comorbidity Index as an example because it is one of the oldest and most widely cited comorbidity indices. According to Web of Science, the Charlson Comorbidity Index has been cited over 10,000 times. The Charlson score has also been adapted for use with claims data (Deyo, Cherkin, and Ciol 1992; Romano, Roos, and Jollis 1993). There are alternative comorbidity indices, and for any given study, an individual researcher should choose the one that is most relevant to the researcher’s goals.

The Charlson score was created by estimating a Cox proportional hazards regression in a patient population in New York. The hazard ratios from the models, derived by exponentiating the coefficients, were used to assign points to conditions. The points were rounded such that hazard ratios ≥1.2 but <1.5 were given a weight of 1, ≥1.5 but <2.5 were given a weight of 2, and so on. Table1 presents the points assigned to conditions included in the Charlson Comorbidity Index.

Table 1.

Points Given for Conditions by Charlson Comorbidity Index

Condition	Points
Myocardial infarction	1
Congestive heart failure	1
Peripheral vascular disease	1
Cerebrovascular disease	1
Dementia	1
Chronic pulmonary disease	1
Connective tissue disease	1
Ulcer disease	1
Mild liver disease	1
Diabetes	1
Hemiplegia	2
Moderate or severe renal disease	2
Diabetes with end organ damage	2
Any tumor	2
Leukemia	2
Lymphoma	2
Moderate or severe liver disease	3
Metastatic solid tumor	6
AIDS	6

Before proceeding to the data example with the Charlson Comorbidity Index, we first provide a proof that demonstrates the analytic appropriateness of comorbidity scores in general.

The Utility of Comorbidity Scores

Here, we reproduce the proof presented by Austin et al. (2013) using slightly different notation, and placing greater emphasis on the proof’s relevance to comorbidity scores derived from hazard ratios. The proof shows that a comorbidity score based on a hazard or hazard ratio is an appropriate balancing score when used in survival analyses. We focus on hazards and hazard ratios as the Charlson Comorbidity Score was estimated using summed hazard ratios from a Cox regression. While the proof is more appropriate to summary measures that use regression coefficients, we can show that the limited numeric range of the Charlson score makes this proof similarly applicable to it. Let T represent the survival time. Let h(∙) represent the hazard, while X is a vector of covariates, X = {X₁, … ,X_n}’.

For ease of notation let b(X) represent the function for a comorbidity score derived from a hazard rate for fixed t: b(X) = h(t|X). We describe extending this to comorbidity scores derived from hazard ratios shortly. To prove that survival time is independent of the covariates given the comorbidity score (i.e., the balancing property), it is necessary to show that h(t|X,b(X)) = h(t|b(X)).

As b(X) is a function of X, it follows that h(t|X,b(X)) = h(t|X). Thus, for the proof it is sufficient to show that h(t|X) = h(t|b(X)). Let E_X[•] denote the expectation over X. Now,

Thus, we have h(t|X,b(X)) = h(t|X) = h(t|b(X)). This proof is extendable to hazard ratios with respect to the baseline hazard, h₀(t), as h₀(t) is a constant for fixed t. Therefore, h(t|X,b(X))/h₀(t) = h(t|X)/h₀(t) = h(t|b(X))/h₀(t) for fixed t. In the case of the Cox regression model, fixing t is appropriate as the hazard ratio is generally assumed to be independent of t; the proportional hazards assumption of the model results in h(t|X)/h₀(t) = exp(β₁X₁+ … +β_nX_n). In the case of the Charlson score, functions of the β_j terms are used to create the score. While the proportional hazards assumption of the Cox model can be relaxed through the use of time varying covariates, time varying covariates were not used in the development of the Charlson Comorbidity Index.

This demonstrates the “balancing” property of comorbidity scores. Once a score is known, then additional information about the comorbidities used to create the score is not informative for prognostic purposes. Of note is that the score is only as useful as the variables that were used to create the score.

While the proof is an important theoretical justification for comorbidity scores, it is a population-level result. In practice, one does not have the actual comorbidity score function (i.e., the comorbidity weights attached to each condition), but only an estimate of the comorbidity score function that has been published in the literature. The Charlson score weights were derived from a Cox regression, while the Elixhauser weights were derived from a logistic regression. The use of comorbidity score using previously estimated weights differs from propensity scores in which researchers estimate the coefficient weights using their own data. As shown in Austin et al. (2013), the use of previously published weights is appropriate when the population from which one’s own sample is drawn is similar enough to the population used for the published weights. A further benefit of using comorbidity scores from published estimates is that they may make researchers’ results more generalizable across studies. Also, previously published weights can be used in settings in which sufficient data are not available to estimate the weights.

A researcher may wonder if his or her sample is similar enough to the sample used to develop a published comorbidity score function for the comorbidity score to be valid. In the case of propensity scores used in the investigation of treatment effects, it is possible to demonstrate balance directly by comparing average characteristics in samples stratified by the propensity score (Wong et al. 2006) or weighted by propensity score-based weights (Lunceford and Davidian 2004; MacKenzie et al. 2006). In the next sections, we present a data example to demonstrate how one may similarly show whether a comorbidity score achieves prognostic balance in a given sample.

Data Example

For the data example, we use data from the Surveillance, Epidemiology, and End Results (SEER) database linked to Medicare claims data (http://appliedresearch.cancer.gov/seermedicare/ as accessed on September 9, 2014). The SEER database is a population-based cancer registry that collects information on cancers occurring within a geographic region that currently includes over one quarter of the U.S. population. Medicare data provide billing information on the procedures that Medicare beneficiaries received. The diagnostic and procedure codes used for billing can provide information on the comorbidities and treatments that Medicare beneficiaries received.

We examined patients who were diagnosed with early-stage (T1/T2) kidney cancer between 1995 and 2009. We further restricted the sample to those who were surgically treated so that our inferences would not be confounded by potential treatment effects. As Medicare only provides near universal coverage to those over 65, we only included all Medicare beneficiaries over 66 years of age who were enrolled in Medicare Part A and Part B (fee for service) for 1 year before diagnosis. This restriction ensured that we had sufficient claims to capture their prediagnosis comorbidities. To be a comorbidity for this demonstration, the patients had to have one claim in the inpatient file (Medpar) or two claims for a condition at least 30 days apart in the 1-year period prior to kidney cancer diagnosis in the outpatient or physician claims files. The restriction to two claims ensured that we did not inappropriately consider “rule out” diagnoses to be comorbidities (as recommended and implemented at http://appliedresearch.cancer.gov/seermedicare/program/comorbidity.html, accessed September 9, 2014).

Table2 presents the characteristics of the sample including the non-Charlson comorbidities and the Charlson score. As everyone in our sample had cancer, we omitted the cancer diagnoses in our calculation of the score.

Table 2.

Characteristics of Sample

Characteristics	Mean (SD) or %
N	17,740
Age	74.4 (5.8)
Year diagnosed	2004 (3.7)
Tumor dimension in mm, 408 missing excluded	45.6 (32.6)
Female	42
Married	62
Race/ethnicity
Asian	2
Black (non-Hispanic)	9
White (non-Hispanic)	85
Hispanic	2
Other	2
Charlson comorbidities
Myocardial infarction	4
Congestive heart failure	8
Peripheral vascular disease	5
Cerebrovascular disease	6
Dementia	<1
Chronic pulmonary disease	13
Rheumatoid arthritis	2
Ulcer disease	1
Mild liver disease	<1
Diabetes	24
Paralysis	<1
Moderate or severe renal disease	7
Diabetes with end organ damage	5
Moderate or severe liver disease	<1
AIDS	<1
Charlson score
Charlson = 0	54
Charlson = 1	25
Charlson = 2	11
Charlson = 3	5
Charlson = 4	3
Charlson = 5	1
Charlson = 6–10	1

SD, standard deviation.

Demonstrating Prognostic “Balance”

In Figure2, we demonstrate how one can demonstrate in one’s own data if the mathematical promise of comorbidity scores matches reality. In Figure2, we demonstrate all-cause mortality outcomes for four of the most common Charlson comorbidities: diabetes, chronic obstructive pulmonary disease, moderate or severe renal disease, and congestive heart failure. For the four diseases, we present the Kaplan–Meier curves overall (i.e., without stratification), and after stratifying by Charlson scores of 1, 2, and >2 (for diabetes, pulmonary disease, and congestive heart failure which were assigned weights of one) or 2, 3, and >3 (for renal disease which was assigned a weight of two). In all four cases, there are highly clinically meaningful and statistically significant differences in mortality between those who do and do not have the conditions prior to stratification (p < .0001 by log rank tests).

Figure 2.

Balance Achieved among Four Charlson Comorbidities before (Overall) and after Stratification by the Charlson Score

After examining survival curves within the Charlson strata, we see that in the cases of diabetes, chronic obstructive pulmonary disease, and renal disease, there do not seem to be clinically meaningful differences in mortality among those with and without the disease. This is as the mathematical proof above would suggest; after stratifying by the comorbidity score, the diseases lose much of their prognostic ability. The majority of the stratified curves for diabetes, chronic obstructive pulmonary disease, and renal disease are no longer statistically significant (p > .05), and only one is still statistically significant at the p < .01 level.

For congestive heart failure, however, highly statistically significant and clinically meaningful differences in survival curves persist after stratification (p ≤ .0001 in all four figures for congestive heart failure). This suggests that of the four diseases, the Charlson Comorbidity Index is least effective in balancing outcomes among those with congestive heart failure. As shown in Table1, the Charlson Comorbidity Index originally gave congestive heart failure a weight of 1 for the calculation of the score. Using our data, however, we found that congestive heart failure had a hazard ratio of 2.4 (95 percent CI: 2.2–2.6) in a simple Cox regression. We hence examined the properties that a recoded Charlson Comorbidity Index would have if the weight given congestive heart failure had instead been 2 or 3.

In Figure3, we demonstrate the prognostic value of congestive heart failure when the original Charlson score equals 1 or 2 and after reweighting the Charlson score with the increased congestive heart failure weights (points assigned for congestive heart failure are 2 for Score A, and points assigned are 3 for Score B). We see that congestive heart failure has much less prognostic value when using the reweighted Charlson scores, particularly when the weight is set to 3. This suggests that in those older individuals with early-stage kidney cancer, the original Charlson weight for congestive heart failure might be too small.

Figure 3.

Examination of the Charlson Comorbidity Index, within Charlson Strata, Reproducing the Original Charlson Index (“Original Charlson” Column) in Which Congestive Heart Failure Receives a Score of 1, and after Giving Congestive Heart Failure a Weight of 2 (“Reweighted Score A” Column) or a Weight of 3 (“Reweighted Score B” Column)

We calculated Gönen and Heller’s (2005) concordance statistic for two Cox regressions: (1) a model in which we included the individual comorbidity indicators as separate variables and (2) a model in which we included the Charlson score without the comorbidity indicators. In both models, we also included age, year of diagnosis, tumor size, sex, marital status (married/not married, and race/ethnicity (white non-Hispanic, black non-Hispanic, Asian non-Hispanic, Hispanic, Other). We obtained a concordance statistic of 0.644 for the model that included all of the comorbidity terms, and a concordance statistic of 0.643 for the model that included the Charlson Comorbidity Index instead of the individual comorbidity terms.

Discussion

There are a variety of comorbidity summary measures available to researchers (Sharabiani, Aylin, and Bottle 2012). Austin et al. (2013) provided an analytic proof that formalized why comorbidity summary measures can be so useful. One question left unanswered by that work was how a researcher determines if a comorbidity measure is appropriate for his or her given data. Those who use propensity scores can demonstrate balance by examining average characteristics of variables after stratification. Similarly, our method is a simple but relatively novel method of determining whether a comorbidity score captures information about comorbidities in a researcher’s sample. That is, for those who may use a published comorbidity score algorithm, our method can help determine whether there may be residual prognostic benefit in knowing the values of the comorbidity variables used to develop the comorbidity score.

In this work, we showed how a researcher can similarly demonstrate whether a comorbidity score appropriately provides prognostic balance. Examining estimated survival curves between individual disease states within strata of the comorbidity score can provide a simple method of evaluating the adequacy of comorbidity summary measures in a given situation. We believe that the visual presentation of the data as shown in Figure2 provides the most robust evidence of the appropriateness of a given comorbidity score. P-values comparing the curves may be less useful as a lack of statistical significance does not necessarily mean equivalence. Small sample sizes can reduce the power to detect true effects. Visual overlap of survival lines may hence be more reassuring evidence that a comorbidity summary measure is applicable to one’s own research.

In the localized kidney cancer example shown here, we found that the Charlson Comorbidity Index generally provided adequate balance for three of the four most common diseases it measures. However, the Charlson Comorbidity Index did not provide adequate prognostic balance for congestive heart failure. This implies that there is a trade-off to using the claims-based Charlson score among older individuals with localized kidney cancer. While the use of the Charlson Comorbidity Index may make a study’s results more comparable with other studies that use the measure, the measure might inadequately capture the incremental impact of congestive heart failure on a patient’s overall comorbidity burden. A researcher would thus need to determine which of generalizability versus bias is of most importance in a given study, and base the potential use of the Charlson score on the resulting trade-off.

Commonly used statistics, such as concordance statistics or areas under the ROC curve, may not adequately capture whether comorbidity scores are accounting for the component comorbidities. Our concordance statistics were very similar when comparing a model that included the Charlson comorbidities as individual covariates (i.e., without the score) with a model that included the Charlson score alone (i.e., without the individual comorbidities). More detailed analyses, similar to those we present here, can help identify subgroups for whom the comorbidity score is not adequately capturing the disease burden. In our detailed analyses, we were able to show that the Charlson Comorbidity Index might not fully capture the effects of congestive heart failure on mortality.

There are several possible reasons why a particular comorbidity score may not provide adequate prognostic balance using contemporary data. The Charlson Comorbidity Index was developed in the 1980s using inpatient medical claims for a general medical ward. It is possible that advances in medical treatment may have changed the prognosis for certain diseases. For example, certain “metastatic solid tumors” (weight of 6 in the index) such as breast or prostate cancer have a very long life expectancy. Their life expectancy is much longer than many other cancers such as pancreatic cancer, or noncancer conditions such as advanced dementia. In addition, the Charlson Comorbidity Index does not capture intensity of illness, and a patient with Class IV congestive heart failure would receive the same weight as a patient with more mild disease. Researchers should be aware of these limitations when designing their studies and interpreting their results. The weights in the Charlson score have been reestimated using more contemporary data (Quan et al. 2011). Our methods can help investigators determine which are the most appropriate weights for their own research.

A limitation of this paper is that we only addressed how one might determine if a comorbidity score appropriately balances comorbidities. We did not address how using a score that does not completely balance comorbidities may bias estimates or affect standard errors. We also did not discuss the causal inference implications of the use of comorbidity scores in the evaluation of treatment effects. Future research can address such issues.

A limitation of comorbidity scores in general is that they are only as good as the variables used to create them. They do not account for unmeasured confounding by comorbidities not included in the scores. Comorbidity scores may have limited utility if the measures of comorbidities included in scores do not fully capture relevant information about comorbidities (e.g., incomplete measures of the severity of individual comorbidities). Finally, incorrect model specification, such as a violation of the proportionality assumption in a Cox regression used to estimate a score, can diminish the usefulness of a comorbidity score.

Despite the proliferation of comorbidity measures and the durability of their use, their statistical properties are still understudied. Here, we present a simple method for investigating the prognostic balance achieved in independent samples that use published comorbidity score algorithms. We hope that our work spurs more methodological development into the utility and limitations of such comorbidity scores.

Supporting Information

Additional supporting information may be found in the online version of this article: