Risk assessment in the genomic era: Are we missing the low-hanging fruit?

Accurate risk prediction is vital to medical decision making. Determination of the screening frequency, institution of primary prevention measures, acceleration of secondary treatment, consideration of advanced therapies versus palliative care, and assessment of cost effectiveness are all dependent on assumptions about risk. Increasingly, risk markers are being shown to be effective tools to identify who will most likely derive benefit from effective therapies, including statins for primary prevention¹ and early aggressive catheterization strategy for acute coronary syndromes.²

Unfortunately, the estimation of risk for adverse outcomes in the current health care system is often haphazard and qualitative. Furthermore, the few statistically rigorous risk models that have gained wide acceptance—such as the Framingham risk score (FRS)³ or the CHADS2 score⁴—tend to have suboptimal discrimination characteristics. In addition, all but the simplest models generally require a health care provider to input disparate data types into an online calculator or handheld electronic device, which is often prohibitive to routine clinical use. In contrast to this underuse of risk assessment in the routine care setting, a burgeoning flow of new prognostic biomarkers are being evaluated and promoted by the research community, and we are on the front of the genomics wave in which a myriad of new genetic and genomic markers will become available to better define disease and refine risk assessment.

Within this context, the current issue of American Heart Journal includes a provocative study from Anderson et al⁵ (p. 946) regarding the use of the basic metabolic panel (BMP—serum sodium, potassium, chloride, bicarbonate, blood urea nitrogen, creatinine, glucose, and calcium) as a highly accurate survival prediction tool. The study evaluated patients from Intermountain Healthcare system's 22 hospitals and affiliated clinics who had BMP assessed between 1999 and 2005. Vital status was known for over 270,000 patients at 30 days and for 40,000 patients at 5 years. Using a careful statistical approach that included split sample training and internal validation sets in the Intermountain data followed by external validation in the NHANES III, the performance measures for a BMP model were determined. It should not be surprising that well-known risk factors reflected in the BMP—hyponatremia, acidosis, renal dysfunction, and hyperglycemia—provide insight into a patient's risk for adverse outcomes. What is surprising is that this ordinary laboratory panel appears to possess such a strong ability to classify patients into high and low risk categories for short- and long-term mortality. In the Intermountain data, the c-statistics of the full model (BMP with age and sex) for 30-day, 1-year, and 5-year mortality were >0.84 at all time points. The BMP risk score, from −3 to 25, provided a fine gradation of predicted death, from 0.1% to over 50% for 1-year mortality.

These findings are remarkable—a simple laboratory panel routinely performed on millions of patients each year provides highly significant, quantifiable information about risk of death. What does this tell us? At a minimum, there is a great deal of prognostic information in these standard laboratories that we are not currently using in a quantitative way. The authors should be commended for this ironic finding, that although enormous resources are being poured into identification of novel genomic risk markers, we have not adequately assessed what we already have at our fingertips every day.

In assessing the usefulness of any new prognostic model or predictive biomarker, the key metric lies in the incremental improvement of risk assessment that the new model provides over existing data already incorporated into the routine clinical assessment of risk. Detailed statistical methods for assessing incremental prognostic value are complicated and controversial. At present, the change in c-statistic is a commonly used measure. What does an increase in c-statistic of 0.06 mean (from 0.788 to 0.85), which was seen with the addition of the BMP to age and sex alone to predict one-year mortality? This is highly statistically significant, but is this degree of improved discrimination clinically useful? In addition, the c-statistic only provides the discriminatory ability of the model, which is often the key characteristic for diagnostic testing but which tends to be less critical in risk modeling.⁶ More relevant to prognostic considerations in clinical care is the improvement in calibration of a model (its ability to more accurately quantify absolute risk). To address these limitations, novel statistical methods such as the integrated discrimination improvement and net reclassification improvement have more recently been promoted.⁷ In the case of the BMP model, a truer measure of its value would involve plotting the predicted probability of events across risk deciles for the base model (eg, FRS) and then for the base model with the novel covariate(s) added (eg, FRS + BMP). Such reclassification tables would give a sense of the magnitude of reclassification provided by BMP. Ultimately, multiple separate measures are necessary to characterize the overall performance of a new predictive model: discrimination, calibration, reclassification, and global fit.⁸

Perhaps more relevant to the usefulness of the proposed BMP model is its potential for practical clinical availability. A novel prognostic model may be superior to existing models without improved risk assessment if it is easier or cheaper to apply. Any health care provider who regularly reads laboratory reports could see the simplicity of adding an absolute risk of death at 30 days, 1 year, and 5 years for each reported BMP. If and how a health care provider or patient chooses to use that information within the context of a specific patient's overall health remain unclear, but at least, the BMP model could remove the work of collecting and inputting covariates into some separate risk modeling instrument. Furthermore, since the BMP is ubiquitous in its use, a BMP model would not typically require additional tests to be performed, and it would rapidly become familiar to providers.

The practical advantage, however, does not address another obvious question: how would having this information readily available improve patient care? How much value is there in having accurate risk assessment without being coupled to a strategy proven to decrease the risk? One concern is that in focusing on laboratory parameters, which are typically surrogates for certain disease processes rather than on the causal risk factors themselves, a BMP risk model may muddy the translation of risk determination into action. For example, we already have multiple proven and modifiable risk factors which form the foundation of our risk assessment. In determining that a patient is at high risk for adverse outcome based on the presence of high blood pressure, abnormal serum lipid levels, diabetes, and active tobacco abuse (as captured in the FRS), the course of action is logical: prescribe lifestyle changes and medications which specifically address these issues.⁹ In contrast, a printed risk of death at the bottom of a BMP report may not so readily emphasize the key interventions that must be implemented to complete the process.

Perhaps the most interesting aspect of the Anderson et al⁵ study is its focus on routine laboratory variables acquired most commonly in the evaluation and care of patients. In a similar analysis, these same authors found good prognostic accuracy for the complete blood count as well.¹⁰ Our group assessed the relationship of 36 routine blood tests and other clinical variables with morbidity and mortality in the CHARM program.¹¹ We found that red cell distribution width (routinely provided as part of a complete blood count), was among the most powerful overall predictors, a finding that was confirmed in the Duke Databank. Unlike the BMP and red cell distribution width examples, most interest in improved risk prediction has focused on finding novel biomarkers of risk. Troponins, natriuretic peptides, and high-sensitivity C-reactive protein provide examples of how new biomarkers can significantly contribute to improvements in prognostic estimates, diagnosis, and/or identification of patients who benefit from specific therapies. However, success is rare, as most promising novel biomarkers have not stood up to more complete statistical assessments.^12–14 While advances in genomics and proteomics promise to transform new opportunities in prognostic modeling, this study highlights the fact that the major challenge moving forward will not be the discovery of new markers but, rather, the optimal selection and validation of subgroups of clinically useful markers from the large pool of new and old candidates.

The Anderson et al⁵ study and others like it suggest that we have readily available clinical data that could easily improve out ability to prognosticate with the appropriate development of practical risk prediction tools. The key message is that routinely available but less “novel” laboratory data must be considered in the process of covariate selection and prognostic model construction. To go one step further, given the importance of efficiency in medical care, the default assumption should be to consider routine laboratory data of greater value than novel biomarkers unless the new techniques are shown by rigorous statistical testing to provide clinically significant incremental improvements in risk assessment or improve the coupling of risk prediction with therapeutic decision-making.

In conclusion, to the extent that accurate risk prediction can lead to improved patient care, there is opportunity to better leverage the wealth of data we currently use in routine practice. Large existing databases, novel statistical approaches, and evolving information technology provide an important opportunity to advance this goal. With improved practical predication models, the next challenge will be to show how this information can and should lead to better clinical decision-making and improved patient care.