Assessment of Biomarkers for Risk Prediction with Nested Case Control Studies

Abstract

Accurate risk prediction plays a key role in disease prevention and disease management; emergence of new biomarkers may lead to an important question about how much improvement in prediction accuracy it would achieve by adding the new markers into the existing risk prediction tools. However, in large prospective cohort studies, the standard full-cohort design, requiring marker measurement on the entire cohort, may be infeasible due to cost and low rate of the clinical condition of interest. To overcome such difficulties, nested case-control (NCC) studies provide cost-effective alternatives but bring about challenges in statistical analyses due to complex datasets generated. To evaluate prognostic accuracy of a risk model, Cai and Zheng¹ proposed a class of nonparametric inverse probability weighting (IPW) estimators for accuracy measures in the time-dependent receiver operating characteristic curve analysis. To accommodate a three-phase NCC design in Nurses' Health Study, we extend the double IPW estimators of Cai and Zheng¹ to develop risk prediction models under time-dependent generalized linear models and evaluate the incremental values of new biomarkers and genetic markers. Our results suggest that aggregating the information from both the genetic markers and biomarkers substantially improves the accuracy for predicting 5-year and 10-year risks of rheumatoid arthritis.

Introduction

Achieving the full promise of personalized medicine requires tools for accurate risk assessment. When the risk of developing a clinical condition varies with the values of the risk markers, one could potentially use the marker information to develop optimal disease management strategies. Through collaborative efforts, several large prospective cohorts have been assembled over the past decade for studying long term disease risk. Examples include the Framingham Heart Study², the Atherosclerosis Risk in Communities (ARIC) study³ the Nurse's Health Study (NHS)⁴, the Health Professional Follow up Study⁵, and the Women's Health Initiative (WHI)⁶. These studies, with long term follow up information on a wide range of clinical conditions, have become a wealth of resources for studying disease risks. In addition to recording phenotype information over time, the bio-specimens of the study participants were often collected at baseline and stored for future molecular marker studies. As knowledge accumulates over time, new markers may emerge as candidates for disease risk prediction. Once such research questions are formed, the stored specimens can be assayed and the marker measurements can be linked to phenotypes of interest to address these questions.

Large cohort studies provide valuable means for assessing the prognostic potential of new novel markers. However, obtaining marker values for the entire cohort may be infeasible due to cost associated with marker measurement, especial if the clinical condition of interest is rare. As cost-effect alternatives to the standard full-cohort design, a sub-cohort sampling design, nested case-control (NCC) studies are often employed in practice. For example, the NCC design was employed to study biomarkers for the risk of rheumatoid arthritis (RA) in the NHS.⁷ Under a standard two-phase NCC design, all subjects in the original phase-I cohort are followed to observe clinical outcomes, however expensive markers are only measured for the phase-II sub-cohort consisting of cases and controls selected randomly from the risk set of the cases. This design, while cost-effective, generates complex datasets for analysis due to outcome dependent missingness and differential follow-up time.

Statistical methods for risk prediction with NCC studies

To develop a risk model with data collected under a NCC design, the hazard ratio parameters in a proportional hazard (PH) model⁸ can be estimated by fitting a conditional logistic regression (CLR) model.⁹ To estimate the absolute risk, Langholz and Borgan¹⁰ used the CLR estimator of the hazard ratios and derived a simple weighted estimator focusing on within stratum comparisons. However, the CLR estimators could be quite inefficient. Furthermore, the CLR based approaches could not be easily extended to make inference beyond the PH model. Fully efficient nonparametric maximum likelihood (NPMLE) estimators have been obtained by Scheike and Juul¹¹ for the Cox model and by Zeng et al.¹² for a class of linear transformation models (LTMs). One major limitation of the NPMLE is that it requires the estimation of the conditional density for the new markers given other clinical variables, which may be infeasible in multi-dimensional settings unless additional modeling assumptions are imposed. One useful alternative to these methods is the inverse probability weighting (IPW) approach proposed by Samuelsen¹³. Besides being more efficient than the CLR, the IPW estimators are less model dependent and could be easily extended to other modeling paradigm. For example, Lu and Liu¹⁴ proposed IPW estimators under the LTMs. The IPW approach also allows for easy estimation of our targeted accuracy summaries.

To evaluate the predictiveness of a risk model, a popular approach is to use the time-dependent receiver operating characteristic (ROC) curve analysis. Statistical methods for estimating the accuracy measures in the ROC paradigm have been proposed using time-to-event data from full cohorts.^15-17 To compare different risk models and quantify the Incremental value (IncV) of new markers over routine clinical risk factors, several approaches have been advocated, for example, by comparing the area under the ROC curve (AUC).^17,18 However, those methods are only applicable to standard full cohort studies. To accommodate NCC sampling, Cai and Zheng¹⁹ developed consistent IPW estimators for time-dependent predication accuracy measures. However, their estimators are based on the PH assumption and therefore may be biased under model misspecification. Cai and Zheng¹ proposed a class of nonparametric IPW estimators of accuracy measures. However, these estimators only accommodate standard two-phase NCC studies and consider only the evaluation of a single marker.

Results for analyzing the NHS data

In NHS, two sets of biomarkers are measured sequentially at two different phases from the original cohort study. Accommodating such a three-phase NCC study, we extend the double IPW estimators of Cai and Zheng¹ to develop absolute risk prediction models under time-dependent generalized linear models and evaluate the incremental values of new biomarkers. Using data from the NHS, we developed four absolute risk models for RA: (i) clinical model with clinical markers alone; (ii) biomarker model with both clinical markers and two biomarkers; (iii) genetic model with clinical markers and a genetic risk score (GRS); (iv) full model with clinical markers, biomarker and GRS, for predicting a short-term, 5-year risk, and a long-term, 15-year risk. Our results suggest that the RA risk increases with pack-years of smoking, and alcohol consumption appears to be inversely associated with RA risk. In addition, higher levels of sTNFRII and IL6 are associated with higher risk of RA; IL6 has stronger effect on the 5-year RA risk than the 15-year risk, whereas sTNFRII appears to have stronger effect for the 15-year risk. The GRS appears to be highly predictive of RA risk. The clinical models appear to have low discriminatory capacity with area under the ROC curve (AUC) around 0.53 ∼ 0.58. Adding the biomarkers appear to be more helpful for predicting the short-term risk compared to the long-term risk. Aggregating information from both the genetic markers and biomarkers increases accuracy substantially when compared with the clinical models with AUC improving to 0.70 for 5-year risk and 0.62 for 10-year risk.

Discussion

It is challenging to assess the prognostic capability of new biomarkers based on sequentially selected subsamples of the study population. The extension of the double IPW estimators provides robust approaches to evaluate the IncV of new biomarkers on top of conventional risk factors in absolute risk prediction allowing for multi-phase NCC sampling designs. In addition, this method is straightforward to extend to other accuracy measures, including the positive and negative values^15,20 as well as integrated discrimination improvement index and the net reclassification improvement index.²¹