Study Design Issues in Evaluating Immune Biomarkers

Abstract

Purpose of review

The dramatic increase in the number and type of immune biomarkers that can be measured, particularly those assessing immune activation, has led to numerous investigations in HIV-infected individuals to explore pathogenesis and to assess therapeutic interventions that aim to attenuate immune activation. An overview is provided on study designs and related statistical and operational issues relevant to these investigations.

Recent findings

Cohort studies and nested case-control studies within these cohorts have identified multiple biomarkers that are associated with increased risk of disease. Early-stage clinical trials of therapies to address these risks in HIV-infected individuals with viral suppression on antiretroviral therapy are a substantial focus of current HIV research.

Summary

Appropriate study design is essential in biomarker research.

INTRODUCTION

A biological marker (“biomarker”) is defined as a characteristic that is objectively measured as an indicator of normal biologic processes, or pharmacological responses to a therapeutic intervention [1,2]. Biomarkers are used for diagnosis and staging disease, and as indicators of disease prognosis. In the current era of antiretroviral treatment (ART), in which individuals can achieve long-term viral suppression, a key focus of HIV research is to better understand the biological processes that contribute to morbidity and mortality in these individuals. Clinical events of interest are no longer AIDS-defining illnesses but rather non-AIDS events such as cardiovascular, renal, and liver disease and non-AIDS malignancies [3-5]. Correspondingly, HIV researchers now evaluate biomarkers and adjunct therapies in subjects on suppressive ART [6*,7**]. The biology of treated HIV disease is fundamentally different from the biology of untreated disease [8], which is important to consider when interpreting and designing studies of biomarkers in HIV disease. The purpose of this review is to highlight statistical and study design issues for biomarker investigations in HIV-infected populations.

Biomarkers include soluble biomarkers, cellular markers, genetic markers and diagnostic markers (e.g., serology for hepatitis infection). Soluble markers include inflammatory and immune activation biomarkers [2,6*,7**], haemostatic markers, lipid markers and others. They are popular because of low cost and ease of batch testing.

Factors to be considered when designing a study to evaluate immune biomarkers include the study population, sample size and type and frequency of measurements. Donna Mildvan and colleagues summarized the typical time-course of biomarker development, from identification in observational studies through validation as a surrogate marker of therapeutic efficacy [9]. Knowledge about prior research on a biomarker can help identify the appropriate type of study. For example, the first investigation of a new biomarker is likely to be observational [9], often in a case-control design. If associated with clinical outcome in observational studies and the biomarker is amenable to intervention, a clinical trial aiming to modify the biomarker could be developed.

Since the study design will depend on the nature of the question, we have organized our review into three sections corresponding to (1) initial investigations in observational studies; (2) early-stage, Phase I-II clinical trials; and (3) efficacy trials and surrogate markers. Because much of the biomarker research is based on observational studies, this is a key focus of our review.

OBSERVATIONAL STUDIES

Initial investigations of biomarkers seek to understand pathogenic mechanisms and whether biomarkers are prognostic of clinical outcome. Relationships of biomarkers to clinical outcomes are typically identified in large, longitudinal observational studies such as the MACS [10-12], WIHS [13] and ALLRT [14]. CD4+ T-cell counts and plasma HIV RNA levels are good examples of prognostic biomarkers in untreated HIV populations, as independent predictors of mortality [10]. The former indicates disease severity and the latter, viral burden.

A key goal now is to understand and evaluate biomarkers in the context of long-term viral suppression. Suppression is often defined as HIV RNA below 400 copies/mL [6*], or 200 copies/mL [15], because of potentially different lower limits of the HIV RNA assays used and the phenomenon of low-level viral blips [16]. Sub-analyses can then be performed using a more stringent definition, as the optimal goal of treatment is a viral load below the lower limit of the assay (e.g., <40 or <50 copies/mL) [15]. Interpretation of biomarker studies is altered when analyses include a mixture of subjects with controlled HIV replication on ART, subjects on ART with uncontrolled viral replication and ART-naive individuals; analyses restricted to those with viral control on ART are increasingly of most interest [17*] as the benefits of long-term ART and viral suppression have been well-established [18].

The clinical endpoints evaluated in the ART era differ from the AIDS-defining events that had previously been the focus. In addition to the differing pathobiology of these non-infectious morbidities [5,19,20**], there are study design features that are important to consider. First, databases for HIV cohorts, especially for retrospective analyses, might not have systematically collected the desired information regarding non-infectious morbidities, or have sufficient follow-up to collect such information. Therefore, endpoint review and adjudication may be needed and incorporated into the study design and timeline [20**,21,22]. The endpoint validation process could require review or abstraction from source documentation and medical records. A second issue concerns sample size for biomarker studies in virally suppressed populations. Since event rates likely will be lower in these individuals [18], large HIV cohorts are necessary. In addition, endpoint review and validation may not have been completed at the design and proposal stage for the biomarker investigation, so the number of clinical endpoints would not be known. Estimating statistical power and determining the number of subjects to evaluate without knowing the precise number of events is challenging when using study designs such as the case-control design reviewed below.

Much of the existing biomarker literature was obtained prior to the development of suppressive ART [9] or in populations with incomplete or mixed levels of viral suppression [2]. Whether these biomarkers are prognostic for disease progression in subjects virally suppressed on ART is not yet fully determined. Initial studies have shown that high levels of soluble markers of inflammation and coagulation are associated with morbidity and mortality in subjects on ART [6*,23,24*,25]. These important findings motivate and inform clinical trials of therapies to lower immune activation in populations who are virally suppressed [7**].

Another contribution of observational studies is the identification of factors that influence biomarker levels. This knowledge is valuable to optimally design clinical trials with biomarkers as endpoints, to reduce heterogeneity of study populations and to reduce sources of variability. For example, the presence of hepatitis C co-infection, which increases immune activation [26,27*], can be an exclusion criterion for a study investigating a therapy to reduce immune activation, although reducing the heterogeneity of the study populations also may limit the generalizability of the findings. Observational studies provide variance estimates for biomarkers and effect sizes with respect to clinical outcomes [23], which are necessary for sample size and statistical power calculations for future studies. Biologic and assay variability identified in observational studies can influence the design of subsequent studies; biomarkers with lower variability may be more reliable choices as primary endpoints.

Observational studies of biomarkers are often done retrospectively on stored specimens [23,24*]. However, the details of sample collection including frequency, type, volume and processing technique need to be determined prior to launching the study to ensure adequate samples are available. Therefore, consideration of future biomarker investigations is an important design component of these observational studies. Standardized sample collection procedures, personnel training and sample acquisition monitoring are critical, especially in a multi-center study. If vaccine responses are of interest [28], vaccine administration needs to be incorporated into an observational study prospectively. Functional measures, like flow-mediated dilation as a marker of cardiovascular risk, need planning in advance and integration into a study. Certain biomarkers, such as innate immune responses, that must be obtained on fresh samples [29] also need to be incorporated prospectively.

Resources and planning are critical to obtain productive information from an observational study. Cohort studies typically need large sample sizes and substantial follow-up to capture clinical endpoints. Visits at least twice a year may be needed to minimize loss to follow-up and ensure clinical events are captured. Missing data, loss to follow-up and failure to capture clinical events can lead to bias in resulting analyses. Differential loss-to-follow-up, for example if the sickest individuals are lost to follow-up, can create substantial bias in results [30,31]; every effort should be made to assure complete information on the study population. Biomarker studies in rare populations, such as those with acute HIV infection [32**], provide valuable information but require substantial resources for subject identification. Observational studies of immune biomarkers in non-blood tissues and body compartments also involve additional efforts for both the study volunteer and site [33].

Since numerous biomarkers are often measured [6*,23,24*,25], the issue of multiple comparisons arises, namely that there is increased likelihood of a false positive finding [34]. Pre-specifying the hypotheses and biomarker(s) of primary interest can address this issue. The likelihood of false-positive findings highlights the importance of confirmatory studies. In contrast, measurement error may result in false-negative results. Both biological and assay variability contribute to biomarker measurements, and the wide range of variability among markers can influence the interpretation of observational study results. In particular, this variability can attenuate the magnitude of associations such as odds ratios, hazard ratios and correlations [35]. Therefore, substantial measurement error in a biomarker assay will negatively impact the statistical power of a study to find an association between this biomarker and other endpoints. Repeat sampling, such as measuring a biomarker from samples drawn on two different days and averaging their values, in particular at baseline [36], or replicate testing of a sample and averaging the measures [17*,24*] can be used to reduce the effect of measurement error on analyses. Assay variability can also be minimized through batch-testing at a single laboratory using the same lot of reagents [37].

We now briefly review four types of population-based studies.

Cohort studies

Conceptually simple, but often complicated and expensive to implement, cohort studies enroll and follow a group of individuals to identify incident disease [20**,38]. For biomarker investigations, prospective cohort studies are optimal because stored samples can then be available prior to disease occurrence. Prospectively defining the study population is another advantage of cohort study design. The major difficulty with a cohort study, especially in the setting of suppressive HIV treatment, is that the study population may need to be large and followed a long time to identify events of interest. It can also be useful to evaluate biomarkers in both HIV-positive and HIV-negative cohorts [25,39]. Numerous biomarkers being studied in HIV-positive populations have not yet been evaluated in other populations, and vice versa.

Case-control studies

The case-control study [14,23,24*,38], often nested within a cohort study, is commonly used because it is an efficient approach; the number of subjects that need to be evaluated can be a small percentage of the cohort. Appropriate selection of cases (individuals with the outcome of interest) and controls (individuals without the outcome) is necessary to avoid bias [40]. The number of cases may be limited. To increase statistical power, more than one control per case may be selected for study with minimal gain in power beyond 4 controls per case [40]. Case-control studies have shown that inflammatory markers measured after one year of ART were associated with increased risk of diabetes [14] and that higher IL-6 levels at study entry were associated with all-cause mortality in both arms of the SMART study [23].

Case-cohort studies

A variation on the case-control design is the case-cohort design [41-43]. Within a cohort, a random sample of subjects is selected (the “subcohort”) in which the biomarker is measured, some of whom have the disease of interest. The biomarker is also tested on any subjects not in the subcohort who have the incident disease. The main advantage of the case-cohort design is that the same subcohort can potentially be used to investigate multiple disease outcomes (i.e., different case definitions). A disadvantage is that the statistical analysis of case-cohort designs is more complicated than a case-control study [42]. Unfortunately, batch-testing, which can limit the statistical impact of assay variability, might not be feasible when using the same subcohort to evaluate multiple outcomes [43]. For example, an initial case-cohort study investigating incident diabetes in a cohort might be done. Several years later, the same subcohort might be considered for an investigation of incident cancers. The biomarkers would then be measured on samples from subjects with incident cancers several years after the biomarkers for the subcohort were measured, resulting in data generated in two separate batches, perhaps with different lots of reagents, different equipment and different personnel. This could create bias, especially if levels of biomarkers are also influenced by the duration of storage [43]. These issues should be considered when choosing a study design.

Cross-sectional correlative analyses

Evaluating correlations between biomarkers and between investigative biomarkers and established measurements in HIV disease such as HIV RNA level and CD4+ T cell count is another informative approach to understanding disease pathogenesis [13,17*]. For example, weak correlations between IL-6 and HIV RNA levels in plasma in a recent report motivated more in-depth investigations into underlying mechanisms driving inflammatory processes in HIV-infected individuals [44**]. Adjusted correlations may also be useful to understand the interrelationships between biomarkers [45], although inferring causal effects is limited in cross-sectional designs [46].

EARLY-STAGE, PHASE I/II CLINICAL TRIALS

Currently, a major HIV research focus is to evaluate therapies that could reduce the level of immune activation [7**,47], which has been associated with increased risk of disease in subjects both pre-ART [2,9,11,12] and on ART [2,6*,23,24*,25]. These Phase I/II studies, typically enrolling a small number of HIV-infected individuals, are laboratory-intensive investigations with two main goals: to understand the underlying mechanisms of immune activation in HIV disease and to identify agents that are safe and of clinical benefit. Multiple biomarkers are invariably measured. In the words of Dr. Daniel Douek, studies will be “collecting as many samples from as many sites, measuring as many things as possible” [7**].

For these early stage studies, several different study designs can be used. The simplest is a single-arm study in which all subjects receive the intervention. This approach is useful for pilot investigations to gather information for designing a larger study, such as appropriate timing of biomarker evaluations. Pilot studies can inform estimates of the variability of biomarker measurements at a particular timepoint (or of changes from baseline) following an intervention to power a larger study as biomarker variability after an intervention may be greater than seen in observational studies without an intervention. Since they lack a control group, pilot studies may benefit from multiple pre-treatment biomarker measurements. These measurements can help establish that biomarkers are at “steady-state” prior to an intervention. If biomarkers are measured pre-intervention at comparable timeframes as the measurements post-intervention (e.g., four weeks pre-intervention and four weeks post-treatment), longitudinal analyses such as those based on linear splines can be used to test if the slope over time in the biomarker levels changes post-intervention. A related study design element is to include a post-intervention phase in which, for example, subjects are followed for eight weeks off treatment after eight weeks on treatment to assess duration of treatment effects. One advantage of a single-arm study is that safety or pharmacokinetic information can be obtained from every participant.

Single-dose studies, and variations such as dose-ranging studies, are important and useful for initial investigations of a therapy. An example is the study of recombinant human IL-7, which examined four different one-time doses using a dose-escalation design [48]. At the completion of each dose group, safety data was reviewed before the next higher dose group could begin enrollment. A key consideration concerns the optimal timing of visits and biomarker assessment. Immunologic effects may be transient [48] and could be missed if early evaluations are not included in the study design, but the biologic relevance of these transient effects may not be clear.

Many recent early-stage studies of therapies to reduce immune activation use a randomized design with an untreated comparison group for a control arm. Without a control group, observed biomarker changes after an intervention are confounded with other treatment-independent changes that may be occurring. Recent examples of controlled studies are a placebo-controlled randomized study of eight weeks of valganciclovir in on-ART subjects most of whom were virally suppressed [49*], a placebo-controlled randomized study of eight weeks of atorvastatin in off-ART subjects [50*] which also included a subsequent washout and cross-over phase, and an open-label randomized study of twelve weeks of a COX-2 inhibitor among off-ART subjects [51*]. These studies all included biomarker measures in a post-treatment follow-up phase.

In designing a randomized clinical trial [52], the primary question to answer, the effect size to identify and the study population are among the most important issues to consider. The primary question should be the one of most interest to the investigators and one that can be adequately answered in the study [52]. Studies that are too small might fail to identify beneficial interventions. Focusing on a primary question allows the study team to consider potential sample sizes by evaluating statistical power to detect treatment differences. Identifying a treatment difference in a biomarker that would be clinically or scientifically meaningful can be challenging. Observational studies provide valuable information to relate biomarker differences to clinical effects. For example, in the SMART study, higher IL-6 levels at baseline were associated with mortality risk [23]. From these data, a certain difference in IL-6 levels can be related to a differential risk of mortality, which could then be incorporated into the sample size considerations of a randomized trial with IL-6 as the endpoint. To calculate sample size for a quantitative biomarker endpoint, such as change from baseline to week 8, an estimate of variability is typically needed [52,53]. This estimate might be available from observation studies. Pilot studies to obtain variance estimates, especially for newly developed biomarkers, may be required in order to perform appropriate power calculations. For a two-arm study investigating a therapy to reduce immune activation, unequal randomization such as two subjects in the intervention arm for each subject in the control arm (2:1 randomization) requires only a modest increase in the total sample size relative to a 1:1 randomization and has the advantage that more subjects are given the investigational therapy to assess safety. In addition, this design provides more subjects for within-arm secondary analyses of those receiving the therapy, such as correlating changes in biomarkers after therapy.

In a clinical trial, the study population is explicitly defined through inclusion and exclusion criteria. These criteria can exclude subjects for whom the intervention might be unsafe and subjects who may skew the data due to active co-infections such as HCV or current medications. To investigate therapies to reduce immune activation, the inclusion criteria can be used to enrich the study population for those with higher activation levels who may be more likely to benefit from the therapy. In addition, it may be easier to identify a treatment effect in subjects with high activation levels pre-treatment. Examples include enrollment of subjects with CD4+ T-cell counts below 350 cells/mm³ [49*] or screening assays requiring an immune activation biomarker above a certain threshold [49*]. Because of regression to the mean, if eligibility is based on a biomarker value being above a threshold, a separate pre-treatment value should be used as the baseline value for assessing change from baseline [52]. An important consideration is whether to evaluate subjects on or off ART. While many studies aim to reduce immune activation in those virally suppressed on ART [7**], immune activation is higher in off-ART subjects and the potential for pharmacological interactions of some agents with ARTs may lead to initial investigations in off-ART subjects. If promising results are seen, a subsequent study could be planned for virally suppressed on-ART individuals.

As mentioned above for observational studies, factors that contribute to variability in the measured biomarkers should also be addressed in interventional studies to minimize extra sources of variation. Approaches include standardized specimen collection and processing, batch-testing of biomarkers, fasting visits if LPS or lipids will be measured, measuring biomarkers repeatedly pre-treatment and averaging for the baseline value, and restricting vaccinations such as the flu vaccine to not occur within a week prior to any study visit. Use of certain concomitant medications such as corticosteroids which can influence immune biomarkers could be limited by study design, or addressed in the analyses; collecting data on the use of such agents is then essential.

SURROGATE MARKERS AND CLINICAL EFFICACY STUDIES

The final stage for biomarker development is validation as a surrogate marker for clinical benefit [54-59]. A surrogate marker is one that can substitute for a clinical endpoint in the evaluation of new therapies; a surrogate marker is expected to predict clinical benefit, or harm, or lack of benefit or harm [58]. Surrogate markers can accelerate the process of evaluating a new therapy [57], as they might be measured months after starting treatment [60,61] rather than the multiple years of follow-up typically needed to observe clinical endpoints.

In HIV research, the two surrogate biomarkers used most extensively are CD4+ T cell counts and HIV RNA levels [60-62], which have been used for evaluating new antiretroviral drugs. Appropriate statistical evaluation of a potential surrogate marker is necessary; details are beyond the scope of this article, but a number of useful references are provided in this section. One key step in surrogate biomarker validation is the meta-analysis of large studies with carefully collected clinical data and specimen repositories. Prentice [55] developed criteria to validate surrogate endpoints in phase III studies. These require that the surrogate must be a correlate of the true clinical outcome and fully capture the net effect of treatment on the clinical outcome. Being correlated with a clinical efficacy measure is not sufficient to establish a biomarker as a valid surrogate [59].

An instructive lesson in immune biomarkers in HIV infection concerns the use of change in CD4+ T-cell count as a surrogate for clinical benefit. While seen as a useful surrogate for the clinical benefit of antiretroviral drugs [61,62], the CD4+ T-cell increases that followed IL-2 administration did not correspond to clinical benefit [63]. This highlights that the assessment and validation of a surrogate marker depends on the context and mechanism of action of the treatment [59]. Evidence for the clinical benefit of adjunct therapies to ART is anticipated to require large clinical endpoint studies.

CONCLUSION

Biomarkers and the emerging field of biomarker science offer the potential to advance our understanding of HIV disease, improve efficiency and reduce the cost of drug development. Like the laboratory assays themselves, study designs to evaluate biomarkers in HIV should be carefully selected and tailored to the question at hand. From evaluating the biology and disease risks of immune activation in observational studies to assessing therapeutics in randomized clinical trials [64,65], the design of the clinical study is an essential component of the research enterprise.

SUMMARY POINTS.

• Current HIV research is increasingly focused on immune biomarkers during suppressive ART and their relationships to clinical outcomes.

• Immune biomarkers are evaluated and serve as endpoints in both observational studies and clinical trials that aim to attenuate immune activation.

• Statistical and operational issues such as sampling type and frequency, inclusion and exclusion criteria, and variability in the biomarker measurements need to be considered during study design of both observational research and clinical trials.