Leveraging Biospecimen Resources for Discovery or Validation of Markers for Early Cancer Detection

Abstract

Validation of early detection cancer biomarkers has proven to be disappointing when initial promising claims have often not been reproducible in diagnostic samples or did not extend to prediagnostic samples. The previously reported lack of rigorous internal validity (systematic differences between compared groups) and external validity (lack of generalizability beyond compared groups) may be effectively addressed by utilizing blood specimens and data collected within well-conducted cohort studies. Cohort studies with prediagnostic specimens (eg, blood specimens collected prior to development of clinical symptoms) and clinical data have recently been used to assess the validity of some early detection biomarkers. With this background, the Division of Cancer Control and Population Sciences (DCCPS) and the Division of Cancer Prevention (DCP) of the National Cancer Institute (NCI) held a joint workshop in August 2013. The goal was to advance early detection cancer research by considering how the infrastructure of cohort studies that already exist or are being developed might be leveraged to include appropriate blood specimens, including prediagnostic specimens, ideally collected at periodic intervals, along with clinical data about symptom status and cancer diagnosis. Three overarching recommendations emerged from the discussions: 1) facilitate sharing of existing specimens and data, 2) encourage collaboration among scientists developing biomarkers and those conducting observational cohort studies or managing healthcare systems with cohorts followed over time, and 3) conduct pilot projects that identify and address key logistic and feasibility issues regarding how appropriate specimens and clinical data might be collected at reasonable effort and cost within existing or future cohorts.

For decades, attempts to identify biomarkers for early detection of cancer have been characterized by cycles of promise followed by disappointment when initial claims of strong discrimination between cancer cases and controls were not reproducible and did not validate in prediagnostic specimens. The issue of reproducibility and validity in early detection biomarker research has received intense scrutiny (1–9). In recent years, it has become increasingly appreciated how features of cohort studies may help to protect internal and external validity in cancer detection biomarker research (10).

Internal validity refers to the strength or fairness of the comparison of groups within the study. Validity is compromised when bias occurs in the comparison of data and specimens of persons with cancer and those without, producing misleading positive results that cannot be replicated in subsequent analyses. When researchers use samples of convenience (readily available to the researcher), systematic differences may lead to misleading positive results. Examples include: 1) a cancer group comprised of older men and a control group of younger women, or 2) specimens collected, prepared, stored, or analyzed in different ways in the compared groups (4). These spurious results from studies that lack internal validity are usually not reproducible.

External validity refers to the generalizability of the comparison results to persons outside of the study. Thus, while there may be no bias in the comparison of groups with and without cancer, results of the investigation may not generalize to a different but clinically relevant group. For example, a test that is positive in persons with advanced cancer may not perform well in persons with localized cancer. Furthermore, claims for early detection have failed when candidate markers developed on clinical specimens (among persons with symptoms) were then evaluated in prediagnostic specimens (11). One biological explanation why candidate markers in early-stage symptomatic persons may differ from controls is that the candidates are acute-phase plasma proteins (12,13) that may be increased by inflammatory and other conditions present only near the time of symptomatic diagnosis. Such candidates would then fail to separate cancer from controls when assessed on prediagnostic samples. Clinical studies seldom have prediagnostic specimens from most patients because obtaining them requires following large cohorts of asymptomatic people, ideally at periodic intervals, to ascertain if they develop cancer.

Although we use the term “early detection” throughout this manuscript, we intend for it to mean “earlier detection,” meaning before clinical diagnosis under standard care. Earlier detection then has the potential to increase the proportion of cases detected in pathological early stage, while they are asymptomatic and perhaps more readily treatable.

Three Current Insights from the Field

Three insights discussed in recent years may help improve the reproducibility and validity of research about biomarkers for early detection of cancer.

Insight #1: Biospecimens are the products of a study whose design and conduct determines the reproducibility or validity of results.

Biospecimen collections should be considered as products of a research study, even if the process of collecting the biospecimens was never thought of as a formal study during the collection of specimens and data (4,14). In other words, a biomarker study does not begin with molecular or biochemical measurements and analysis of the specimens. Rather, it begins with the study design that produces the specimens. Biomarker researchers must assess whether an unbiased analysis can be made, regardless of the reason specimens were collected in the first place, by considering the design and conduct that were actually implemented to collect the specimens (4,14). The investigator may anticipate what details of design and conduct are important by considering what would need to be described in a Methods section of a manuscript according to guidelines such as the Standards for the Reporting of Diagnostic accuracy studies (STARD) statement for reporting a study of diagnosis (15,16).

There are two major reasons to think about biospecimens as products of a study. First, the lack of attention to internal validity or external validity may fatally compromise the research results (14). Second, more thorough study design description is increasingly required by scientific journals and funding agencies so that external and internal validity can be judged by readers, as part of efforts to improve reproducibility of research (8,9,17,18). The biomarker study that will ultimately be reported to a journal is a combination of the design and conduct of the original biospecimen collection along with whatever additional studies or analyses are undertaken.

Insight #2: Internal validity in early stage research (discovery) is as important as in later stage research (validation).

Some researchers may feel that early stage (discovery) research might not require the internal validity and rigorous design expected for late-stage validation studies. However, it is increasingly recognized that discovery research needs to be well designed, conducted, and interpreted because false leads will be costly if subsequent studies are based on biased initial findings. The consequence of biased discovery research is that subsequent validation will simply show that the original discovery was not reproducible. Recognizing the need for better discovery research will further increase demand for appropriate specimen collections from studies with strong internal and external validity. In response to this imperative for stronger discovery research, the NCI’s Early Detection Research Network (EDRN) has recently increased its focus and effort on generating such specimen sets (http://edrn.nci.nih.gov/).

Insight #3: Cohort studies of individuals followed over time may provide specimens and data that help improve both internal and external validity.

In cohort studies, healthy individuals may be followed over time to assess outcomes such as cancer diagnoses and/or death. The inherent design of a cohort study may incorporate important features that help ensure a fair comparison between persons who develop an outcome such as cancer and those who do not (internal validity) and that may involve collection of specimens at a prediagnostic stage (external validity). Infrastructure of such studies could be leveraged at marginal cost to collect data on cancer diagnoses and prediagnostic specimens periodically from entry into the study until the occurrence of the outcome(s) of interest. For example, if blood samples are collected routinely at times before a person becomes symptomatic and if the person is followed to an outcome, those specimens would necessarily have been collected and handled similarly among those who end up subsequently in the cancer group and those who do not, because disease status would not be known at the time of collection, storage, and processing of the specimens (19). For blood-based tests, for example, it may be particularly important that blood specimens be drawn prior to the diagnostic exam and biopsy that establish the cancer diagnosis, to avoid spurious signal that may be introduced into blood by the biopsy itself or by medications, diet, or behavior after the cancer diagnosis.

The use of a cohort study to collect unbiased and prediagnostic specimens and data on cancer diagnoses has recently been described as the “Pivotal Evaluation of the Accuracy of a Biomarker Used for Classification or Prediction” or “Prospective Specimen Collection Retrospective Blinded Evaluation” (PRoBE) design (20). The approach involves a case-control study nested within a cohort that includes collection of specimens. The biomarker is assayed in a blinded fashion on blood specimens collected prior to and near the time of diagnosis among the case patients and at similar times in the control patients (20). While the idea of a nested case-control study is not new, the PRoBE concept effectively describes the approach and its application to diagnostic marker research (21). Studies using such banked specimens and data collected prior to symptoms or diagnosis are increasingly recognized as precious resources for making comparisons that have strong internal and external validity.

While the concept of the PRoBE design sounds simple, the logistics involved may be difficult, so that only a few such biospecimen resources are currently available, such as the Prostate, Lung, Colorectal and Ovarian Cancer Screening Trial (PLCO), the Women’s Health Initiative (WHI), the Carotene and Retinol Efficacy Trial (CARET), and the Nurses’ Health Study (NHS) (11,22). In these studies, blood specimens were collected on more than one occasion after study entry. Specimens from such studies are increasingly in demand for validation studies of biomarkers for early detection. As a result, investigators have typically developed processes to govern use of specimens. Because cohort studies with prediagnostic samples are currently rare and samples are limited in number and volume and large prospective studies solely aimed at early detection are very expensive, it will be increasingly important to leverage the infrastructure of existing and planned cohort studies conducted for other reasons to collect prediagnostic periodic biospecimens and data on cancer diagnoses and on symptoms. This will increase availability of the type of biobanks necessary to evaluate biomarker candidates for early detection of cancer.

Meeting Overview

With this background, the Division of Cancer Control and Population Sciences (DCCPS) and the Division of Cancer Prevention (DCP) of the National Cancer Institute (NCI) held a joint workshop in August 2013.The meeting was intended to advance discussion about how to develop or cultivate existing cohort infrastructures and resources to obtain blood specimens and data that can be used to conduct studies of early detection cancer biomarkers that have strong internal and external validity (23).

Participants included investigators experienced in statistics and research design of studies of diagnostic markers, including leaders of large cohort studies and investigators in health maintenance organizations (HMOs). Below are the summarized considerations and key outcomes from the discussion. Further details about the meeting can be found here: http://epi.grants.cancer.gov/workshops/biomarkers/.

Two main areas discussed in detail included study design issues and logistical issues (Table 1).

Table 1.

The fundamental issues and considerations in studies of early detection biomarker discovery and validation: discussion from the meeting*

Issue	Consideration	Discussions
Study design
Fundamental comparison	Internal validity: If the fundamental comparison is biased—with a systematic difference between the compared groups—then results will not be reproducible or valid.	The process of collecting subjects, specimens, and data should be reported using accepted reporting guidelines such as STARD (15,16). Consideration should be given to whether any aspect of the comparison would be considered a serious or fatal bias. The PRoBE approach (prospective specimen and data collection and blinded retrospective evaluation) (20) helps provide features of design and conduct that avoid bias and ensure internal validity.
Subjects	External validity: Methods should assure that subjects can be determined to be asymptomatic when specimens are collected.	Periodic collection of serial samples at prespecified intervals within a cohort followed over time helps assure that symptoms are not the reason that a specimen was collected. Use of prediagnostic samples also allows earlier detection than clinical detection, helping to differentiate useful biomarkers from acute phase proteins.
Specimens/data	Specimens should be collected serially over time.	Frequent serial collection helps assure that a specimen will be available near and before the time of cancer diagnosis, and prior to biopsy or other change that occurs after diagnosis is established. Specimens collected prior to outcome will be handled blinded to outcome status. The PRoBE approach, which involves collecting (and storing) before the outcome occurs, helps avoid bias in handling, processing, and storage between case patients and control patients.
Outcome	All subjects should be followed with similar intensity to ascertain the outcome.	Unequal follow-up may lead to preferential ascertainment in persons with symptoms or with a positive test result. In a formal prospective research cohort, uniform methods to assess outcome are often utilized. In clinical cohorts, such as within HMO settings, special procedures may need to be created to ascertain outcomes similarly in all subjects.
Logistics
Sample collection	Special collection procedures may be needed for biological molecules to maintain integrity (eg, protein, mRNA); special handling procedures (collection, centrifugation, freezing, etc.) may be needed.	Appropriate standard operating procedures for sample collection and handling may be needed.
Sample storage	Sample storage may be costly for biospecimens like serum, plasma, etc.	Centralized storage and economy of scale may help to alleviate costs. Support should be considered for continued storage of specimens after a study has ended.
Specimen sharing	Appropriate specimens may have already been collected in NCI-sponsored trials and studies.	An inventory of NCI-funded studies that collected prediagnostic biospecimens may identify resources that already exist and can be used for earlier detection biomarker studies. Such an inventory would need to include not only details of specimens but also of the purpose, design, and conduct of the cohort studies that led to their collection. An ongoing dialogue among groups that have applied or developed appropriate methods to create unbiased sample and data collection is needed.
Process sharing	Material transfer agreements and data use agreements may be hard to facilitate.	Existing infrastructures created by the NIH and other groups (eg, NIH TAD system) to ease burden on researchers.

* HMO = health maintenance organization; NCI = National Cancer Institute; NIH = National Institutes of Health; NIH TAD = NIH’s Transfer Agreement Dashboard; PRoBE = Pivotal Evaluation of the Accuracy of a Biomarker Used for Classification or Prediction; STARD = Standards for the Reporting of Diagnostic accuracy studies.

Study Design Issues

Discussants identified several study design features that should be considered in research cohort and HMO settings. A formal cohort study aimed at addressing questions about etiology and risk factors or the natural history of disease likely will have been designed and conducted with explicit attention to subject selection, measurements at baseline and follow-up, and methods to assess outcome. On the other hand, a cohort may never have been part of a formal study but be comprised of a group of persons followed over time without any specific scientific aim or a priori question. Such cohorts (eg,, persons being followed in an HMO) would have no selection criteria or effort to regularly conduct measurements or assess outcome. Since HMO members may receive most of their care within a closed, integrated system, it might be possible to add such selection criteria and determination of outcome and symptom status at a later time or to add appropriate specimen collection or data collection in the future. Researchers will have to assess on a case-by-case basis if specimens and data collected from healthy visits in an HMO setting are appropriate for studies of early detection biomarkers.

The collection of serial blood samples in a cohort may provide multiple strengths in studies of biomarkers for early cancer detection. First, collection over time results in the increased likelihood of the availability of specimens prior to the time of diagnosis. Second, biomarkers in serial specimens from the same person may show change over time that, with suitable interpretation, increases prediagnostic detection of cancer while maintaining the same specificity. Several studies suggest that longitudinal measures of markers may be more predictive than single measures (24–30).

Logistical Issues

Meeting participants outlined a number of practical considerations involving specimen collection, storage, and sharing. For example, sample collection techniques must ensure that analytes (RNA, proteins, etc.) maintain their integrity (31). Appropriate standard operating procedures (SOPs) must be developed regarding collection, processing, and storage of specimens. Additionally, sample storage and sharing can be costly and labor intensive. Such issues are magnified by the large numbers of specimens and samples needed for early detection biomarker projects, because serial sample collection and low-incidence diseases are involved and the time window of interest for biospecimens may be short, immediately preceding disease diagnosis, for target cancers with short natural prediagnostic history. Under these circumstances, only a small proportion of biospecimens collected will ultimately be informative.

Additional logistical considerations included those related to privacy, confidentiality, and use of specimens for research. In the context of cohort studies or clinical trials in which study participants have been consented for research purposes, one concern is whether consents under which persons enrolled permit uses of clinical data or biospecimens not originally specified. Informed consent documents may have been drawn narrowly, potentially limiting ability to share materials or data beyond the scope of work outlined in these documents. For use of clinical specimens in which specific research consent was not obtained, deidentification of specimens and associated clinical data may be possible; however, the use of such specimens will be limited and would require institutional review board (IRB) approval before a study is launched. With advances in genomic and other technologies, it may be difficult to guarantee that there is no potential for reidentification of participants (32).

Three Overarching Recommendations

Three overarching recommendations emerged from the discussion (Table 2; Supplemental Table 1, available online) including: 1) facilitate sharing of existing prediagnostic specimens and associated data through creation of a comprehensive inventory, 2) continue dialogue and collaboration among scientists involved in early detection biomarker discovery or validation projects and those involved in cohort studies or conducting studies in healthcare delivery systems, and 3) establish pilot projects to address key questions regarding economic and logistic feasibility of adding, to existing and future cohorts, collection of periodic prediagnostic specimens and data about cancer diagnosis and, where possible, symptoms.

Table 2.

Overarching workshop recommendations for leveraging biospecimen resources in discovery or validation of markers for early cancer detection*

Overarching recommendation	Examples for implementation of recommendation
1. Facilitate sharing of existing specimens and data	Examples
Create NCI-wide inventory of prediagnostic specimens and cancer diagnosis data	Create an easily searchable, NCI-wide inventory of prediagnostic biospecimens and cancer diagnosis data in studies (leverage the Specimen Resource Locator). Inventory should include adequate study details.
Ease burden of establishing material transfer agreements and DUAs	Encourage utilization of systems developed by the NIH or other groups (eg, NIH TAD) that facilitate MTAs and DUAs for sharing of specimens.
2. Ongoing dialogue and collaboration	Examples
Learn how to cultivate existing and future resources	Clinical researchers and epidemiologists with experience in cohort research can share lessons learned in collecting specimens and data for biomarker studies. Basic scientists, epidemiologists, clinicians, statisticians, and bioinformaticians need to collaborate and consider how to achieve strong study designs that lead to robust scientific findings.
3. Pilot projects	Examples
Develop portfolio of appropriate pilot projects	Engage the broader research community to identify the most important questions to address in pilot projects.
Leverage existing and planned infrastructures	To leverage infrastructure of existing and planned cohorts that already assess cancer diagnosis outcomes, establish pilot projects to demonstrate how periodic prediagnostic specimen and data collection can be added at marginal cost. For existing and planned noncancer cohorts that serially collect prediagnostic specimens and data, consider the feasibility of adding the ascertainment of cancer diagnosis outcomes.
Use of combined specimen sets	Investigate the feasibility and appropriateness of combined samples from existing studies to minimize specimen depletion.
Explore new specimen technologies	Explore alternative ways for preserving specimens (eg, room- temperature DNA; FFPE blocks, dried blood spots on filter paper for biomarker studies) and technologies that can utilize smaller amounts of sample.

* DUAs = data use agreements; FFPE = formalin-fixed paraffin-embedded; MTAs = material transfer agreements; NCI = National Cancer Institute; NIH = National Institutes of Health; NIH TAD = NIH’s Transfer Agreement Dashboard.

Comprehensive Inventory

Meeting participants suggested that while many useful resources currently exist, researchers may not be familiar with them. A centralized cataloging of existing specimens and data may help researchers identify appropriate prediagnostic specimens and collaborate with scientists who manage them. The first step would be to catalogue the types and numbers of biospecimens currently available that might be suitable for the discovery or validation of early detection biomarkers. The creation of a searchable, comprehensive inventory of specimens and associated data funded by the NCI would be a step forward for the extramural early detection biomarker community. The NCI’s Specimen Resource Locator (SRL- https://specimens.cancer.gov/tissue/default.htm), which currently contains information mostly regarding tumor or other pathology specimens, could be expanded in the future to serve as a platform to develop such an inventory useful for early detection marker research. By creating such an inventory, it may be possible to determine gaps in specimen collections that could address critical hypotheses. Facilitating and standardizing material transfer agreements (MTAs) and data use agreements (DUAs) may help to ease process issues associated with the sharing of specimens. For example, the National Institutes of Health (NIH)’s Transfer Agreement Dashboard (TAD) system (https://techtransferagreements.nih.gov/Pages/About.aspx) is designed to facilitate the completion and tracking of MTAS for materials transferred in and out of the NIH.

While there was enthusiasm to develop such an inventory of available resources, other participants cautioned that, while some of NIH’s past efforts to develop catalogues of biospecimen resources have been successful, such as the National Heart, Lung, and Blood Institute BioLINCC (https://biolincc.nhlbi.nih.gov/home/) and the National Institute of Diabetes and Digestive and Kidney Diseases repository (https://www.niddkrepository.org/home/), other efforts have had limited usefulness for early detection research. Further, there was concern that, since so few cohort studies have collected serial prediagnostic blood samples suitable for the PRoBE analysis, few good resources may now exist for early detection biomarker studies. Workshop participants suggested that the main focus should not be on identifying existing banked specimens and data but rather on what can be done to leverage ongoing and new studies in the future.

Dialogue Between Basic Scientists and Epidemiologist/Clinical Researchers

Participants encouraged continuation of the dialogue among basic biologists or technology-based researchers and epidemiologists or clinical investigators involved in large cohort studies or in research in healthcare settings. This will help develop this field and its resources. Infrastructure grants from the NCI such as those supporting cohorts under the UM1 funding mechanism (see http://epi.grants.cancer.gov/funding/cohorts/) or collaborations such as the Cancer Research Network (http://crn.cancer.gov/) could potentially be leveraged to explore the creation of inventories noted above or the consideration of PRoBE applications in the new collection of periodic prediagnostic biospecimens. Some participants noted that this meeting was one of the largest efforts to support discussions among these groups.

Collaboration and shared expertise will advance research. Researchers in health care delivery systems have worked to develop relationships and trust with patients and providers in these systems. Collaboration between those researchers and researchers outside of the network not only ensures utilization of network expertise, but it may facilitate navigation of the clinical system for researchers who are unfamiliar with such systems.

Pilot Projects

One major recommendation from the meeting was to identify key questions on how to effectively cultivate cohort resources and to initiate pilot studies to address them. The overall goal would be to evaluate the feasibility of leveraging large existing infrastructures for future use in early detection cancer biomarker discovery or validation. Several examples were discussed during the meeting, including pilot studies to add appropriate collection of prediagnostic specimens and cancer diagnosis data to existing and ongoing infrastructures, both in research and HMO settings. Additionally, pilot studies that evaluate feasibility of combining specimen sets from suitable studies could help elucidate the variability in specimens. Pilot studies that explore new specimen preservation technologies might be pursued to help alleviate storage and collection costs for researchers.

Conclusions

The ongoing cycle of promise and disappointment in studies of early detection cancer biomarkers calls for a change in how researchers approach study design to better address challenges of both internal validity and external validity. The current use of blood specimens from cohorts within screening and cancer prevention clinical trials such as PLCO, WHI, and CARET, as well as in other prospective cohort studies or in clinical practice settings such as HMOs, suggests that leveraging existing studies and infrastructures may provide larger numbers of unbiased samples for the discovery or validation of early detection biomarkers at marginal cost. Workshop discussion identified three overarching recommendations with concrete action items, described above and in Table 2, to move the field forward. These include: 1) facilitate specimen and data sharing, 2) continue the dialogue between diverse groups of scientists, and 3) nurture pilot projects that provide examples of productive approaches and creation of serial prediagnostic biospecimen resources. By addressing the recommendations from the meeting, we hope that early detection cancer biomarker study results will be more reproducible and valid in prediagnostic samples. The recommendations described herein should be a priority not only for the NCI but also for the larger extramural research community, potentially including studies of other biomarker types.

Funding

This publication was supported with grants from the National Center for Advancing Translational Sciences (NCATS) (OCTRI-TR000128) and (UNC-1UL1TR001111), the National Cancer Institute’s (NCI’s) Cancer Research Network grant number (CA171524 and CA079689), and the NCI’s Early Detection Research Network grant (CA152990).

The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health nor the Department of Health and Human Services.

Christine Berg: Consultant, Medial Cancer Screening, Ltd. Steven Shak: Genomic Health Employee and Stockholder

Steven Skates: Consultant, Abcodia. Massachusetts General Hospital has licensed software to Abcodia.