Framework to Select Multi-Cancer Detection Assays in the National Cancer Institute’s Vanguard Study

Abstract

Background:

The Cancer Screening Research Network is a new clinical trials network funded by the NCI. The first Cancer Screening Research Network study, the Vanguard Study (VS), will assess the feasibility of using multi-cancer detection (MCD) tests in future randomized controlled trials.

Methods:

This article describes the framework NCI developed to engage MCD assay developers, evaluate emerging technologies using biobank reference sets, and select fit-for-purpose MCD assays for inclusion in the VS.

Results:

NCI evaluated 23 technologically diverse MCD assays, all utilizing machine learning and artificial intelligence components. Nine assays underwent blinded performance evaluations using specimens from three biobanks. Assay developers were provided with independent assessments of their assays, which enabled them to make further assay refinements to enhance performance. The assay selection process resulted in participation by two assay companies in the VS.

Conclusions:

NCI created a fair and transparent process to streamline the evaluation of assay performance and to select promising assays for clinical research and public health initiatives.

Impact:

Significant resources are required for large-scale cancer screening trials; therefore, promising technologies must be prioritized for inclusion in a definitive trial. NCI’s assay selection framework can be used and repurposed by other networks and institutions.

Introduction

Multi-cancer detection (MCD) assays offer the possibility of screening for multiple different cancer types with a single blood draw. These tests could be a major step toward improved early detection of malignancies, but the full spectrum of benefits and harms for screening asymptomatic individuals with MCD tests is not known (1–4).

In January 2024, the NCI launched a new clinical trials network, the Cancer Screening Research Network (CSRN), to systematically evaluate novel technologies for cancer screening. A randomized controlled trial (RCT) with the primary endpoint of cancer mortality remains the gold standard for assessing benefits and harms for any screening modality. However, designing a large RCT to study MCD assays that screen concurrently for multiple cancers introduces new and complex challenges compared with the design for a trial evaluating assays that screen for a single cancer type (5, 6).

The preliminary design of a large feasibility study, called the Vanguard Study (VS), was included within the Requests for Funding Announcements (7–9) for the CSRN. In parallel with the development of the CSRN, the NCI created a fair, transparent, and rigorous process to select MCD assays for inclusion in the VS. This article describes the assay evaluation and selection process and the rationale for each stage.

Materials and Methods

Figure 1 describes the overall assay selection process. In stage 1, NCI solicited stakeholder input on the necessary characteristics of MCD assays to qualify as fit-for-purpose candidates for the VS. The NCI conducted a technology landscape analysis, developed assay review criteria, and gauged interest of assay developers to collaborate with NCI.

Figure 1.

Assay selection process. In stage 1, NCI solicited stakeholder input, conducted an environmental scan, developed review criteria, and prioritized assays. In stage 2, developers performed their assays on blinded specimens, and NCI statisticians calculated performance measures. In stage 3, the NCI assay selection committee evaluated nine assays. Two developers with top-performing assays agreed to participate in the VS and established clinical trial agreements with NCI. CLIA, Clinical Laboratory Improvement Amendments.

The NCI ascertained that dozens of assays using diverse technology platforms were in various stages of development, relatively few developers had published performance results, and the assay evaluation methods varied widely (1, 10). To support a uniform, independent approach to evaluate how well each assay performed, NCI funded the Alliance for Clinical Trials in Oncology (Alliance) to develop a biospecimen reference set designed for MCD assay performance evaluation, denoted as the Alliance Reference Set Study (ClinicalTrials.gov identifier NCT05334069; ref. 11).

The NCI hosted a workshop for MCD developers and invited applications. Applications were reviewed for technical merit and prioritized for inclusion in stage 2, the assay evaluation process. Developers ran their assays using samples from NCI-funded, blinded biospecimen reference sets, including the Alliance Reference Set. Study statisticians then evaluated assay performance. In stage 3, a group of NCI reviewers collectively reviewed assay performance and considered the readiness of each developer to participate in the VS with a locked down assay. The NCI then selected and invited top-performing and fit-for-purpose assay developers for inclusion in the VS. The performance characteristics of assays utilizing the Alliance Reference Set will be described in a separate publication.

Data availability

Restrictions apply to the availability of these data, which were generated under transactional agreements with third parties. Data are available from the authors when publicly disclosed or upon reasonable request with the permission of third parties.

Results

Stage 1

MCD developer interest and state of the field

In 2021, NCI hosted a workshop on study design considerations (5). Subsequently, NCI issued a request for information (RFI; ref. 12) as an environmental scan to understand the landscape of MCD tests for potential inclusion in an NCI-sponsored clinical utility RCT and to determine the willingness of MCD assay developers to participate in such NCI initiatives. The NCI received 18 responses from developers with technologically diverse assays. Eleven assays used cell-free DNA, three were based on circulating tumor cells, and the remaining four assays used other analytes (circulating protein biomarkers, exosome proteins, other exosome components, physicochemical characteristics of blood components, and blood and/or urine glycans). Most assays were in an early stage of development, Early Detection Research Network (EDRN)–defined phase 2 (13, 14), but several assays were in phase 3 or 4. Review of RFI responses and follow-up virtual meetings were conducted under confidentiality disclosure agreements and are summarized as key findings in Box 1. The key findings informed planning decisions for the Alliance Reference Set Study, for example, cancer types included, specimen type, and blood collection tube type.

Box 1. Key findings on MCD assays from RFI (NOT-CA-22-033; ref. 12):

• The NCI received 18 responses from MCD assay developers.

• Analytes were diverse: cell-free DNA (11 assays), circulating tumor cells (three assays), and other biological features and characteristics (four assays).

• Machine learning and artificial intelligence were components of all assays.

• Different types of liquid specimens may be required for different MCD assays (e.g., plasma, serum, or urine).

  • Ask to describe required specimen.

  • Determine what is feasible for the potential study design.

• Biospecimen volume requirements can be variable (range was microliters to >10 mL of whole blood).

  • If collecting specimens for a repository, large volumes will be needed.

  • If using existing specimen collections, only a limited number of assays can be tested.

• Collection workflow, such as the sample collection tube, processing, and storage, impacts which stored biospecimens can be used for assay validation.

• MCD assays are at different stages in development and have diverse supporting data that are not easily compared across studies. To compare assays, data will need to be collected in a uniform manner.

• Not all MCD assays in development are in the published peer-reviewed literature.

  • Communication with industry and academic MCD assay developers under confidentiality disclosure agreements is crucial to fully understand the landscape and to support ongoing collaborations with the private sector.

• MCD assay developers are interested in working with the NCI in the future for RCTs and studies.

Development of an MCD assay application

To ensure comparability across assays and to harmonize data, NCI developed an application package and review criteria for assay developer submissions. Information requested included assay name, analytes and targeted biological features, cancers detected, specimen type, volume requirements, sensitivity by stage and cancer type, specificity, a description of studies assessing assay performance, and, where applicable, tissue-of-origin (TOO) prediction accuracy for primary and secondary organ sites. Information on suitability for inclusion in the VS, such as the ability to meet regulatory requirements, was also requested.

Workshop to engage assay developers

In May 2023, NCI hosted a public virtual workshop (15) to inform assay developers about the CSRN, plans for the VS, and the application and selection process (16). More than 300 individuals representing more than 110 institutions attended the workshop. Assay developers were invited to request the application package and submit materials to a secure online NCI-affiliated data storage system. Although the immediate objective was to identify MCD assay candidates for the VS, all applications were accepted for review, including those of possible interest for future CSRN studies.

Application prioritization

Following the public virtual workshop, NCI received 27 applications from 24 assay developers. Four applications for single cancer assays were deferred for consideration in future CSRN studies. Twenty-three applications for MCD assays from 20 developers were independently reviewed for technical merit by an NCI assay selection committee consisting of 13 members. The committee, comprised of NCI scientists and independent consultants, had expertise in cancer screening and epidemiology, biomarker development for early detection, clinical trial design, data science, in vitro diagnostics regulation, and commercialization. The review model was patterned on the NIH Peer Review process, with each application assigned to three lead reviewers. All of the selection committee members could review each application and provide input into the discussion.

Review criteria included the spectrum of cancers the assay was purported to detect, the strength of the training and validation study designs, sensitivity, specificity, and TOO prediction accuracy (if applicable). Each application received a numerical score on each criterion ranging from 1 (outstanding) to 5 (fair). Although NCI did not set specific requirements for the assays, the following attributes were prioritized: the ability to detect at least three cancer types, with one being lung cancer specifically, and the ability to detect early-stage cancers. There was no requirement for the assay to include a TOO prediction. The final score was computed as a weighted average of the five review criteria. The assigned reviewers led the discussion on the applications’ technical strengths and weaknesses, and the review committee prioritized applications after all reviews concluded. Ten assays were selected for inclusion in stage 2.

Stage 2

Evaluation of assay performance

The NCI worked with the developers to identify appropriate biospecimen sets that aligned with the assay’s specimen requirements (e.g., serum and plasma) and targeted cancers. In some instances, the reference sets used did not include all targeted cancer types. To shed light on the accuracy of the TOO prediction in real-world settings, NCI also included specimens for cancer types not targeted by the assay.

The NCI utilized three biospecimen resources to support the evaluation process (Table 1; refs. 11, 17–21). The Alliance Reference Set was created specifically to verify the performance of MCD assays. Specimens were collected under a cross-sectional study of 18 newly diagnosed cancer types and age- and sex-matched noncancer controls (11). Plasma samples were collected in Cell-Free DNA BCT Streck tubes, and blood was drawn before treatment for the cancer cases. The Prostate, Lung, Colorectal, and Ovarian (PLCO) Cohort Specimen Set is a prospective collection of blood samples obtained prior to cancer diagnosis (17). The EDRN single-cancer prospective reference sets, including the Breast Cancer Reference Set (18), Colon Cancer Reference Set (19), Lung Reference Set C (20), and PCA3 Prostate Cancer Set (21), were used collectively as a single reference set. In all, about 6,000 samples representing 2,500 unique individuals were sent to nine assay developers for testing.

Table 1.

Biospecimen resources: Alliance, PLCO, and EDRN.

​	Alliance Reference Set	PLCO biorepository	EDRN reference set
Collection details	​	​	​
EDRN phase (13, 14)	2	3	2
Biospecimen material	Plasma and buffy coat	Serum and plasma	Serum
Blood collection tube	Streck BCT	Red top blood tube (only clot activator) and EDTA	BD Vacutainer Serum Tube BD366430 (prostate and lung), red top without additive (colon); and red top (breast)
Collection years	2022–ongoing	1993–2006	2006–2012
Controls matched on	Age and sex	Age, sex, and time in storage	Age and sex
Reference	(11)	(17)	(18–21)
Number of developers shipped to	6	2	1
Cancer types shipped	​	​	​
Bladder	X	X	​
Breast	X	X	X
Colorectal	X	X	X
Endometrial	X	​	​
Esophageal/gastric	X	​	​
Liver	X	​	​
Lung	X	X	X
Lymphomaa	X	​	​
Ovarian	X	X	​
Pancreatic	X	X	​
Prostate	X	X	X
Renal	X	​	​
Head and neck	X	​	​

^aLymphoma was the only listed cancer type that was not targeted by any assay.

Although it would have been ideal to use a single reference set for all developers, not all assays were able to analyze plasma processed from Streck tubes. The PLCO and EDRN plasma and serum specimen sets have notable limitations compared with the freshly collected Alliance Reference Set. As the PLCO specimen set was collected from 1993 to 2006, the specimens are older. Specimens from the PLCO were utilized if a cancer diagnosis had occurred within six months of collection and specimens had a maximum of two freeze–thaw cycles. The EDRN cancer-specific specimen sets were collected from 2006 to 2012 and did not use a harmonized protocol across cancer types or did not come from identical populations. Specimen storage duration and prospective collection compared with cancer diagnosis may depress performance of MCD assays. Heterogeneity in collection protocols by cancer type may also affect MCD performance. The varying characteristics of the biospecimen resources and the potential impact on assay performance were recognized and served as a consideration during the review process.

Prior to accessioning of reference set samples, one developer withdrew from the process. Of the remaining nine developers that received biospecimens, six were able to utilize Streck blood collection tubes in the Alliance Reference Set, two developers opted for the PLCO specimen set (one for plasma and one for serum), and one developer opted for the EDRN reference set. Table 1 summarizes the specimen sets, matching criteria, and cancer types.

The assay developers received the specimens with the knowledge of participants’ age and sex but blinded to cancer status, performed their assays, and then returned the results to NCI through a secure online portal using a standardized reporting template. The results included a positive/negative call for each sample, a primary and secondary TOO prediction (if applicable), and a risk score (if applicable). Study statisticians calculated the assays’ specificity, sensitivity by cancer type and stage, including non-targeted cancer types, and TOO prediction accuracy (if applicable). In addition, for those assays that reported a risk score, the risk score was used to compute sensitivity at a fixed level (98%) of specificity. The assay algorithm outputs were also assessed for bias in terms of the demographic factors of age, sex and race, and ethnicity.

Although specimen testing was underway, NCI offered meetings with each developer to assess the readiness and level of support required to participate in the VS. Key considerations also included access to a Clinical Laboratory Improvement Amendments–certified facility, scalability of the assay, and stage of development to achieve design lock for the trial.

Stage 3

Assay selection

In stage 3, the 13 members of the NCI assay selection committee collectively evaluated each assay using a review template covering seven categories. Five categories for assay performance on reference set samples were numerically scored one (outstanding) through five (fair):

1. Cancer type detection capability: variety, number, and clinical impact of cancer types targeted. Consideration was given to whether cancer types already had standard-of-care screening.

2. Sensitivity of cancer detection: sensitivity using reference set specimens and agreement with the assay’s original application data. Sensitivity by stage, cancer type, and by stage within cancer type was evaluated with emphasis placed on early-stage cancer sensitivity.

3. Specificity of cancer detection: specificity using reference set specimens and agreement with the assay’s original application data.

4. TOO prediction accuracy (if applicable): accuracy using the reference set and agreement with the assay’s original application data.

5. Assay/biospecimen failure rate: percent of specimens that failed to be tested successfully.

The two remaining categories, “Readiness for Vanguard” and “Other Concerns” allowed for free text assessments by the reviewers. Other concerns included any evidence of demographic bias in the assays’ algorithmic outputs.

Each of the 13 reviewers evaluated all nine assays and provided an overall summary score. The committee deliberated about the quantitative scores, discussed the free text categories, and selected candidates to offer participation in the VS. Two developers agreed to participate in the VS (22), and they established clinical trial agreements with NCI.

Discussion

Significant resources are required for large-scale cancer screening trials; therefore, promising technologies must be prioritized for inclusion in a definitive trial. The framework described in this article can be used to select other promising technologies for evaluation in large-scale clinical trials. This process facilitated the systematic and transparent evaluation and prioritization of more than 20 assay candidates, culminating in clinical trial agreements with two developers.

The Alliance Reference Set Study was a key component of the assay evaluation process. Anticipating the need for such a reference set, the Alliance study was initiated well before the need to test the performance of the developers’ assays, thereby shortening the selection process. The Alliance Reference Set included specimens from a wide range of cancer types and all stages using a blood collection tube favored by most MCD assay developers, which allowed for the uniform evaluation of six of the nine assays. For the most part, assays using the Alliance samples received specimens from the same case and control subjects. This provided a unique opportunity to directly compare results within and among technologically diverse assays assessing distinct analytes.

Direct comparisons of tests using identical sets of blinded samples are uncommonly reported. One such study was conducted by NCI, which provided identical blood specimens prospectively collected from the PLCO screening trial to five laboratories to assess the clinical performance of ovarian cancer screening tests that used overlapping biomarker panels and different algorithms (23). We used an analogous approach to evaluate the clinical performance of assays with varying biomarkers and algorithms, across 13 cancer types, using three biospecimen resources. We layered the selection framework described in this article onto the test performance comparison, which can be leveraged for future CSRN studies.

Several facets of the evaluation framework contributed to a productive outcome. Engagement and open discussion with the MCD community at a conceptual level at scientific meetings and in multiple workshops laid the groundwork for a process to select fit-for-purpose assays. The continuous communication between NCI and assay developers under confidentiality disclosure agreements fostered the trust needed to discuss sensitive, proprietary information. Our implementation of uniform data collection allowed for the rapid analysis and review of MCD performance and developer capacity. The inclusion of nontargeted cancer types allowed reviewers to gauge the potential breadth of cancer type detection for an assay beyond the reported “targeted” cancer types. This was deemed important as shared cancer biology was hypothesized to enable detection capacity beyond targeted cancers, which can influence investigator decision-making and affect the diagnostic workup. Importantly, assay developers reported deriving value from participation, particularly the opportunity to further develop their assays. The nine companies that moved on to stage 3 each received a large set of highly curated specimens and the results of NCI’s data analysis, which some used to correct internal biases relating to the use of different populations and cancer subtypes. Lastly, the broad range of expertise of the NCI assay selection committee provided an informed perspective on finalizing the data elements, criteria, and review process.

One limitation of the process was the inability to use a single biospecimen resource to evaluate the performance of all nine assays. Engagement with experts at an NCI-hosted workshop and with RFI respondents indicated that the Streck BCT would support the majority of assays and had the advantage of minimal processing by participating institutions and extended shipping time. To meet the needs of other assays, we leveraged NCI’s biospecimen collections. Nonetheless, assays that utilized the PLCO and EDRN specimen sets were at a disadvantage because of limitations discussed above, and although we attempted to correct for this in assay scoring, there was no entirely objective method to accomplish this. Future biospecimen resources should collect a variety of specimen types and collection methods to accommodate the diverse and evolving field of MCD technologies.

The assay selection process described here is flexible and can be adapted for differing study requirements. We applied this framework in the context of cancer screening using MCD tests, but it can be adapted to other situations in which there is a breadth of technologies available for a specific clinical purpose and it is feasible to apply an independent assessment of test performance using a reference set. The framework may need to account for changing study designs, technological advances, and regulatory requirements for future CSRN studies. The process may be expanded to assess machine learning or artificial intelligence components of assays to improve transparency and interpretability and to reduce bias.

In summary, stakeholder engagement in the MCD space allowed NCI to be proactive in the assessment of new MCD technologies. By establishing the Alliance Reference Set, leveraging existing collections from PLCO and EDRN, and streamlining data collection and statistical reports, reviewers were able to efficiently evaluate MCD test performance across a range of technologies. NCI’s biospecimen reference sets enabled the generation of independent data to assist in verifying assay developers’ internal studies. All assay developers who participated were provided with independent assessments of their assays, which enabled them to make further assay refinements to enhance performance. The assay selection process resulted in participation by two assay companies in the VS and has provided a framework that can be used to verify the readiness of technologies for incorporation into large clinical trials.

Authors’ Disclosures

D.F. Ransohoff reports personal fees from NCI during the conduct of the study, as well as grants from Mercy BioAnalytics outside the submitted work. No disclosures were reported by the other authors.

Authors’ Contributions

E. LeeVan: Conceptualization, data curation, formal analysis, validation, investigation, visualization, methodology, project administration, writing–review and editing, co-first author. A.L. Skarlupka: Conceptualization, data curation, software, formal analysis, validation, investigation, methodology, writing–original draft, writing–review and editing, co-first author. C. Patriotis: Conceptualization, data curation, formal analysis, validation, investigation, visualization, methodology, writing–original draft, writing–review and editing. W.S. Rubinstein: Conceptualization, formal analysis, validation, investigation, visualization, methodology, writing–original draft, writing–review and editing. P.F. Pinsky: Conceptualization, data curation, formal analysis, validation, investigation, visualization, methodology, writing–review and editing. W. Bolton: Formal analysis, validation, investigation, writing–review and editing. A. Dickherber: Formal analysis, validation, investigation, writing–review and editing. D.C. Edelman: Formal analysis, validation, investigation, writing–review and editing. L.N. Harris: Formal analysis, Validation, investigation, writing-review and editing. H.A. Katki: Formal analysis, validation, investigation, writing–review and editing. E.B. Lavik: Formal analysis, validation, investigation, writing–review and editing. A. Martin: Formal analysis, validation, investigation, writing–review and editing. M.J.C. Ong: Formal analysis, validation, investigation, writing–review and editing. P.C. Prorok: Conceptualization, formal analysis, validation, investigation, methodology, writing–review and editing. D.F. Ransohoff: Formal analysis, validation, investigation, writing–review and editing. S.M. Temkin: Formal analysis, validation, investigation, writing–review and editing. L.M. Minasian: Conceptualization, resources, formal analysis, supervision, funding acquisition, validation, investigation, visualization, methodology, writing–review and editing, senior author.