Abstract
In cohort studies, it can be infeasible to collect specimens on an entire cohort. For example, to estimate sensitivity of multiple multi-cancer detection (MCD) assays, we desire an extra 80 mL of cell-free DNA (cfDNA) blood, but this much extra blood is too expensive for us to collect on everyone. We propose a novel epidemiologic study design that efficiently oversamples those at highest baseline disease risk from whom to collect specimens, to increase the number of future cases with cfDNA blood collection. The variance reduction ratio from our risk-based subsample versus a simple random (sub)sample (SRS) depends primarily on the ratio of risk model sensitivity to the fraction of the cohort selected for specimen collection subject to constraining the risk model specificity. In a simulation where we chose 34% of the Prostate, Lung, Colorectal, and Ovarian Screening Trial cohort at highest risk of lung cancer for cfDNA blood collection, we could enrich the number of lung cancers 2.42-fold. The standard deviation of lung-cancer MCD sensitivity was 31%-33% reduced versus SRS. Risk-based collection of specimens on a subsample of the cohort could be a feasible and efficient approach to collecting extra specimens for molecular epidemiology.
Keywords: epidemiologic sampling design, case-cohort, nested case–control, diagnostic testing
Introduction
Obtaining prospective biospecimens is crucial in molecular epidemiology. However, it may be infeasible to prospectively bank every type of biospecimen on everyone in a large cohort. For example, to study respiratory diseases, we might like to collect blood, nasal swabs, sputum, and computed tomography of the chest, but collecting and storing all specimens on 100 000 or more people may be infeasible due to invasiveness or cost.
However, it may be feasible to collect specimens on a subsample. For example, the UK Biobank is conducting imaging on an approximately 20% random subsample,1 but more efficient sampling designs are possible. We are starting from 2-phase sampling designs, which have been well-explored in the statistical and epidemiologic literature.2-6 The case–cohort and nested case–control design are specific types of 2-phase sampling,7,8 where at the first phase a cohort is sampled with biospecimens stored on everyone, and at the second phase, cases/controls are sampled to measure the biomarker of interest. Designs extending to more than 2 phases have also been explored.3,9
We propose a novel 3-phase epidemiologic study design that efficiently chooses, using baseline individual risk of each outcome, from whom to collect biospecimens. In the first phase (P1), a participant enrolls in the cohort and answers a baseline questionnaire. Using the baseline questionnaire, each participant’s individual risk of each disease of interest is calculated. In the second phase (P2), a subsample of participants are offered the opportunity to contribute a biospecimen related to outcomes for which they have high risk: (1) participants at highest risk of an outcome, and (2) a simple random sample of everyone else. The intuition behind this sampling design is to oversample people at highest risk of disease to prospectively bank biospecimens on as many future cases as possible, with a simple random sample of everyone else ensuring that everyone has a chance to be sampled into Phase 2. In Phase 3 (P3), cases/controls with biospecimens are sampled to be tested for biomarkers of interest, and the analysis is conducted.
We are motivated by the need to collect prospective biospecimens to estimate the diagnostic performance (eg, sensitivity and specificity) of multicancer detection (MCD) tests (also known as multicancer early detection tests). MCD tests show promise for screening for cancer at multiple organ sites using only a blood sample.10 However, to date, there are few studies of MCD diagnostic performance with prospectively collected blood.11 The major reason is that most MCD tests require a specific cell-free DNA (cfDNA) blood collection of large volume (≥ 10 mL) that is not typically collected in cohorts. For the Connect for Cancer Prevention (“Connect”) cohort, which plans to recruit 200 000 participants,12 we plan to collect an extra 80 mL blood for MCD tests, but it is infeasible for us to collect this much extra blood on everyone. We examine efficiency of collecting baseline cfDNA blood for various P2 sampling designs, oversampling people at high risk of at least 1 of several cancer types (along with a simple random sample of everyone else), and focus on the precision of estimating MCD test sensitivity for multiple cancer types. For different sampling strategies at P2, we conduct simulation studies using both purely simulated datasets and datasets where we simulate our proposed 3-phase epidemiologic study design in the Prostate, Lung, Colorectal and Ovarian (PLCO) Cancer Screening Trial. We investigate the effects of basing the sampling scheme on an incorrect risk model due to issues of model misspecification or lack of transportability on the accuracy and efficiency of estimates.
Methods
Subsampling designs for estimating diagnostic performance
The goal of this paper to examine the statistical efficiency of various sampling strategies for mainly estimating diagnostic test sensitivity, which is defined as the probability of a positive test result given that the individual has the disease.13 The MCD assay, a specific diagnostic test, seeks to predict whether an individual is likely to develop cancer; it serves as an early warning that can prompt medical action, such as advising the patient to undergo regular cancer screening.10,11
From a cohort, a subsample is desired from which biospecimens will be collected for later use in conducting the diagnostic test. Each sampled individual has a sample weight that is the inverse of the probability of selection for selection of the subsample.14 The sample weights are used to estimate the total count in each cell in the cohort (Table 1). The sensitivity (SEN), specificity (SPEC), positive predictive value (PPV), and negative predictive value (NPV) for the cohort are
Table 1.
For binary diagnostic tests, 2 × 2 table notation that summarizes data for an entire cohort, definition of diagnostic performance metrics, and calculation of weights for weighted estimators of diagnostic performance.
| Negative test | Positive test | Totals | |
|---|---|---|---|
| Control | N 11 | N 12 | N 1+ |
| Case | N 21 | N 22 | N 2+ |
| Totals | N +1 | N +2 | N |
 |
 |
 |
 |
Each cell count 
is estimated as 
, where N is the cohort size, 
if individual k is in the sample (
otherwise), 
when sampled individual k is in cell 
(
otherwise), and 
is the sampling weight for individual k. These estimators of 
are then used as plug-in estimators for the parameters for interest. For example, for sensitivity, the estimator is 
, where 
(similarly for 
). Estimators are implemented in the CompareTests R package.15
A key point is that Neyman16 demonstrated that unbiasedness of estimators requires that the investigator has total control of the sample design, and that everyone has a non-zero chance of being sampled. We take the first condition for granted, and only consider sampling schemes that fulfill the second. In other words, because we are using risk-based sampling to select the P2-sample, there will be selection bias in this sample.17 However, because we know the risk status on each member of the cohort, and each member of the cohort has non-zero probability of being sampled, we can use inverse-probability sampling weights to ensure that the estimation is unbiased with respect to the cohort.
Three-phase study design
A cohort of size N is recruited in P1, called the P1-sample, and a questionnaire is administered to everyone. At P2, a random subsample of size n is selected to provide biospecimens to be stored (P2-sample). The sampling probabilities in P2-sample will be based on calculating each person’s risk of each disease outcome of interest, based on their individual risk factors. If risk can be calculated immediately at enrollment, then the P2-sample can be immediately selected to request biospecimens where the P2-sample is enriched for those at high risk. The cohort is then followed, and when it is deemed time for analysis, another subsample is taken (P3-sample), this time from among only the P2-sample. The diagnostic test is conducted on the biospecimens in this P3sample and the results from the tests are analyzed. Since the focus of this design is maximizing the number of cases available for analysis in P3, the strategy for collecting the P3-sample is straightforward: We select all cases available from the P2-sample and Simple Random Sample (SRS) some number of controls. Because no extra information is collected on the P2-sample (only stored biospecimens), for analysis, P2 and P3 can be combined with sample weights estimated as in Table 1, and the weighted analysis to estimate diagnostic performance15 can be applied. For example, the weight for cases equals that of the total number of cases in the cohort divided by the number of cases in the P3-sample. Note that cases not in the P2-sample still contribute to the analysis via the computation of sampling weights for the cases. Since the method for selecting the P3-sample is fixed once the P2-sample is determined, the critical aspect of the design is in how the P2-sample is selected. Proposed strategies for this aspect are discussed in the succeeding section.
Proposed P2-sample strategies
A full cohort analysis (“COHORT”), which implicitly selects everyone for the P2-sample, is useful for comparing relative performance of P2-sample strategies. The simplest P2-sample strategy is a simple random sample of size 
, then measure biomarkers on all cases and an SRS of 
controls (“P2-SRS”). Sample weights are calculated separately for cases and controls: 
for j = 2 (cases) and 
for j = 1 (controls).
Risk-based P2-sampling strategies practically require an external risk model (we use a logistic model, but a Cox model can also be used) given 
risk factors 
available on the full cohort with corresponding coefficients 
. Risk factors could be obtained from the cohort questionnaire or biomarker tests that have been quickly conducted on the full cohort. The odds ratio (OR) score for each member i is:
 |
and risk status 
is assigned such that 
if 
for some preset 
and 0 otherwise. The 
does not include the intercept since we do not need to assume that absolute risks equal those in the cohort. This is because, to oversample those at highest risk, we need only rank cohort members by risk, which requires only the odds-ratio and not absolute risk. All individuals with 
are included in the P2-sample with probability 1, and we denote the size of this as R. Among the rest (denoted 
), we take an SRS of size 
, so that the total P2-sample size is 
. We will examine different choices of 
(or equivalently, 
), which reflect the sensitivity and specificity of the risk model at a chosen cutoff. The P3-sample selects all cases from the subsample and takes an SRS of controls of size c. For computing the poststratification weights for the P3-sample that will be used for estimation, there are 2 options. The sampling depends on case/control status and “risk status” 
. Thus, one option is to calculate 4 weights by case/control and risk status:
 |
 |
 |
 |
where 
is the weight for element k that belongs in cell ij of the cohort represented in Table 1.
Still, the estimation might be biased when considering the nature of the relationship of risk status and diagnostic test result. Indeed, a key consideration is whether the diagnostic test result is independent of risk status among cases (and also among controls). Naturally, both diagnostic test status and risk status have a dependence due to their dependence with disease status. For example, someone who will have cancer would likely have a positive risk status and a positive diagnostic test status. However, we say that there is independence within disease status if this relationship does not exist when conditioned on disease status. Following the same example, independence is implied if, for the people who will have cancer, having a positive risk status does not affect diagnostic test status and vice versa. If this independence exists, then the above weights do not need to depend on risk status, and using them will increase variance18 (see Appendix S1). Thus, a second option for weights has them depend only on case/control status:
 |
We refer to the strategy that uses these weights as P2-risk-ind. However, if this assumption of independence were false, then using the P2-risk-ind approach will bias the estimates (see Appendix S2). Thus, we also propose an adaptive strategy (“P2-risk-adaptive”) that tests for dependence and appropriately chooses P2-risk-dep or P2-risk-ind. The dependence test we chose is based on Cohen’s 
to test 
versus 
(dependence) separately for D = 1 (cases) and D = 0 (controls):
 |
where 
is the observed proportion of agreement and 
is the hypothetical proportion of agreement due purely to chance.19,20 If either test rejects the null (α = 0.05), then there is evidence of dependence, and P2-risk-dep is used, otherwise P2-risk-ind used.
Standard errors for P2-risk-ind and P2-risk-dep are readily calculated by the R package CompareTests.15 Because P2-risk-adaptive is not a “smooth” statistic across the P value threshold, asymptotic inference is challenging and we calculate standard errors using bootstrap, which has been similarly used in other studies involving diagnostic test sensitivity estimation.21-23
Misspecification or lack of transportability of the risk model
Ensuring the transportability and proper specification of risk models is challenging and requires considerable assumptions.24-26 However, we use the risk model only to define the sampling design, and as long as the sample design is under the complete control of the investigators and study individuals have a non-zero probability of being sampled, fundamental results in sampling theory18 demonstrate that unbiased estimates of cohort quantities are obtainable by using inverse-probability sampling weights. However, the efficiency of the sampling design depends on the applicability of the risk model to the cohort. We examined the issues of statistical bias and efficiency from using misspecified models in our simulation to estimate diagnostic performance of MCD tests.
Results
Relative efficiency comparing P2 simple random sampling (P2-SRS) and P2 risk-based sampling (P2-risk-ind)
The relative gain 
in number of cases for P2-risk-ind versus P2-SRS is
 |
where 
is the risk model sensitivity, 
is the fraction of the cohort selected for P2, 
is the fraction of the cohort at high risk (ie, fraction that is 
), 
is specificity, and 
is disease prevalence (see Appendix S3), which also shows that the relative efficiency is approximately 
under independence of risk model predictions and test result). Figure 1 plots 
versus risk model sensitivity, fixing lines of equal P2 sampling fractions from 10% to 90% of the cohort and computing the minimum specificity needed to ensure that the 
constraint is met. Note that case enrichment increases most rapidly when sampling fractions are low where modest levels of model sensitivity result in considerable case enrichment versus P2-SRS. For example, a 2-fold case-enrichment versus P2-SRS is obtained by a risk-based f = 10% subsample with only risk-model sensitivity s = 20%, but a larger f = 40% risk-based subsample requires risk-model sensitivity s = 80% to achieve the same 2-fold case enrichment.
Figure 1.
versus risk model sensitivity for different sampling fractions. Each curve plots the relative case enrichment for a given sampling fraction of the method versus simple random (sub)sample (SRS) as risk model sensitivity is increased. The legend shows the sampling fraction as well as the minimum specificity for this sampling fraction to ensure that 
.
Simulations on the performance of P2 sampling strategies
We initialize cohort size n = 200 000, P2 sample size n = 100 000 (ie, we choose an f = 50% subsample), and disease prevalence P = 0.02. For the diagnostic test, we set the sensitivity (msen = 0.255) and specificity (mspec = 0.995). For each setting, we specify the sensitivity (rsen) and specificity (rspec) of the risk model used in P2. For each strategy, we do the following:
Implemented the sampling strategy at P2. We selected all of the individuals who are risky based on the risk model and take an SRS from among the nonrisky to have a total sample size of n.
Conducted the analysis using the CompareTests15 R function with the actual disease status as the gold standard, MCD result on the selected P2 sample as the test to be evaluated, and stratification based on the sampling strategy used.
Calculated MCD sensitivity, specificity, PPV, NPV, and number of cases selected
Iterated 2000 times and averaged estimates, bias, and empirical variances.
Figure 2 shows the relative efficiency and mean squared error (MSE) of each strategy compared with P2-SRS for estimating diagnostic test sensitivity. Naturally, the full cohort performs best with a relative efficiency of 1.4 over P2-SRS (ie, reduces variance by 29%). Even at a low Youden’s index of 0.2, P2-risk-ind and P2-risk-dep outperform P2-SRS consistently in both relative efficiency and relative MSE. P2-risk-dep is expected to be less efficient than P2-risk-ind when there is no dependence, but it is also less efficient than P2-SRS when risk model specificity is lowered from 0.85 to 0.65. This shows the importance of P2-risk-adaptive, which chooses P2-risk-ind over P2-risk-dep in almost all simulations and thus always perform better than P2-SRS (although not quite as good as P2-risk-dep). Table S1. Shows that simulation results matched the expected case enrichment based on the expression derived for 
. Figure S1 shows simulation results for specificity, PPV, and NPV.
Figure 2.
Simulation results for each P2 sampling strategy, assuming no dependence between risk status and diagnostic test status within disease status. Risk model specificity is fixed at 0.85 for A-B and 0.65 for C-D while sensitivity is made to vary such that overall model quality (Youden’s indices) varies from 0.2 to 0.6. MSE, mean squared error; SRS, simple random (sub)sample.
We next introduced dependence in the simulations by forcing some randomly selected proportion of risk statuses to agree with the diagnostic test results (Figure 3). P2-risk-ind is badly biased in all situations. In contrast, P2-risk-adaptive chose P2-risk-dep over P2-risk-ind in every simulation, avoided bias, and outperformed P2-SRS. We note that the advantage of P2-risk-adaptive over P2-SRS increases as dependence increases. In the extreme case where the risk status and the diagnostic test result always agree, P2-risk-adaptive (and P2-risk-dep) provide the exact same result as the full cohort.
Figure 3.
Simulation results when 20% agreement (dependence) is imposed between risk status and diagnostic test status. Displayed risk model sensitivity/specificity are only true within the proportion of the data (80%) where agreement was not forced. MSE, mean squared error; SRS, simple random (sub)sample.
We next limited the subsample to only f = 12.5% of the cohort (Figure 4). There remains considerable increase in the relative efficiency of P2-risk-ind over P2-SRS, but not as much for P2-risk-adaptive versus P2-SRS. For smaller P2 sample sizes, P2-risk-adaptive is hindered by unnecessary added stratification. Figure S2 reduces the significance level from 0.05 to 0.01, and for the same settings as in Figure 4, this allows P2-risk-adaptive to outperform P2-SRS consistently. However naturally, reducing the significance level will adversely affect P2-risk-adaptive performance when there is a strong dependence in the data. Thus, when f is small and if there is prior information available on whether risk and diagnostic test result are independent within disease status, this information may be used to select between P2-risk-ind and P2-risk-dep instead of using P2-risk-adaptive.
Figure 4.
Simulation results for each P2 sampling strategy (P2 sample size =25 000), assuming no dependence between risk status and diagnostic test status within disease status. Risk model specificity is fixed at 0.85 for A-B and 0.65 for C-D while sensitivity is made to vary such that overall model quality (Youden’s indices) varies from 0.2 to 0.6. MSE, mean squared error; SRS, simple random (sub)sample.
Application to deciding on whom to collect specimens to estimate diagnostic performance of multicancer early detection (MCD) tests
To help decide the P2 sampling plan for the National Cancer Institute Connect cohort for obtaining extra cfDNA blood samples, we simulated the MCD test result for the preexisting PLCO cohort (n = 145 044) without dependence on risk status, and then examined the relative efficiency of different P2 sampling plans vs. SRS (P2-SRS). For these simulations, the P2 sample size is fixed to n = 50 000 (34% subsample) and MCD sensitivity = 0.255 (same for each cancer organ site) and specificity = 0.995. The P3 sample selects, among those selected at P2, all cancers over 5 years and a random sample of 1000 controls. We calculated MCD sensitivity for detecting any cancer, and for specifically lung, breast, colorectal, and pancreatic cancers.
The risk model is a logistic regression model fit to PLCO data, using lung cancer within 5 years as the outcome and smoking history, any family history of cancer, gender, race, age, and body mass index as predictors. The 
risk cutoff is selected to maximize Youden’s index for detecting lung cancer (0.56), selecting 36 915 risky individuals, which comprised 74% of the P2-sample, with the remaining 26% randomly selected among the nonrisky using SRS to ensure that weighted estimators can reflect the full PLCO cohort. Although separate risk models could be used for each disease of interest (see Discussion), for simplicity, a single risk model providing high Youden’s index for a single disease that is correlated with other diseases of interest, might be used in practice, which will affect case enrichment for every disease.
Table 2 shows the results of the application to PLCO. For sensitivity of MCD for lung cancer, lung cancers are enriched 2.42-fold (from 417 to 1009 cases), and thus the 34% risk-based subsample included 83% of the 1211 lung cancers. The case enrichment led to a 1.45 relative-efficiency gain over SRS (31%-33% reduction in standard error) for estimating MCD test sensitivity for lung cancer. Lung-cancer enrichment drove the 1.29-fold enrichment for all cancers (from 4163 to 5352 cases). Note that, in this situation, the risk model was fitted to PLCO and applied to PLCO. Hence it is by definition properly specified and transportable to PLCO, and estimators were unbiased with large reductions in standard error.
Table 2.
Simulation in PLCO data for estimating simulated MCD test sensitivity (true:0.255) from each P2 sampling strategy.a
| Lung cancer model | Estimate | # of cases | Empirical SD | RE to S1 | Case enrichment | Bootstrap SD | |
|---|---|---|---|---|---|---|---|
| Any cancer | COHORT | 0.25508 | 12 080 | 0.0039 | 1.70 | 2.90 | 0.0042 |
| P2-SRS | 0.25513 | 4163 | 0.0067 | 1.00 | 1.00 | 0.0068 | |
| P2-risk-ind | 0.25514 | 5352 | 0.0056 | 1.20 | 1.29 | 0.0060 | |
| P2-risk-adaptive | 0.25487 | 5352 | 0.0061 | 1.10 | 1.29 | 0.0069 | |
| Lung | COHORT | 0.25443 | 1211 | 0.0126 | 1.63 | 2.91 | 0.0131 |
| P2-SRS | 0.25445 | 417 | 0.0206 | 1.00 | 1.00 | 0.0222 | |
| P2-risk-ind | 0.25449 | 1009 | 0.0137 | 1.50 | 2.42 | 0.0141 | |
| P2-risk-adaptive | 0.25432 | 1009 | 0.0142 | 1.45 | 2.42 | 0.0146 | |
| Breast | COHORT | 0.25541 | 2005 | 0.0099 | 1.67 | 2.90 | 0.0100 |
| P2-SRS | 0.25495 | 692 | 0.0166 | 1.00 | 1.00 | 0.0175 | |
| P2-risk-ind | 0.25565 | 618 | 0.0171 | 0.97 | 0.89 | 0.0154 | |
| P2-risk-adaptive | 0.25528 | 618 | 0.0177 | 0.93 | 0.89 | 0.0163 | |
| Colon | COHORT | 0.25508 | 994 | 0.0136 | 1.72 | 2.90 | 0.0138 |
| P2-SRS | 0.25471 | 342 | 0.0234 | 1.00 | 1.00 | 0.0211 | |
| P2-risk-ind | 0.25531 | 443 | 0.0202 | 1.16 | 1.29 | 0.0199 | |
| P2-risk-adaptive | 0.25502 | 443 | 0.0211 | 1.11 | 1.29 | 0.0213 | |
| Pancreas | COHORT | 0.25540 | 222 | 0.0290 | 1.74 | 2.91 | 0.0308 |
| P2-SRS | 0.25668 | 76 | 0.0504 | 1.00 | 1.00 | 0.0525 | |
| P2-risk-ind | 0.25558 | 105 | 0.0426 | 1.18 | 1.38 | 0.0463 | |
| P2-risk-adaptive | 0.25503 | 105 | 0.0439 | 1.15 | 1.38 | 0.0511 |
Abbreviations: MCD, multi-cancer detection; PLCO, Prostate, Lung, Colorectal and Ovarian Cancer Screening Trial; risk-ind, strategy assuming independence; risk-adaptive, strategy using a test to check for independence; SRS, simple random (sub)sample.
aThe bootstrap SD was computed from only one of the simulated datasets using 200 bootstrap replicates and hence is not expected to be close to the empirical SD. Results are shown for using the lung cancer model as the risk calculator and using a multicancer model as the risk calculator.
Because pancreatic and colorectal cancers share risk factors with lung cancer, these cancers are also enriched by using a lung-cancer risk model (pancreas: 76 to 105 (1.38-fold); colorectal: 342 to 443 (1.29-fold)), and sensitivity estimates were unbiased and standard errors were reduced 10%-15%. Note that, in this situation, the risk model is misspecified (or alternatively, not transportable), but because risks of pancreatic and colorectal cancers are correlated with lung cancer risk, some standard error reduction is obtained.
In contrast, female breast cancer does not share risk factors with lung cancer (except for age), and oversampling people based on lung cancer risk will oversample men, who have relatively negligible risk of breast cancer. Oversampling people at high risk of lung cancer still provided unbiased sensitivity estimates but led to a deficit of breast cancers (0.89-fold, from 692 to 618 cases) that slightly (3%-8%) increased standard error. Note that, in this situation, the risk model is badly misspecified, and although statistical theory ensures unbiased estimators, standard errors are increased.16
Finally, we noted that the OR score, which uses only the odds ratios (ie, leaves out the intercept), ranks people by risk correctly without concern that the intercept might be misspecified. To demonstrate this, we simulated misspecifying the intercept (Table S2), and the 
score was unaffected. Table S2 also shows that if an absolute risk score were instead used with an underestimated intercept (yet used a fixed risk cutoff), then the number of cases selected across the different cancers was reduced, which reduces the efficiency gain of the sampling scheme.
Discussion
When it is infeasible to collect certain biospecimens on all cohort members, we proposed selecting a risk-based subsample of the cohort on which to collect all biospecimens. Our proposed risk-based 3-phase design enriches the number of cases with stored biospecimens to improve statistical efficiency. Case enrichment depends on having a risk model with high sensitivity at the chosen subsampling fraction, and hence a cohort with smaller subsampling fraction benefits more from risk-based subsampling. For example, a 2-fold case-enrichment, versus simple random subsampling, is obtained by a risk-based 10% subsample with only risk-model sensitivity 20%, but a larger 40% risk-based subsample requires risk-model sensitivity of 80% to achieve the same 2-fold case enrichment. We focused on estimating diagnostic test characteristics for a study of MCD test performance being planned for the National Cancer Institute Connect cohort. To illustrate how our sampling design can be applied, we simulated MCD test results in the PLCO cohort as an example and selected 34% of the cohort at highest risk for lung cancer. This enriched lung cancers 2.42-fold (includes 83% of all lung cancers) and all cancers 1.29-fold. This also enriched pancreatic cancers 1.38-fold and colorectal cancers 1.29-fold because these cancers share risk factors with lung cancer. However, there may be a deficit in cancers not sharing risk factors with lung cancer, such as breast cancers (0.89-fold, from 692 to 618 cases) that slightly (3%-8%) increased the standard error for estimating MCD test sensitivity for breast cancer.
The risk-based sampling plan we examined has only 2 strata: oversampling those at highest risk (~75% of the subsample) and a simple random sample of the rest (~25% of the subsample). High risk people are oversampled, which reduces variance, and the random sample of the rest ensures that estimators remain unbiased for the entire cohort because finite inverse-probability sampling weights are known for the entire cohort and are applied in the analysis to account for the sampling plan. As a technical statistical issue, our estimators are sensitive to the dependence between risk status and diagnostic test result within disease status, but we proposed an adaptive estimator (P2-risk-adaptive) that in simulations provided a good bias-variance tradeoff regardless of dependence. For simplicity, it may be preferable to use P2-risk-ind if it is known that there is little dependence between risk status and diagnostic test result within disease status. We constructed a flowchart in Figure 5 encompassing these recommendations on how to implement the design. Once data is obtained, analysis can be done with the R CompareTests15 package and we provide sample code in Appendix S4.
Figure 5.
Flowchart for implementing the sampling design. SRS, simple random (sub)sample.
The key to gaining efficiency with risk-based subsampling is a risk model that is predictive of the diseases of interest for the data set used for the application. We showed that even a dichotomous stratification can lead to efficiency loss when the risk model is not better than SRS at identifying future cases, but that this loss is easily overcome by gains from case enrichment yielded when the risk model is much better. Technically, if a highly predictive risk model is available, it may be better to use more levels of risk stratification, even possibly sampling probabilities proportional to the individual risk values. However, at least for cancer studies, the consensus seems to be that highly predictive and externally validated models are difficult to come by.27-29 Even models that have high AUC can have considerable transportability issues that can reduce their actual AUC when applied to an external dataset. As such, what we recommend as a good default is keeping the stratification dichotomous and thereby minimizing the variance due to additional levels of stratification that can cause more variable sample weights.
More research needs to be done on risk-model misspecification. We provided some simulation results that show the effects of having a misspecified intercept term for the PLCO dataset, but note that there are innumerable ways in which the risk model may be misspecified. Still, although model predictiveness is crucial, we showed that even a modestly predictive model can meaningfully enrich the number of cases for small subsamples. In addition, it must be noted that there are a number of practical issues in any cohort study that can arise and must be addressed in the implementation of the sampling designs we proposed. For example, it is possible for individuals selected for the P2 sample to be lost to follow-up or to not consent to biospecimen collection. This can be addressed by adding an additional stratum for nonresponse. It may also be possible for there to be missing data in the preliminary information gathering on the full cohort which can affect the ability to assign risk status on some individuals. Imputation methods may be used, or modifications to the risk model may be done.
Cohorts typically examine multiple diseases but oversampling for each of them is challenging. In our example, we oversampled those at high risk of lung cancer, which also naturally oversamples all smoking-related diseases, and because lung cancer has such high incidence, all cancers were also oversampled. However, this approach can undersample some diseases—in this situation, breast cancers. A key area of future research is developing approaches to using risk models optimized for each disease of interest. For example, risk models with polygenic risk scores (PRS) are becoming more predictive30,31 and could be candidates for cohorts that will obtain blood and quickly calculate PRS for each cohort member for use in oversampling people at high risk of multiple disease to obtain extra specimens.
There are many avenues for further research in risk-based selection of a subsample for specimen collection. The analysis methods need to be extended to measures of association, such as odds ratios and hazard ratios, and to longitudinally collected biospecimens. More research needs to be done on the use of risk models transported to the cohort from outside data, especially sensitivity to transporting incorrect odds-ratios. How to optimally set the parameters for the risk-based sampling plan is an open question, and in particular, how to identify a cost-effective subsampling fraction.
Obtaining prospective biospecimens is crucial to molecular epidemiology. We hope that this work raises awareness to working epidemiologists of the possibility of obtaining extra biospecimens on efficient risk-based subsamples of the cohort and using available software to analyze the data.
Supplementary Material
Contributor Information
Mark Louie F Ramos, Division of Cancer Epidemiology and Genetics, National Cancer Institute, National Institutes of Health, Department of Health and Human Services, Bethesda, MD 20892, United States.
Anil K Chaturvedi, Division of Cancer Epidemiology and Genetics, National Cancer Institute, National Institutes of Health, Department of Health and Human Services, Bethesda, MD 20892, United States.
Barry I Graubard, Division of Cancer Epidemiology and Genetics, National Cancer Institute, National Institutes of Health, Department of Health and Human Services, Bethesda, MD 20892, United States.
Hormuzd A Katki, Division of Cancer Epidemiology and Genetics, National Cancer Institute, National Institutes of Health, Department of Health and Human Services, Bethesda, MD 20892, United States.
Supplementary material
Supplementary material is available at American Journal of Epidemiology online.
Funding
This study was supported by the Intramural Research Program of the US National Institutes of Health.
Conflict of interest
None declared.
Disclaimer
The NIH had no role in the design and conduct of the study; in the collection, analysis, and interpretation of the data; or in the preparation, review, or approval of the manuscript. The authors alone are responsible for the views expressed in this article and they do not necessarily represent the views, decisions, or policies of the institutions with which they are affiliated.
Data availability
The data set used for this study were obtained from the US National Cancer Institute. Researchers can request access to the data from the Prostate, Lung, Colorectal and Ovarian (PLCO) Cancer Screening Trial online: https://cdas.cancer.gov/plco/
References
- 1. Littlejohns TJ, Holliday J, Gibson LM, et al. The UK Biobank imaging enhancement of 100,000 participants: rationale, data collection, management and future directions. Nat Commun. 2020;11(1):2624. 10.1038/s41467-020-15948-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Breslow NE, Lumley T, Ballantyne CM, et al. Improved Horvitz-Thompson estimation of model parameters from two-phase stratified samples: applications in epidemiology. Stat Biosci. 2009;1(1):32-49. 10.1007/s12561-009-9001-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Amorim G, Scott A, Wild C. Chapter 12: Multi-phase sampling. In: Borgan, Ø, Breslow N, Chatterjee N, et al. Handbook of Statistical Methods for Case-Control Studies. Accessed January 13, 2023. https://www.mn.uio.no/math/english/research/groups/statistics-data-science/handbook-of-case-control-studies/chapter-12/index.html
- 4. Schildcrout JS, Haneuse S, Tao R, et al. Two-phase, generalized case-control designs for the study of quantitative longitudinal outcomes. Am J Epidemiol. 2020;189(2):81-90. 10.1093/aje/kwz127 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Laitila T, Olofsson J. A two-phase sampling scheme and πps designs. J Stat Plan Inference. 2011;141(5):1646-1654. 10.1016/j.jspi.2010.11.004 [DOI] [Google Scholar]
- 6. White JE. A two stage design for the study of the relationship between a rare exposure and a rare disease. Am J Epidemiol. 1982;115(1):119-128. 10.1093/oxfordjournals.aje.a113266 [DOI] [PubMed] [Google Scholar]
- 7. Liddell FDK, McDonald JC, Thomas DC, et al. Methods of cohort analysis: appraisal by application to asbestos mining. J R Stat Soc Ser Gen. 1977;140(4):469-483. 10.2307/2345280 [DOI] [Google Scholar]
- 8. Prentice RL. A case-cohort design for epidemiologic cohort studies and disease prevention trials. Biometrika. 1986;73(1):1-11. 10.1093/biomet/73.1.1 [DOI] [Google Scholar]
- 9. Whittemore AS, Halpern J. Multi-stage sampling in genetic epidemiology. Stat Med. 1997;16(2):153-167. 10.1002/(SICI)1097-0258(19970130)16:2<153::AID-SIM477>3.0.CO;2-7 [DOI] [PubMed] [Google Scholar]
- 10. Kisiel JB, Papadopoulos N, Liu MC, et al. Multicancer early detection test: preclinical, translational, and clinical evidence—generation plan and provocative questions. Cancer. 2022;128(S4):861-874. 10.1002/cncr.33912 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Lennon AM, Buchanan AH, Kinde I, et al. Feasibility of blood testing combined with PET-CT to screen for cancer and guide intervention. Science. 2020;369(6499):eabb9601. 10.1126/science.abb9601 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. National Cancer Institute . Connect for Cancer Prevention Study. 2019. Accessed April 20, 2023. https://dceg.cancer.gov/research/who-we-study/cohorts/connect
- 13. Yerushalmy J. Statistical problems in assessing methods of medical diagnosis, with special reference to X-ray techniques. Public Health Rep 1896-1970. 1947;62(40):1432-1449. 10.2307/4586294 [DOI] [PubMed] [Google Scholar]
- 14. Horvitz DG, Thompson DJ. A generalization of sampling without replacement from a finite universe. J Am Stat Assoc. 1952;47(260):663-685. 10.1080/01621459.1952.10483446 [DOI] [Google Scholar]
- 15. Katki HA, Li Y, Edelstein DW, et al. Estimating the agreement and diagnostic accuracy of two diagnostic tests when one test is conducted on only a subsample of specimens. Stat Med. 2012;31(5):436-448. 10.1002/sim.4422 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Neyman J. On the two different aspects of the representative method: the method of stratified sampling and the method of purposive selection. J R Stat Soc. 1934;97(4):558-625. 10.2307/2342192 [DOI] [Google Scholar]
- 17. Lu H, Cole SR, Howe CJ, et al. Toward a clearer definition of selection bias when estimating causal effects. Epidemiol Camb Mass. 2022;33(5):699-706. 10.1097/EDE.0000000000001516 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Cochran WG. Sampling Techniques. 3rd ed. John Wiley & Sons; 1977. [Google Scholar]
- 19. McHugh M. Interrater reliability: the kappa statistic. Biochem Med. 2012;22(3):276-282. 10.11613/BM.2012.031 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Fleiss JL, Cohen J, Everitt BS. Large sample standard errors of kappa and weighted kappa. Psychol Bull. 1969;72(5):323-327. 10.1037/h0028106 [DOI] [Google Scholar]
- 21. Platt RW, Hanley JA, Yang H. Bootstrap confidence intervals for the sensitivity of a quantitative diagnostic test. Stat Med. 2000;19(3):313-322. 10.1002/(SICI)1097-0258(20000215)19:3<313::AID-SIM370>3.0.CO;2-K [DOI] [PubMed] [Google Scholar]
- 22. Tang NS, Luo XG. Confidence interval construction for sensitivity difference of two continuous-scale diagnostic tests at the fixed level of two specificities. Stat Probab Lett. 2015;97:32-40. 10.1016/j.spl.2014.10.016 [DOI] [Google Scholar]
- 23. Rutter CM. Bootstrap estimation of diagnostic accuracy with patient-clustered data. Acad Radiol. 2000;7(6):413-419. 10.1016/S1076-6332(00)80381-5 [DOI] [PubMed] [Google Scholar]
- 24. Westreich D, Edwards JK, Lesko CR, et al. Transportability of trial results using inverse odds of sampling weights. Am J Epidemiol. 2017;186(8):1010-1014. 10.1093/aje/kwx164 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Steingrimsson JA, Gatsonis C, Li B, et al. Transporting a prediction model for use in a new target population. Am J Epidemiol. 2023;192(2):296-304. 10.1093/aje/kwac128 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Cole SR, Edwards JK, Breskin A, et al. Illustration of 2 fusion designs and estimators. Am J Epidemiol. 2023;192(3):467-474. 10.1093/aje/kwac067 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Shen A, Wei X, Zhu F, et al. Risk prediction models for breast cancer-related lymphedema: a systematic review and meta-analysis. Eur J Oncol Nurs. 2023;64:102326. 10.1016/j.ejon.2023.102326 [DOI] [PubMed] [Google Scholar]
- 28. Harrison H, Thompson RE, Lin Z, et al. Risk prediction models for kidney cancer: a systematic review. Eur Urol Focus. 2021;7(6):1380-1390. 10.1016/j.euf.2020.06.024 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Gray EP, Teare MD, Stevens J, et al. Risk prediction models for lung cancer: a systematic review. Clin Lung Cancer. 2016;17(2):95-106. 10.1016/j.cllc.2015.11.007 [DOI] [PubMed] [Google Scholar]
- 30. Lee SM, Shivakumar M, Xiao B, et al. Genome-wide polygenic risk scores for hypertensive disease during pregnancy can also predict the risk for long-term cardiovascular disease. Am J Obstet Gynecol. 2023;229(3):298.e1-298.e19. 10.1016/j.ajog.2023.03.013 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Levin MG, Rader DJ. Polygenic risk scores for dyslipidemia and atherosclerotic cardiovascular disease: Progress toward clinical implementation. Best Pract Res Clin Endocrinol Metab. 2023;37(3):101702. 10.1016/j.beem.2022.101702 [DOI] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The data set used for this study were obtained from the US National Cancer Institute. Researchers can request access to the data from the Prostate, Lung, Colorectal and Ovarian (PLCO) Cancer Screening Trial online: https://cdas.cancer.gov/plco/