Abstract
Background:
Recent guidelines recommend the Lynch Syndrome prediction models MMRPredict, MMRPro, and PREMM1,2,6 for the identification of MMR gene mutation carriers. We compared the predictive performance and clinical usefulness of these prediction models to identify mutation carriers.
Methods:
Pedigree data from CRC patients in 11 North American, European, and Australian cohorts (6 clinic- and 5 population-based sites) were used to calculate predicted probabilities of pathogenic MLH1, MSH2, or MSH6 gene mutations by each model and gene-specific predictions by MMRPro and PREMM1,2,6. We examined discrimination with area under the receiver operating characteristic curve (AUC), calibration with observed to expected (O/E) ratio, and clinical usefulness using decision curve analysis to select patients for further evaluation. All statistical tests were two-sided.
Results:
Mutations were detected in 539 of 2304 (23%) individuals from the clinic-based cohorts (237 MLH1, 251 MSH2, 51 MSH6) and 150 of 3451 (4.4%) individuals from the population-based cohorts (47 MLH1, 71 MSH2, 32 MSH6). Discrimination was similar for clinic- and population-based cohorts: AUCs of 0.76 vs 0.77 for MMRPredict, 0.82 vs 0.85 for MMRPro, and 0.85 vs 0.88 for PREMM1,2,6. For clinic- and population-based cohorts, O/E deviated from 1 for MMRPredict (0.38 and 0.31, respectively) and MMRPro (0.62 and 0.36) but were more satisfactory for PREMM1,2,6 (1.0 and 0.70). MMRPro or PREMM1,2,6 predictions were clinically useful at thresholds of 5% or greater and in particular at greater than 15%.
Conclusions:
MMRPro and PREMM1,2,6 can well be used to select CRC patients from genetics clinics or population-based settings for tumor and/or germline testing at a 5% or higher risk. If no MMR deficiency is detected and risk exceeds 15%, we suggest considering additional genetic etiologies for the cause of cancer in the family.
Lynch Syndrome accounts for approximately 3% of colorectal cancers (CRC). It is caused by germline mutations in the DNA mismatch repair (MMR) system involving the MLH1, MSH2, MSH6, PMS2, and EPCAM genes (1,2). If Lynch Syndrome is diagnosed in a patient with CRC, he or she may benefit from more intensive post-treatment colonoscopic surveillance, more extensive surgery, and management of extracolonic cancer risks. Furthermore, identification of a Lynch Syndrome mutation in individuals with CRC has implications for their families because carriers have a 35% to 75% lifetime risk of developing CRC and other cancers, often at young ages (3,4). Early identification of these individuals allows for implementation of cancer prevention strategies such as intensified surveillance, prophylactic surgery, and/or chemoprevention to reduce cancer risks and improve survival (5).
The identification of Lynch Syndrome has traditionally relied on screening via clinical criteria such as the Amsterdam or Revised Bethesda guidelines (6–8). Systematic molecular tumor testing is increasingly supported for newly diagnosed patients with CRC, either as “reflex testing” (all patients undergo microsatellite instability [MSI] and/or immunohistochemistry [IHC] testing for protein expression of the MMR genes related to Lynch Syndrome) or based on age (1,9,10). Recent guidelines by the National Comprehensive Cancer Network (NCCN) and the US Multi-Society Task Force on CRC for the risk assessment and management of Lynch Syndrome recommend genetic evaluation if the predicted risk of carrying an MMR mutation is 5% or higher using one of three prediction models: MMRPro, MMRPredict, or PREMM1,2,6 (11–15). These prediction models quantify an individual’s risk of carrying an MMR gene mutation and can support decision-making regarding genetic evaluation, including germline testing or molecular tumor testing (16). However, their performance in diverse populations has not systematically been compared. We aimed to externally validate and assess the potential clinical usefulness of MMRPro, MMRPredict, and PREMM1,2,6 for selecting patients with MMR gene mutations in a multicenter international study.
Methods
Data Sources and Patient Eligibility
Individual-level data were obtained from eleven international cohorts of patients with CRC: six were clinic based and five population based (Supplementary Materials, available online). Patients with CRC and molecular tumor testing and/or MMR gene mutational analyses results were eligible. Only one individual per family (referred to as the proband) was included for analysis, and patients with polyposis syndromes were excluded.
The clinic-based cohorts recruited patients through genetics clinics and/or family cancer registries and included the: 1) Medical Genetics Program of Newfoundland (Newfoundland, Canada), 2) Colon Cancer Family Registry (CCFR; http://epi.grants.cancer.gov/CFR/) (17), 3) Dana Farber Cancer Institute Gastrointestinal Cancer Genetics and Prevention Program (Boston, MA), 4) participating centers in the Hereditary Cancer Group of the Spanish Medical Oncology Society (SEOM), 5) Erasmus MC Genetic Registry (Rotterdam, the Netherlands) (18), and 6) participating centers in the Fondazione IRCCS Istituto Nazionale Tumori (Milan, Italy). The population-based cohorts included the: 1) Newfoundland Colorectal Cancer Registry (19), 2) CCFR (17), 3) EPICOLON Consortium (Spain) (20), 4) LIMO Study group (the Netherlands) (10), and 5) the Ohio State University (1). Information regarding the evaluation process for DNA mutational analysis and/or molecular tumor testing by specific site is as previously described (1,10,17–20). This study was approved by the Dana-Farber/Harvard Cancer Center and Columbia University Medical Center Institutional Review Boards.
Variables for Risk Prediction Models
The primary outcome was MMR gene mutation carrier status for the most common genes, MLH1, MSH2, and MSH6. PMS2 gene mutation carriers were not included, as few sites conducted germline testing for PMS2 mutations. Patients without germline testing results were classified as noncarriers if tumor testing showed no evidence of MMR deficiency. Each site provided deidentified datasets to Dana-Farber/Harvard Cancer Center and Columbia University investigators for analysis (RO, RM, CA, FK). Data for probands included demographic information, cancer history (including ages of cancer diagnoses and date of last follow-up), tumor testing results, and results of germline testing. Family history of cancer was limited to first-degree relatives (FDR) or second-degree relatives (SDR) affected with Lynch Syndrome cancers (colon, endometrial, stomach, ovaries, urinary tract, small intestine, pancreas, bile ducts, brain, sebaceous glands), including ages of diagnosis and/or date of last follow-up. For relatives unaffected by cancer, the age, sex, and date of last follow-up were included.
For every patient, predicted probabilities were estimated for carrying an MMR gene mutation using the MMRPredict, PREMM1,2,6, and MMRPro models (Supplementary Materials, available online) (13–15). The MMRPredict and PREMM1,2,6 predictions were generated using published formulas, and for MMRPro probabilities were derived using software provided by the developing investigators. Predictions were verified by comparison of probabilities from our calculations with those from web-based calculators for a sample of patients. Model comparisons were based on pedigree data alone and did not incorporate tumor testing information.
Data Analysis
We compared predicted probabilities from each model with observed frequencies of mutations (MLH1, MSH2, MSH6, or any of these) in each cohort. We stratified by cohort type (population vs clinic) and considered site-specific results before pooling data over sites. We tested for differences in predicted probabilities by cohort in the pooled data by modeling an interaction term (cohort*logit of predicted probability) in logistic regression with any one of the mutations as the outcome. An interaction term with a P value of less than .05 indicated that the relation between predicted probabilities and mutation status varied by cohort.
Discrimination and Calibration
Discrimination is the model’s ability to differentiate between a mutation carrier and noncarrier. It was assessed by the area under the receiver operating characteristic curve (AUC). An AUC of 0.5 reflects performance of a coin flip, while 1.0 is perfect. Calibration is the agreement between observed and predicted mutation frequencies and can be depicted graphically (21). Systematic under- or overestimation (‘calibration-in-the-large’) was quantified by the intercept in a logistic regression model with the log odds of predictions as an offset variable: y ~ offset(log odds[prediction]), with y indicating the presence of a genetic mutation as a binary outcome, and prediction the predicted probability of that mutation. For ease of interpretation, we converted the intercept estimates to observed to expected (O/E) ratios: O/E = exp(intercept). We also estimated a calibration slope to indicate the agreement with the 45-degree line in a validation plot by logistic regression analysis: y ~ log odds(prediction). An intercept of zero (O/E = 1) and calibration slope of 1 indicate perfect calibration (21). All statistical tests were two-sided.
Clinical Usefulness
When models are used to guide decisions, ie, for further diagnostic testing for presence of a mutation, decision curve analysis has been advocated to quantify the potential clinical usefulness, considering both true-positive and false-positive classifications (22–26). A decision curve shows the net benefit of using a model for a range of potential decision thresholds. The net benefit is the sum of the number of true positives (mutation carriers for whom benefit is obtained) minus a weighted number of false-positive classifications (who should not have been tested): NB = (TP – wFP)/n. Here n is the total sample size and w is the relative weight of the harm of unnecessary testing versus the benefit of identifying a mutation carrier. The weight w is defined by the threshold probability that is applied to define at-risk patients that need genetic testing. For example, a threshold of 5% implies that we value unnecessary testing as 1/19 as important as identifying a mutation carrier. We calculated the net benefit of each prediction model and two reference strategies: test none or test all. We considered threshold probabilities between 0% (very liberal testing) and 20% (restricted testing of high-risk probands) and focused on the threshold of 5%, in line with current guidelines (11,12). The model with the highest net benefit is the most clinically useful considering the weighted sum of true- and false-positive classifications. Additional information on net benefit analysis and its interpretation is provided in the Supplementary Materials (available online). We used R version 2.8.1 (R Foundation for Statistical Computing, Vienna, Austria) for analyses.
Results
We studied 5755 individuals, including 2304 from clinic-based and 3451 from population-based CRC registries. The median age of CRC diagnosis in the clinic-based cohorts was between 45 and 50 years, while patients in the population-based cohorts were older (median age = 60–70 years) (Tables 1 and 2). The prevalence of MMR gene mutations in the clinic-based cohorts was 23% (539/2304), with similar numbers of MLH1 and MSH2 mutations (237 and 251 respectively) and fewer MSH6 mutations (51/2304, 2.2%). The prevalence of gene mutations was lower in the population-based cohorts (150/3451, 4.4%) (Table 2), with more MSH2 than MLH1 and MSH6 mutations (71 vs 47 and 32, respectively).
Table 1.
Characteristics of participants in six clinic-based cohorts of probands assessed for Lynch Syndrome*
| Characteristics | Total (n = 2304) |
CCFR (n = 529) |
DFCI (n = 229) |
Milan, Italy (n = 232) |
Newfoundland, Canada (n = 120) |
Rotterdam, Netherlands (n = 514) |
Spanish Consortium (n = 680) |
|---|---|---|---|---|---|---|---|
| Male, %, No. (%) | 1136 (49.3) | 263 (49.7) | 99 (43.3) | 118 (50.9) | 65 (54.2) | 243 (47.3) | 348 (51.2) |
| Median age of CRC diagnosis, y (IQR) | 46 (39–55) | 45 (38–51) | 43 (35–50) | 44 (37–53) | 53 (43–62) | 51 (42–61) | 45 (39–55) |
| Mutation carriers, No. (%) | |||||||
| Any mutation, % | 539 (23.4) | 166 (31.4) | 62 (27.1) | 99 (42.7) | 19 (15.8) | 58 (11.3) | 135 (19.9) |
| MLH1, % | 237/539 (44) | 71/166 (42) | 30/62 (48) | 44/99 (44) | 3/19 (16) | 19/58 (33) | 70/135 (52) |
| MSH2, % | 251/539 (47) | 84/166 (51) | 26/62 (42) | 47/99 (47) | 15/19 (79) | 21/58 (36) | 58/135 (43) |
| MSH6, % | 51/539 (9) | 11/166 (7) | 6/62 (10) | 8/99 (9) | 1/19 (5) | 18/58 (31) | 7/135 (5) |
* CCFR = Colon Cancer Family Registries; DFCI = Dana Farber Cancer Institute; IQR = interquartile range; OSU = Ohio State University.
Table 2.
Characteristics of participants in five population-based cohorts of probands assessed for Lynch Syndrome*
| Characteristics | Total (n = 3451) |
CCFR (n = 1196) |
Newfoundland, Canada (n = 731) |
OSU (n = 191) |
Rotterdam, Netherlands (n = 196) |
Spanish Consortium (n = 1137) |
|---|---|---|---|---|---|---|
| Male, %, No. (%) | 1905 (55.2) | 581 (48.6) | 443 (60.6) | 89 (46.6) | 117 (59.7) | 675 (59.4) |
| Median age of CRC diagnosis, y (IQR) | 64 (54–72) | 58 (48–67) | 64 (56–71) | 63 (51–72) | 59 (52–64) | 71 (63–78) |
| Mutation carriers, No. (%) | ||||||
| Any mutation | 150 (4.4) | 78 (6.5) | 14 (1.9) | 30 (2.5) | 18 (9.2) | 10 (0.9) |
| MLH1, % | 47/150 (31.3) | 30/78 | 2/14 (14.3) | 8/30 (26.7) | 4/18 (22.2) | 3/10 (30) |
| MSH2, % | 71/150 (47.3) | 36/78 | 10/14 (71.4) | 15/30 (50) | 4/18 (22.2) | 6/10 (60) |
| MSH6, % | 32/150 (21.3) | 12/78 | 2/14 (14.3) | 7/30 (23.3) | 10/18 (55.6) | 1/10 (10) |
* CCFR = Colon Cancer Family Registries; IQR = interquartile range; OSU = Ohio State University.
Prediction of Any MMR Gene Mutation
Clinic-Based Cohorts
The AUCs for any MMR gene mutation for the pooled data were 0.76 (95% confidence interval [CI] = 0.74 to 0.79), 0.82 (95% CI = 0.80 to 0.84), and 0.85 (95% CI = 0.83 to 0.87) for MMRPredict, MMRPro, and PREMM1,2,6, respectively (Figure 1A and Table 3). The O/E ratio of predicted risk for PREMM1,2,6 was 1.0 (95% CI = 0.89 to 1.2) compared with 0.38 (95% CI = 0.32 to 0.45) for MMRPredict and 0.62 (95% CI = 0.53 to 0.74) for MMRPro (Table 3). The calibration slope for PREMM1,2,6 was 0.81 vs 0.42 and 0.28 for MMRPro and MMRPredict respectively (Figure 2, A-C). Using Figure 2A as an example, MMRPredict overpredicted mutation carrier status at predictions under 15% and more so at predictions under 5%. Conversely, the model underpredicted at predictions higher than 15%. In Figure 2B, MMRPro overpredicted at predictions under 15% and deviated consistently from perfect agreement between 0% and 15%. The model also underpredicted carrier status, at a risk of higher than 20%. MMRPro and PREMM1,2,6 identified a similar percentage of carriers (sensitivity 95% and 96%, respectively) at a threshold of 5% or higher (Table 5).
Figure 1.
Receiver operating characteristic curves for MMRPredict, MMRPro, and PREMM1,2,6 models. A) Receiver operating characteristic curves for discriminating mismatch repair mutation carriers from noncarriers with MMRPredict, MMRPro, and PREMM1,2,6 in clinic-based cohorts. B) Receiver operating characteristic curves for discriminating mismatch repair mutation carriers from noncarriers with MMRPredict, MMRPro, and PREMM1,2,6 in population-based cohorts. AUC = area under the curve.
Table 3.
Pooled performance characteristics of MMRPro, PREMM1,2,6, and MMRPredict for prediction of MMR gene mutations associated with colorectal cancer cases in international clinic-based cohorts*
| Characteristics | MMRPro† | PREMM1,2,6‡ | MMRPredict§,‖ |
|---|---|---|---|
| Discrimination AUC (95% CI) | |||
| Any mutation¶ | 0.82 (0.80 to 0.84) | 0.85 (0.83 to 0.87) | 0.76 (0.74 to 0.79) |
| MLH1 | 0.87 (0.84 to 0.89) | 0.88 (0.86 to 0.90) | |
| MSH2 | 0.83 (0.80 to 0.86) | 0.87 (0.84 to 0.89) | |
| MSH6 | 0.57 (0.49 to 0.65) | 0.67 (0.60 to 0.75) | |
| Calibration | |||
| O/E ratio (95% CI) | |||
| Any mutation¶ | 0.62 (0.53 to 0.74) | 1.0 (0.89 to 1.2) | 0.38 (0.32 to 0.45) |
| MLH1 | 0.49 (0.39 to 0.62) | 0.97 (0.82 to 1.2) | |
| MSH2 | 0.51 (0.41 to 0.64) | 0.93 (0.78 to 1.1) | |
| MSH6 | 0.26 (0.18 to 0.37) | 1.1 (0.84 to 1.5) | |
| Slope (95% CI) | |||
| Any mutation¶ | 0.42 (0.38 to 0.45) | 0.81 (0.73 to 0.88) | 0.28 (0.25 to 0.32) |
| MLH1 | 0.47 (0.42 to 0.53) | 0.81 (0.72 to 0.90) | |
| MSH2 | 0.42 (0.37 to 0.47) | 0.77 (0.68 to 0.86) | |
| MSH6 | 0.09 (-0.01 to 0.20) | 0.69 (0.44 to 0.93) | |
* n = 2304. AUC = area under the receiver operating characteristic curve; CI = confidence interval; O/E = observed/expected.
† Two-sided test of statistical interaction between cohorts and predicted probabilities: P < .001. An interaction term with P < .05 indicated that the relation between predicted probabilities and mutation status varies by cohort.
‡ Two-sided test of statistical interaction between cohorts and predicted probabilities: P = .03. An interaction term with P < .05 indicated that the relation between predicted probabilities and mutation status varies by cohort.
§ Two-sided test of statistical interaction between cohorts and predicted probabilities: P < .001. An interaction term with P < .05 indicated that the relation between predicted probabilities and mutation status varies by cohort.
‖ MMRPredict does not generate gene-specific probabilities.
¶ MLH1, MSH2, or MSH6 mutation.
Figure 2.
Calibration plots for MMRPredict, MMRPro, and PREMM1,2,6 models. A-C) Clinic-based cohort: (A-C) display calibration plots for external validation of (A) MMRpredict, (B) MMRPro, and (C) PREMM1,2,6 for predicting MMR mutations for individuals in clinic-based settings. D-F) Population-based cohort: (D-F) display calibration plots for external validation of (A) MMRpredict, (B) MMRPro, and (C) PREMM1,2,6 for predicting MMR mutations for individuals in population-based settings. The x-axis represents predicted probabilities, the y-axis represents the observed proportion of MMR mutations, and the long dashed diagonal line represents the ideal model with perfect prediction. The short dashed line represents the relation between MMR mutations and model-based predictions (according to a loess smoother). The triangles represent observed frequencies by quintiles of predicted probability with corresponding 95% confidence limits (vertical lines). The distribution of predicted probabilities is displayed for individuals with and without a mutation in the lower portion of the figure. ROC = receiver operating characteristic curve.
Table 5.
Proportion of patients identified as gene mutation carriers by MMRPro and PREMM1,2,6 at different decision thresholds
| Model and risk score category | High-risk patients* No. (%) |
Identified gene mutation carriers (true positive) No. (%) |
Predicted but not actual carriers (false positive) No. (%) |
Missed gene mutation carriers (false negative) No. (%) |
|---|---|---|---|---|
| Clinic-based cohorts | ||||
| PREMM1,2,6, % | ||||
| >0 | 2294/2294 (100) | 536/536 (100) | 1758/1758 (100) | 0/536 (0) |
| >5 | 1754//2294 (76) | 516/536 (96) | 1238/1758 (70) | 20/536 (4) |
| >10 | 1156/2294 (50) | 467/536 (87) | 689/1758 (39) | 69/536 (13) |
| >20 | 744/2294 (32) | 403/536 (75) | 341/1758 (19) | 133/536 (25) |
| MMRPro, % | ||||
| >0 | 2304/2304 (100) | 539/539 (100) | 1765/1765 (100) | 0/539 (0) |
| >5 | 1595/2304 (69) | 510/539 (95) | 1085/1765 (61) | 29/539 (5) |
| >10 | 1338/2304 (58) | 482/539 (89) | 856/1765 (48) | 57/539 (11) |
| >20 | 990/2304 (43) | 422/539 (78) | 568//1765 (32) | 117/539 (22) |
| Population-based cohort | ||||
| PREMM1,2,6, % | ||||
| >0 | 3451/3451 (100) | 150/150 (100) | 3301/3301 (100) | 0/150 (0) |
| >5 | 887/3451 (26) | 130/150 (87) | 757/3301 (23) | 20/150 (13) |
| >10 | 375/3451 (11) | 102/150 (68) | 273/3301 (8) | 48/150 (32) |
| >20 | 179/3451 (5) | 81/150 (54) | 98/3301 (3) | 69/150 (46) |
| MMRPro, % | ||||
| >0 | 2314/2314 (100) | 140/140 (100) | 2174/2174 (100) | 0/140 (0) |
| >5 | 906/2314 (39) | 119/140 (79) | 787/2174 (36) | 21/140 (15) |
| >10 | 730/2314 (32) | 111/140 (74) | 619/2174 (28) | 29/140 (21) |
| >20 | 532/2314 (23) | 102/140 (68) | 430/2174 (20) | 38/140 (27) |
* High-risk = number of patients within each designated risk score category.
Population-Based Cohorts
The pooled AUCs for any MMR gene mutation prediction were slightly higher for population-based than clinic-based cohorts (0.77, 0.85, 0.88 for MMRPredict, MMRPro, and PREMM1,2,6, respectively) (Figure 1B and Table 4). Predictions were too high for all models, with O/E ratios of 0.70 (95% CI = 0.58 to 0.84) for PREMM1,2,6, 0.36 (95% CI = 0.27 to 0.47) for MMRPro, and 0.31 (95% CI = 0.24 to 0.39) for MMRPredict. Predictions were too extreme for MMRPro (slope = 0.43, 95% CI = 0.37 to 0.48) and for MMRPredict (slope = 0.37, 95% CI = 0.31 to 0.42) (Figure 2, D-F, and Table 4). MMRPro and PREMM1,2,6 identified a similar proportion of high-risk patients who were mutation carriers at a threshold of 5% (13% vs 15% respectively) (Table 5). Overall, MMRPro had fewer observations than PREMM1,2,6 because data needed to generate an MMRPro score, such as information on all relatives affected and unaffected by cancer, including ages at last follow-up, were not available for all patients. All analyses were also performed by site and confirmed the patterns noted for the pooled datasets (Supplementary Table 1, A and B, available online).
Table 4.
Pooled performance characteristics of MMRPro, PREMM1,2,6, and MMRPredict for prediction of MMR gene mutations associated with colorectal cancer cases in international population-based cohorts*
| Characteristics | MMRPro† | PREMM1,2,6‡ | MMRPredict§,‖ |
|---|---|---|---|
| Discrimination | |||
| AUC (95% CI) | |||
| Any mutation¶ | 0.85 (0.81 to 0.88) | 0.88 (0.85 to 0.92) | 0.77 (0.73 to 0.82) |
| MLH1 | 0.84 (0.77 to 0.91) | 0.90 (0.85 to 0.96) | |
| MSH2 | 0.92 (0.90 to 0.95) | 0.92 (0.88 to 0.96) | |
| MSH6 | 0.68 (0.61 to 0.76) | 0.75 (0.66 to 0.84) | |
| Calibration | |||
| O/E ratio (95% CI) | |||
| Any mutation¶ | 0.36 (0.27 to 0.47) | 0.70 (0.58 to 0.84) | 0.31 (0.24 to 0.39) |
| MLH1 | 0.21 (0.13 to 0.32) | 0.61 (0.44 to 0.85) | |
| MSH2 | 0.27 (0.18 to 0.39) | 0.71 (0.54 to 0.94) | |
| MSH6 | 0.28 (0.18 to 0.45) | 0.70 (0.49 to 1.00) | |
| Slope (95% CI) | |||
| Any mutation¶ | 0.43(0.37 to 0.48) | 1.07 (0.94 to 1.19) | 0.37 (0.31 to 0.42) |
| MLH1 | 0.42 (0.34 to 0.51) | 0.93 (0.77 to 1.09) | |
| MSH2 | 0.52 (0.44 to 0.61) | 1.03 (0.88 to 1.18) | |
| MSH6 | 0.21 (0.10 to 0.32) | 1.12 (0.82 to 1.43) | |
* n = 3451. AUC = area under the receiver operating characteristic curve; CI = confidence interval; O/E = observed/expected.
† Two-sided test of heterogeneity between cohorts: P = .47; MMRPro results do not include data from the Spanish cohort.
‡ Two-sided test of heterogeneity between cohorts: P < .001.
§ Two-sided test of heterogeneity between cohorts: P < .001.
‖ MMRPredict does not generate gene-specific probabilities.
¶ MLH1, MSH2, or MSH6 mutation.
Gene-Specific Predictions
The PREMM1,2,6 and MMRPro models provide gene-specific predictions for MLH1, MSH2, and MSH6. In both clinic- and population-based cohorts, these models performed similarly in discrimination of MLH1 and MSH2 mutations (Tables 3 and 4). However, discrimination of MSH6 mutations from no mutation carriers was more difficult. Gene-specific O/E ratios and the calibration slopes were better for PREMM1,2,6 than MMRPro for each gene in both types of cohorts (Table 3 and 4). Specific analyses confirmed these patterns (Supplementary Materials, available online).
Clinical Usefulness
In clinic-based cohorts, the decision to recommend genetic testing based on PREMM1,2,6 or MMRPro estimates at any threshold 5% or higher provided a higher net benefit compared with MMRPredict (Figure 3A). There was no net benefit in using MMRPredict to select patients for testing at a threshold up to 10% compared with a strategy of testing all patients. The PREMM1,2,6 model provided the highest net benefit at thresholds of 15% or higher where more true-positive carriers were identified compared with MMRPro.
Figure 3.
Net benefit analyses comparing MMRPredict, MMRPro, and PREMM1,2,6 to identify mismatch repair mutation carriers at different risk thresholds. A and B) Display of the net benefit curves comparing the three prediction models among the clinic-based cohort. The y-axis measures net benefit, which is calculated by summing the benefits (true positives) and subtracting the harms (false positives), where the latter are weighted by a factor related to the relative harm of a missed mutation carrier compared with the harm of unnecessary genetic testing. A model is considered of clinical value if it has the highest net benefit compared with other models and simple strategies such as performing genetic testing in all patients (dashed black line) or no patients (horizontal black line) across the full range of threshold probabilities at which a patient would undergo genetic testing. For example, the net benefit of using PREMM1,2,6 or MMRPro to selectively test for mutation carriers exceeds that of testing all at a risk of 5% or higher. A) The net benefit at the 10% threshold is 0.18 for PREMM1,2,6 vs 0.15 for testing all and 0 for testing none among clinic-based cases. The net benefit of the PREMM1,2,6 model over testing all is thus 0.03, which means that three individuals would be identified as mutation carriers for every 100 people assessed with PREMM1,2,6 without an increase in the number of false-positive results. At the 10% threshold, this calculation assumes that we value a true-positive classification worth incurring up to nine false-positives (since 1:9, which is the odds corresponding to a probability of 10%). B) Display of the net benefit curves for the three models among the population-based cohort. The benefit at the 10% threshold is 0.02 for PREMM1,2,6 vs 0 for testing all and 0 for testing none. This means that two individuals would be identified as mutation carriers for every 100 people assessed with PREMM1,2,6 without an increase in the number of false positives.
In population-based cohorts, using any of the models to determine who should undergo genetic testing was superior to testing all patients for thresholds 5% and higher. More carriers were identified with PREMM1,2,6 than MMRPro and MMRPredict at higher thresholds (Figure 3B). Both in clinical and population-based cohorts, thresholds under 5% would exclude few patients from genetic testing, leading to no clinical usefulness beyond that of testing all patients.
The net benefit for PREMM1,2,6 was higher for gene-specific testing for MLH1 and MSH2 compared with MMRPro for both clinic- and population-based cohorts. The net benefit for MSH6 gene testing was limited to risk thresholds from 5% to 15% for PREMM1,2,6 in clinic-based cohorts (Supplementary Figures 1 and 2, available online).
Discussion
Both MMRPro and PREMM1,2,6 better discriminated MMR gene mutation carriers from noncarriers than MMRPredict in this large dataset of international cohorts of individuals diagnosed with CRC in both clinic- and population-based settings. These models were clinically useful at a 5% or higher risk threshold as recommended in recent guidelines for consideration of predictive genetic testing (11,12).
We assessed expected clinical usefulness through decision curve analyses because this approach offers important information beyond the standard performance metrics of discrimination and calibration (21–26). This methodology allowed us to estimate the net number of carriers identified by each model over different risk thresholds to select cases for further testing, penalizing for the number of patients having unnecessary testing. The numbers of identified carriers and those having unnecessary testing are also used in sensitivity and specificity calculations and are appropriately summarized in the net benefit. (23–26).
We considered thresholds between 5% and 20% as clinically plausible. With thresholds of 5% or greater, MMRPro and PREMM1,2,6 are clinically useful in clinic-based cohorts. PREMM1,2,6 also had an appreciable net benefit in the population-based cohorts, despite being originally developed using clinic-based patients. This is explained by the better calibration of PREMM1,2,6 than MMRPro or MMRPredict. While all models overestimated the probability of being a carrier among population-based cases, they most often deviated in predictions under 5%, where the predicted number of carriers far exceeded those observed. While this affects overall calibration, it has limited clinical significance because germline testing has not been recommended in patients with predicted probabilities under 5%. However, consideration can be given to a lower threshold in the future if costs of mutation analysis decrease or if multigene panel testing based on next-generation DNA sequencing becomes incorporated into clinical practice as the standard of care. In this study, there was no net benefit of any of the models at relatively low risk thresholds (<5%).
The geographic diversity of the cohorts provides a more comprehensive assessment of external validity than previous analyses (18,19,27–33), in addition to a recent meta-analysis of studies that have validated the models (34). The clinic-based sample included individuals evaluated at cancer genetics clinics where personal and family histories of cancers were well characterized. We further addressed the potential utility of the models in general medical settings by the inclusion of population-based series. Our results also provide new information about the ability of MMRPro and PREMM1,2,6 to predict gene-specific risk estimates. Both models performed equally well in identifying MLH1 and MSH2 gene mutation carriers but had low overall discrimination for MSH6 gene mutations. MSH6 gene mutation carriers may be challenging to identify, as CRC diagnoses occur usually at older ages than in MLH1 and MSH2 carriers and cases may appear as “sporadic CRC.” Endometrial cancer may present as the sentinel malignancy in female MSH6 carriers and at younger ages than CRC.
Several limitations of our study should be considered. The mechanisms of case identification at each site may have contributed to site-specific variation in model performance, although the overall patterns of the validity of the prediction models were consistent (Supplementary Materials, available online). Differences in the prevalence of carriers between sites could be attributable to heterogeneity in assessments between sites (ie, referral filter) (35). Another possible limitation is that some sites screened individuals for MMR deficiency based on tumor testing results and did not pursue germline testing when tumor testing was normal. This partial verification bias may misclassify some individuals as noncarriers (36) and may be more relevant for individuals with MSH6 mutations, whose tumors are not always microsatellite unstable and where certain pathogenic missense mutations do not completely abrogate protein expression yielding false-negative IHC results. Lastly, the current models do not predict PMS2 and EPCAM mutations. Because many sites did not include these mutational analyses, the models’ performance for these genes could not be assessed.
The results of our study have several implications for individuals with CRC. Assessment of family history of cancer by the Amsterdam criteria or Bethesda Guidelines (7,8) has been the cornerstone for the diagnosis of Lynch Syndrome, but multiple studies have demonstrated their limited sensitivity and specificity (37,38). The performance of PREMM1,2,6 and MMRPro exceeds that of existing clinical criteria to identify mutation carriers (13,14,19,28,29,33), and prediction models should replace clinical criteria as prescreening tools in the risk assessment process for Lynch Syndrome (19,27,32). For such an application, the PREMM1,2,6 model has the advantage of being simpler to apply than MMRPro (27) while its performance is at least as good. It does not require information on unaffected family members, and it has a simple, web-based platform (27). Another approach is systematic tumor testing for MMR deficiency through MSI or IHC for all newly diagnosed individuals with CRC (1,12,37). This approach may be feasible for some centers and superior to any prediction model if we accept a high rate of unnecessary testing. Unnecessary testing may be a consequence of false-positive tumor results because of somatic causes of MSI via MLH1 gene promoter hypermethylation rather than germline MMR deficiency, particularly in older patients. A combined approach may be attractive (39), such as using IHC and prediction model risk estimates in those with CRC older than 70 years (16). Model predictions can also complement results from tumor testing becasue false-negatives are possible with IHC testing (16). In light of these considerations, the PREMM1,2,6 or MMRPro models can be recommended to direct germline testing or genetic referral when risk exceeds 5% if tumor testing is unavailable or when resources are limited and the universal tumor testing approach cannot be adopted. However, there is a growing trend toward a very broad and routine molecular characterization of CRC tumors, if only to identify mutations that are targetable with newer chemotherapeutic agents. Because MSI testing is invariably included in such evaluations, we may anticipate the routine availability of these results in the very near future, which can potentially guide the assessment for Lynch Syndrome. The MMRPro model can readily incorporate molecular tumor testing results in the risk prediction. Lastly, individuals with high risk, ie, 15% or higher, may need to be considered for inherited cancer syndromes other than Lynch Syndrome. Screening patients with CRC for Lynch Syndrome based on molecular tumor testing alone may miss the opportunity to identify other familial cancer syndromes. For patients without a germline MMR mutation but with high predictions, intensive surveillance may be considered and additional genetic testing may be warranted in the future as novel genes associated with familial CRC are discovered.
In summary, the MMRPro and PREMM1,2,6 models are clinically useful tools to assess patients who are newly diagnosed with CRC for Lynch Syndrome. A threshold of 15% in the absence of MMR deficiency may identify individuals at high risk for other familial CRC syndromes and prompt further genetic evaluation and testing. These patients may need modified cancer surveillance tailored to their specific cancer spectrum while additional genetic etiologies for the cause of cancer in their families are investigated.
Funding
This work was supported by the National Cancer Institute (K07 CA151769-02 to FK; R01 CA132829 and K24 CA113433 to SS) and through cooperative agreements with members of the Colon Cancer Family Registry and PIs including Australasian Colorectal Cancer Family Registry (U01 CA097735), the University of Southern California (USC) Colorectal Cancer Family Registry (U01 CA074799), the Mayo Clinic Cooperative Family Registry for Colon Cancer Studies (U01 CA074800), the Ontario Registry for Studies of Familial Colorectal Cancer (U01 CA074783), the Seattle Colorectal Cancer Family Registry (U01 CA074794), and the University of Hawaii Colorectal Cancer Family Registry (U01 CA074806). The content of this manuscript does not necessarily reflect the views or policies of the National Cancer Institute or any of the collaborating centers in the Colon Cancer Family Registry (CCFR), nor does mention of trade names, commercial products, or organizations imply endorsement by the US Government or the CCFR.
Supplementary Material
Design and conduct of the study (FK, RO, CA, RM, SS, ES); collection, management, analysis, and interpretation of the data (all authors); preparation, review, or approval of the manuscript (all authors). Full access to all of the data in the study and responsibility for the integrity of the data and the accuracy of the data analysis (RO, FK).
There are no potential conflicts of interest, including relevant financial interests, activities, relationships, and affiliations related to this study, for any of the authors.
We would like to acknowledge the following Colon Cancer Family Registries that provided data for the analysis: Australasian Colorectal Cancer Family Registry, PI: Jenkins M; the Stanford Consortium Colorectal Cancer Family Registry, PI: Haile R; Mayo Clinic Cooperative Family Registry for Colon Cancer Studies, PI: Lindor NM; Ontario Registry for Studies of Familial Colorectal Cancer, PI: Gallinger S; Seattle Colorectal Cancer Family Registry, PI: Newcomb PA; and University of Hawaii Colorectal Cancer Family Registry, PI: LeMarchand L.
We would also like to acknowledge members of the Hereditary Cancer Group of the Spanish Medical Oncology Society (SEOM) and the EpiCOLON Consortium (Gastrointestinal Oncology Group of the Spanish Gastroenterological Association) for contributing data for the analyses.
The following contributors provided data for the analyses:
The Lynch Syndrome Identification through Molecular analysis (LIMO) investigators from Erasmus MC, the Netherlands: van Lier MG, Wagner A, Dubbink HJ, Dinjens WN, Ramsoekh D, van Leerdam ME, Kuipers EJ.
Members of the Hereditary Cancer Group of the Spanish Medical Oncology Society (SEOM): Blanco I, Torres A, Ramón y Cajal T, Chirivella I, Guillén C, Brunet J, Martínez-Bouzas C, Alonso A, Pérez Segura P, Robles L, Graña B, Urioste M, Lastra E; and the EpiCOLON Consortium (Gastrointestinal Oncology Group of the Spanish Gastroenterological Association): National coordinators: Andreu M, Carracedo A, Castells A, Jover R, and Llor X; Members: Hospital Clínic, Barcelona: Castells A (local coordinator), Piñol V, Castellví-Bel S, Gonzalo V, Ocaña T, Giráldez MD, Pellisé M, Serradesanferm A, Moreira L, Cuatrecasas M, Piqué JM; Hospital 12 de Octubre, Madrid: Morillas JD (local coordinator), Muñoz R, Manzano M, Colina F, Díaz J, Ibarrola C, López G, Ibáñez A; Hospital Clínico Universitario, Zaragoza: Lanas A (local coordinator), Alcedo J, Ortego J; Complexo Hospitalario de Ourense: Cubiella J (local coordinator), Mª Díez A, Salgado M, Sánchez E, Vega M; Hospital del Mar, Barcelona: Andreu M (local coordinator), Abuli A, Bessa X, Iglesias M, Seoane A, Bory F, Navarro G, Bellosillo B; Dedeu J, Álvarez C, Gonzalez B; Hospital San Eloy, Baracaldo and Hospital Donostia, San Sebastián: Bujanda L(local coordinator) Cosme A, Gil I, Larzabal M, Placer C, del Mar Ramírez M, Hijona E, Enríquez-Navascués JM, Elosegui JL; Hospital General Universitario de Alicante: Payá A (local coordinator), Jover R (local coordinator), Alenda C, Sempere L, Acame N, Rojas E, Pérez-Carbonell L; Hospital General de Granollers: Rigau J (local coordinator), Serrano A, Giménez A; Hospital General de Vic: Saló J (local coordinator), Batiste-Alentorn E, Autonell J, Barniol R; Hospital General Universitario de Guadalajara and Fundación para la Formación e Investigación Sanitarias Murcia: García AM (local coordinator), Carballo F, Bienvenido A, Sanz E, González F, Sánchez J, Ono A; Hospital General Universitario de Valencia: Latorre M (local coordinator), Medina E, Cuquerella J, Canelles P, Martorell M, García JA, Quiles F, Orti E; Hospital Meixoeiro, Vigo: Clofent J (local coordinator), Mª de Castro L (local coordinator), Seoane J, Tardío A, Sanchez E, Hernández V; Hospital Universitari Germans Trias i Pujol, Badalona and University of Illinois at Chicago, IL: Llor X (local coordinator), Xicola RM, Piñol M, Rosinach M, Roca A, Pons E, Hernández JM, Gassull MA; Hospital Universitari Mútua de Terrassa: Fernández-Bañares F (local coordinator), Viver JM, Salas A, Espinós J, Forné M, Esteve M; Hospital Universitari Arnau de Vilanova, Lleida: Reñé JM (local coordinator), Piñol C, Buenestado J, Viñas J; Hospital Universitario de Canarias, Tenerife: Quintero E (local coordinator), Nicolás D, Parra A, Martín A; Hospital Universitario La Fe, Valencia: Argüello L (local coordinator), Pons V, Pertejo V, Sala T; Hospital de Sant Pau, Barcelona: Gonzalez D (local coordinator) Roman E, Ramon T, Poca M, Mª Concepción M, Martin M, Pétriz L; Hospital Xeral Cies, Vigo: Martinez D (local coordinator); Fundacion Publica Galega de Medicina Xenomica, Santiago de Compostela: Carracedo A (local coordinator), Ruiz-Ponte C, Fernández-Rozadilla C, Mª Castro M; Hospital Universitario Central de Asturias, Oviedo: Riestra S (local coordinator), Rodrigo L; Hospital de Galdácano, Vizcaya: Fernández J (local coordinator), Cabriada JL; Fundación Hospital de Calahorra: Carreño L (local coordinator), Oquiñena S, Bolado F; Hospital Royo Villanova, Zaragoza: Peña E (local coordinator), Blas JM, Ceña G, Sebastián JJ; Hospital Universitario Reina Sofía, Córdoba: Naranjo A (local coordinator).
References
- 1. Hampel H, Frankel WL, Martin E, et al. Screening for the Lynch Syndrome (hereditary nonpolyposis colorectal cancer). N Engl J Med. 2005;352 (18):1851–1860. [DOI] [PubMed] [Google Scholar]
- 2. Jasperson KW, Tuohy TT, Neklason DW, Burt RW. Hereditary and Familial Colon Cancer. Gastroenterology. 2010;138 (6):2044–2058. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Stoffel E, Mukherjee B, Raymond VM, et al. Calculation of risk of colorectal and endometrial cancer among patients with Lynch Syndrome. Gastroenterology. 2009:137 (5):1621–1627. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Bonadona V, Bonaïti B, Olschwang S, et al. Cancer Risks Associated with Germline Mutations in MLH1, MSH2, and MSH6 Genes in Lynch Syndrome. JAMA. 2011;305 (22):2304–2310. [DOI] [PubMed] [Google Scholar]
- 5. Lindor NM, Petersen GM, Hadley DW, et al. Recommendations for the Care of Individuals with an Inherited Predisposition to Lynch Syndrome: A Systematic Review. JAMA. 2006;296 (12):1507–1517. [DOI] [PubMed] [Google Scholar]
- 6. Vasen HF, Mecklin JP, Khan PM, Lynch HT. The international collaborative group on hereditary non-polyposis colorectal cancer (ICG-HNPCC). Dis Colon Rectum. 1991;34 (5):424–425. [DOI] [PubMed] [Google Scholar]
- 7. Rodriguez-Bigas MA, Boland CR, Hamilton SR, et al. National Cancer Institute workshop on hereditary nonpolyposis colorectal cancer syndrome: Meeting highlights and Bethesda guidelines. J Natl Cancer Inst. 1997;89 (23):1758–1762. [DOI] [PubMed] [Google Scholar]
- 8. Umar A, Boland CR, Terdiman JP, et al. Revised Bethesda Guidelines for hereditary nonpolyposis colorectal cancer (Lynch Syndrome) and microsatellite instability. J Natl Cancer Inst. 2004;96 (4):261–268. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Palomaki GE, McClain MR, Melillo S, Hampel HL, Thibodeau SN. EGAPP supplementary evidence review: DNA testing strategies aimed at reducing morbidity and mortality from Lynch Syndrome. Genet Med. 2009;11 (1):42–65. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. van Lier MG, Leenen CH, Wagner A, et al. Yield of routine molecular analyses in colorectal cancer patients ≤70 years to detect underlying Lynch Syndrome. J Pathol. 2012;226 (5):764–774. [DOI] [PubMed] [Google Scholar]
- 11. National Comprehensive Cancer Network (NCCN) Clinical Practice Guidelines in Oncology. Guidelines for detection, prevention, and risk reduction. Genetic/Familial High-Risk Assessment: Colorectal Cancer. http://www.nccn.org/professionals/physician_gls/pdf/genetics_colon.pdf. Version 2.2014. Accessed June 17, 2014.
- 12. Giardiello FM, Allen JI, Axilbund JE, et al. Guidelines on genetic evaluation and management of Lynch Syndrome: a consensus statement by the U.S. Multi-Society Task Force on Colorectal Cancer. Gastroenterology. 2014;147 (2):502–526. [DOI] [PubMed] [Google Scholar]
- 13. Chen S, Wang W, Lee S, et al. Prediction of germline mutations and cancer risk in the Lynch Syndrome. JAMA. 2006;296 (12):1479–1487. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Barnetson RA, Tenesa A, Farrington SM, et al. Identification and survival of carriers of mutations in DNA mismatch-repair genes in colon cancer. N Engl J Med. 2006;354 (26):2751–2763. [DOI] [PubMed] [Google Scholar]
- 15. Kastrinos F, Steyerberg EW, Mercado R, et al. The PREMM1,2,6 model predicts risk of MLH1, MSH2, and MSH6 germline mutations based on cancer history. Gastroenterology. 2011;140 (1):73–81. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Kastrinos F, Steyerberg EW, Balmaña J, et al. Comparison of the clinical prediction model PREMM1,2,6 and molecular testing for the systematic identification of Lynch Syndrome in colorectal cancer. Gut. 2013;62 (2):272–279. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Newcomb PA, Baron J, Cotterchio M, et al. Colon Cancer Family Registry: An international resource for studies of the genetic epidemiology of colon cancer. Cancer Epidemiol Biomarkers Prev. 2007;16 (11):2331–2343. [DOI] [PubMed] [Google Scholar]
- 18. Ramsoekh D, van Leerdam ME, Wagner A, Kuipers EJ, Steyerberg EW. Mutation prediction models in Lynch Syndrome: evaluation in a clinical genetic setting. J Med Genet. 2009;46 (11): 745–751. [DOI] [PubMed] [Google Scholar]
- 19. Green RC, Parfrey PS, Woods MO, Younghusband HB. Prediction of Lynch Syndrome in consecutive patients with colorectal cancer. J Natl Cancer Inst. 2009;101 (5):331–340. [DOI] [PubMed] [Google Scholar]
- 20. Piñol V, Castells A, Andreu M, et al. Accuracy of revised Bethesda guidelines, microsatellite instability, and immunohistochemistry for the identification of patients with hereditary nonpolyposis colorectal cancer. JAMA. 2005;293 (16):1986–1994. [DOI] [PubMed] [Google Scholar]
- 21. Austin PC, Steyerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using less smoothers. Stat Med. 2014;33 (15):517–535. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J. 2014;35 (29):1925–1931. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26 (6):565–574. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Vickers AJ, Cronin AM. Traditional statistical methods for evaluating prediction models are uninformative as to clinical value: towards a decision analytic framework. Semin Oncol. 2010;37 (1):31–38. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21 (1):128–138. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Localio AR, Goodman S. Beyond the usual prediction accuracy metrics: reporting results for clinical decision making. Ann Intern Med. 2012;157 (4):294–295. [DOI] [PubMed] [Google Scholar]
- 27. Khan O, Blanco A, Conrad P, et al. Performance of Lynch Syndrome predictive models in a multi-center US referral population. Am J Gastroenterol. 2011;106 (10):1822–1827. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. Balaguer F, Balmaña J, Castellví-Bel S, et al. Validation and extension of the PREMM1,2 model in a population-based cohort of colorectal cancer patients. Gastroenterology. 2008;134 (1):39–46. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Balmaña J, Balaguer F, Castellví-Bel S, et al. Comparison of predictive models, clinical criteria and molecular tumor screening for the identification of patients with Lynch Syndrome in a population-based cohort of colorectal cancer patients. J Med Genet. 2008;45 (9):557–563. [DOI] [PubMed] [Google Scholar]
- 30. Pouchet CJ, Wong N, Chong G, et al. A comparison of models used to predict MLH1, MSH2 and MSH6 mutation carriers. Ann Oncol. 2009;20 (4):681–688. [DOI] [PubMed] [Google Scholar]
- 31. Monzon JG, Cremin C, Armstrong L, et al. Validation of predictive models for germline mutations in DNA mismatch repair genes in colorectal cancer. Int J Cancer. 2010;126 (4):930–939. [DOI] [PubMed] [Google Scholar]
- 32. Tresallet C, Brouquet A, Julié C, et al. Evaluation of predictive models in daily practice for the identification of patients with Lynch Syndrome. Int J Cancer. 2012;130 (6):1367–1377. [DOI] [PubMed] [Google Scholar]
- 33. Balmaña J, Stockwell DH, Steyerberg EW, et al. Prediction of MLH1 and MSH2 mutations in Lynch Syndrome. JAMA. 2006;296 (12):1469–1478. [DOI] [PubMed] [Google Scholar]
- 34. Win AK, MacInnis R, Dowty JG, Jenkins MA. Criteria and prediction models for mismatch repair gene mutations: a review. J Med Genet. 2013;50 (12):785–793. [DOI] [PubMed] [Google Scholar]
- 35. Leeflang MM, Bosuyt PM, Irwig L. Diagnostic test accuracy may vary with prevalence: implications for evidence-based diagnosis. J Clin Epidemiol. 2009;62 (1):5–12. [DOI] [PubMed] [Google Scholar]
- 36. Whiting P, Rutjes AW, Reitsma JB, Glas AS, Bossuyt PM, Kleijnen J. Sources of variation and bias in studies of diagnostic accuracy: A systematic review. Ann Intern Med. 2004;140 (3):189–202. [DOI] [PubMed] [Google Scholar]
- 37. Hampel H. Point: Justification for Lynch Syndrome screening among all patients with newly diagnosed colorectal cancer. JNCCN. 2010;8 (5):597–601. [DOI] [PubMed] [Google Scholar]
- 38. Hampel H, Stephens JA, Pukkala E, et al. Cancer risk in hereditary nonpolyposis colorectal cancer syndrome: later age of onset. Gastroenterology. 2005;129 (2):415–421. [DOI] [PubMed] [Google Scholar]
- 39. Moreira L, Balaguer F, Lindor NM, et al. Identification of Lynch Syndrome among patients with colorectal cancer. JAMA. 2012;308 (15):1555–1565. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.