Preliminary validation of a consumer-oriented colorectal cancer risk assessment tool compatible with the U.S. Surgeon General’s My Family Health Portrait

Abstract

Purpose

This study examines the analytic validity of a software tool designed to provide individuals with risk assessments for colorectal cancer (CRC) based on personal health and family history information. The software is compatible with the U.S. Surgeon General’s My Family Health Portrait.

Methods

An algorithm for risk assessment was created using accepted colorectal risk assessment guidelines, and programmed into a software tool (MFHP). Risk assessments derived from 150 pedigrees using the MFHP tool were compared to “gold standard” risk assessments developed by three expert cancer genetic counselors (GCs).

Results

Genetic counselor risk assessments showed substantial, but not perfect, agreement. MFHP risk assessments for CRC cancer yielded a sensitivity for CRC risk of 81% (95% CI 54–96%) and specificity of 90% (95% CI 83–94%), respectively, when compared to GC pedigree review. The positive predictive values for risk for MFHP was 48% (95% CI 29–68%), while the negative predictive values was 98% (95% CI 93–99%). Agreement between MFHP and GC pedigree review was moderate (Kappa = 0.54)

Conclusions

The analytic validity of the MFHP CRC risk assessment software is similar to other types of screening tools used in primary care. Future investigations should explore the clinical validity and utility of the software in diverse population groups.

Introduction

Patient-completed electronic family history (FH) collection tools have been proposed as a way to overcome several of the barriers to collection and use of family history information in health care settings^1,2. Several patient completed tools designed to aid in the collection and interpretation of FH have been developed and validated, often in settings such as cancer and primary care clinics^3–5. A 2009 review of FH literature conducted for the U.S. Agency for Health Care Research and Quality (AHRQ) and the National Institutes of Health (NIH) identified a paucity of validated FH tools suitable for use in primary care or public health contexts⁶. In 2014 the National Colorectal Cancer Roundtable found that electronic health record systems (EHRs) are poorly prepared to store and interpret FH relevant to CRC risk assessment⁷.

The U. S. Centers for Disease Control and Prevention (CDC)-sponsored Family Healthware™ Trial investigated behavioral change related to the clinical use of a patient completed electronic family history tool (FHT) that provided risk assessment for several conditions including, type 2 diabetes, heart disease, stroke, as well as colorectal, breast, and ovarian cancer⁸. FHT was not designed for stand-alone CRC risk assessment, but extensive expertise was brought to bear in its development, suggesting that FHT may be a useful point of comparison for the evaluation of other automated CRC risk assessment tools. The public version of the My Family Health Portrait (MFHP) tool was developed and launched in 2004 as part of the U.S. Surgeon General’s Family History Initiative. The tool was designed to help the public collect, organize, and share family history information with relatives and health professionals⁹. The current version of MFHP was designed to collect FH in a format compatible with electronic health record systems using nomenclature and data standards for storing and sharing FH and has been studied in several settings and is freely available to the public^10,11,12,13.

An important limitation of the current public MFHP tool is that it does not provide users with feedback or education regarding their individual disease risk. Here we report on the development of a CRC risk assessment module compatible with the public MFHP tool, and, potentially EHRs. As well, we present data regarding the analytic validity of the new module and the FHT tool according to the Analytical validity, Clinical validity; Clinical utility; and Ethical, legal, and social issues (ACCE) framework proposed as a method for evaluating family health history and other genomic health tools¹⁴.

Materials and Methods

Creation of the CRC risk module

The MFHP tool output is a structured XML output file that is computable for risk assessment. An algorithm based on National Comprehensive Cancer Network^® (NCCN) and United States Preventive Services Task Force (USPSTF) CRC risk assessment guidelines was developed to dichotomize individuals into “elevated” and “not elevated” risk categories based on data entered into the MFHP tool (online appendix A). To avoid inducing potentially inappropriate distress in individuals without benefit of immediate access to evaluation by a health professional, “elevated” risk individuals were not further subdivided into “high” and “moderate” risk categories by the MFHP tool. The model did not incorporate other data elements such as proband gender, body mass index, or self-identified race, as there were no evidence-based guidelines for assigning risk for these factors in conjunction with family history data. The final algorithm was reviewed by a group of cancer risk assessment experts prior to being programmed into an open-source software tool. This input resulted in the final decision to provide a dichotomous rather than a more quantitative risk assessment, and to consider all reported polyps as potential sources of risk warranting further discussion with a health care provider. The output of the MFHP tool is risk specific CRC information printable as letters for public users and their health care providers.

Study population and risk algorithm validation

We used 150 patient-entered pedigrees sequentially derived from the ClinSeq^® cohort as the substrate for assessing the MFHP tool and CDC’s FHT CRC module’s analytic performance¹⁵. The cohort consists of self-selected volunteers with an interest in learning their personal genomic risks through genome or exome sequencing; individuals entered the study without investigator foreknowledge of cancer family histories. The cohort is more likely to be white, well educated, and affluent than the general U.S. population. Forty additional patient-entered pedigrees derived from the cohort served as internal controls; 20 were deliberately modified to increase colorectal cancer risk to different levels (10 “strong” risk, 10 “moderate” risk) by adding additional accepted risk factors to the pedigree data.

Three different methods were used to derive risk estimates from the test pedigrees: the MFHP tool (including the new CRC risk module, referred to as MFHP hereafter), the CDC’s FHT (1.0 Beta Version for Research Studies), and independent review by three GCs expert in colorectal cancer risk assessment. The GCs were blinded to which pedigrees had been modified to alter risk, the risk assessments made by the MFHP tool, FHT, each other’s risk assessments, and proband self-reported BMI and race. The GCs were instructed that they should categorize the proband’s CRC risk as “weak”, “moderate”, or “strong” according to the criteria outlined by Yoon et al⁸. This study was approved by the National Human Genome Research Institute Institutional Review Board.

Statistical analysis

Review of pedigrees by a panel of three expert cancer GCs was defined as the gold-standard for risk of CRC in the test pedigrees¹⁶. Risk estimates of “weak” by the FHT and GC review were considered “not elevated” while “moderate” and “high” risk assignments were considered to be “elevated”. Pedigrees were scored as showing “elevated” risk for CRC only when two of three counselors reported risk levels as “moderate” or “strong” (true positives). Kappa statistics were generated to assess GC inter-rater agreement for CRC assessment using the “weak”, “moderate” and “strong” risk estimates for all 190 pedigrees. For the calculation of sensitivity, specificity, positive predictive values (PPV), negative predictive values (NPV) and Kappa statistics for the presence of CRC risk for the MFHP tool and FHT versus the GC review only the 150 unmodified pedigrees were used.

Results

GC risk estimates for CRC in the ClinSeq^® pedigrees

Sixteen of 150 (11%) of the unmodified pedigrees were identified as having “elevated” risk by at least two of three GCs rating risk as “moderate” or “strong”; 6/150 (4%) were identified as having “strong” risk by at least two of three counselors. For the modified internal control pedigrees, the three counselors agreed for 10/10 (100%) and 6/10 (60%) pedigrees intended to be at “strong” or “moderate” risk, respectively. Pair-wise Kappa statistics for the three counselors calculated for all 190 pedigrees were 0.760, 0.776, and 0.851, indicating substantial agreement between the GCs according to the Landis-Koch guidelines¹⁷.

Comparison of the MFHP tool versus expert GC risk estimates

Twenty-seven of 150 (18%) of unmodified pedigrees were assigned “elevated” risk status by the MFHP tool. The MFHP tool had a sensitivity and specificity of 81% (95% CI 54–96%) and 90% (95% CI 83–94%), when compared to pedigrees judged to be “elevated” by GC review. The PPV and NPV were 48% (95% CI 29–68%) and 98%(95% CI 93–99%), respectively (Figure 1). All twenty of the modified pedigrees intended to be at “strong” or “moderate” risk were identified as at “elevated” risk by the MFHP tool. The Kappa statistic between the MFHP tool and GC review suggested moderate agreement (0.54).

Figure 1.

Diagram showing the process used for developing interpretations of 150 unmodified pedigrees derived from the ClinSeq^® population (see text) and resulting calculated values for sensitivity, specificity, positive predictive value and negative predictive value calculated Kappa for the My Family Health Portrait colorectal risk algorithm for detection of elevated colorectal cancer risk using expert genetic counselor pedigree interpretation as the “gold standard” for risk interpretation. (TP= true positives; TN= true negatives; PPV= positive predictive value; NPV= negative predictive value; CI= confidence interval)

Comparison of the FHT versus GC risk estimates

Fourteen of 150 (9%) of the unmodified pedigrees were considered to be at “elevated” risk by FHT; of these 14 pedigrees 3/150 (6%) were classified as being at “strong” risk and 11/150 (7%) were rated to be at “moderate” risk. The FHT had a sensitivity and specificity of 75% (95% CI 48–93%)and 99% (95% CI 95–100%) respectively, when compared to pedigrees judged to be “elevated” by GC review. The PPV and NPV for FHT were 86% (95% CI 57–98%) and 97% (95% CI 93–99%) respectively. The Kappa statistic between FHT and GC review indicated substantial agreement (0.78). For the modified control pedigrees there was 10/10 (100%) agreement between the FHT and pedigrees intended to be at “strong” risk. Eight of 10 (80%) of the modified pedigrees intended to be at “moderate” risk were identified as such by FHT, for 2/10 (20%) FHT rated the risk as “strong” instead of “moderate”.

Assessment of disagreement between GC risk estimates and MFHP

There were three unmodified pedigrees that were identified as at “elevated” risk by GC review that were rated “not elevated” risk by the MFHP tool (“false negatives”) and 14 pedigrees for which the MFHP tool rated risk as “elevated” and GC review rated risk as “not elevated” (“false positives”). The sources of the discrepancies were diverse (Table 1). The “false negative” pedigrees shared the characteristic of a lack of CRC cases or colorectal polyps in the proband or relatives. For one of the pedigrees the error source was a failure to enter endometrial cancer as a structured term due to misspelling. The “false positive” pedigrees often appeared to arise from differing assignment of risk arising from personal or family histories of colon polyps or a history of CRC or CRC-associated cancers in second-degree relatives.

Table 1.

Potential sources of heritable risk for colorectal cancer (CRC) present in pedigrees yielding “false negative” and “false positive” results with the My Family Health Portrait colorectal cancer risk assessment (MFHP) tool in comparison to a gold standard of pedigree review by three expert cancer genetic counselors (GC).

“False negative”	“False positive”
GC risk higher than MFHP	MFHP risk higher than GC
2° GasCA; 2° OvCA	2° CRC; 1° PanCA
1° BrCa, age 40–49; 2° PrCA	2° CRC, age 40–49
Patient data entry error	2° with both CRC, BrCA
1° EndoCA, age 30–39	2° CRC, age 50–59
	2° CRC; 2° CRC, both age unknown
	2° CRC; 2° CRC, both age unknown
	1° ColP, age 40–49
	1° GasCA
	2° with both CRC, BrCA
	2° CRC, age 40–49
	1° AdenoCA; 2° PanCA; 2° PanCA; 2° EndoCA
	1° ColP; 2° ColP; 2° CRC
	Proband ColP; 1° ColP
	1° PrCA, 2° GasCA, 2° EndoCA

Each cell represents a single pedigree, individuals are separated by a semi-colon. Unspecified age ranges for cancer are either unknown or were 60 or greater years of age.

1° – first degree relative; 2° – second degree relative; GasCA – gastric cancer; EndoCA – endometrial/uterine cancer; BrCA – breast cancer; PrCA – prostate cancer; PanCA – pancreatic cancer; ColP – colon polyp; AdenoCA – adenocarcinoma, site unspecified

Discussion

This study provides preliminary evidence supporting the analytic validity of an automated, consumer-oriented screening algorithm for heritable CRC risk assessment that is compatible with the U.S. Surgeon General’s MFHP. To our knowledge it is the first study to formally evaluate the predictive value of a family history risk algorithm designed specifically to work with the MFHP tool. It is important to note that the “gold standard” in this study is not development of disease; rather the risk prediction arrived at by a group of three expert cancer GCs. The PPV values for MFHP and FHT exceed the 10% threshold that is considered an acceptable value for PPV for a genomic screening test¹⁸. Because only a fraction of individuals with elevated CRC will actually develop disease, the PPV regarding development of clinical disease is likely to be lower than the values observed in this study, and the NPV higher, for both the MFHP and FHT tools.

The MFHP algorithm was not designed to be diagnostic of CRC syndromes, rather it was designed to be a potential component of an educational tool that helps individuals and their health care providers stratify risk for heritable CRC in order to have an informed conversation regarding further evaluation. As a screening tool intended for use by the general public, it was designed to have a higher sensitivity at the cost of specificity, based on the rationale that missing elevated risk of colon cancer incurs a greater penalty than encouraging additional patients to discuss their risk of colorectal cancer with a health care provider. This choice could be debated, as available evidence suggests family history is not a strong predictor of absolute risk for colorectal cancer in the population¹⁹.

Evaluation of the pedigrees for which the risk ratings were discordant between GC review and the MFHP tool suggests that the MFHP tool assigns “elevated” risk to pedigrees with second degree relatives having CRC, multiple Lynch syndrome-related cancers, and personal or family histories of colon polyps. The GCs might not attribute much risk to histories of colon polyps lacking details of pathology. The FHT also does not fully incorporate personal and family history of colon polyps when assigning risk. Assuming such polyps are benign may underestimate risk, and at least one counselor desired additional information²⁰. Two of three instances where the MFHP tool did not assign elevated risk represent somewhat grey areas of risk assessment for CRC. The lack of clarity in guidelines for such grey areas is reflected in the inconsistency in GC risk assignment; for 2 of 3 of the “false negative” pedigrees at least one expert GC rated risk as “weak” while another rated risk as “strong”.

This study has a number of important limitations. Measured internal agreement between the counselors suggests that expert GCs do not always concur when independently assigning CRC risk to pedigrees. The discrepancies between risk estimates provided by GC review and the MFHP algorithm highlight that currently there is no universally accepted “gold standard” for the detection of CRC risk in family histories in routine clinical care. Similar to 2005 study by Qureshi et al., this study uses review by expert GCs to arrive at a proxy gold standard for CRC risk.¹⁶ Additional limitations of this study include the small sample size, the nature of the ClinSeq^® population, which may not represent the full diversity of the US population, and that the study was only able to examine the analytic validity (and not the clinical utility) of the MFHP CRC algorithm. Future studies should seek to replicate these findings and define the clinical validity and clinical utility of the MFHP tool in additional patient populations.