Early Ascertainment of Breast Cancer Diagnoses Comparing Self-Reported Questionnaires and Electronic Health Record Data Warehouse: The WISDOM Study

Abstract

PURPOSE

The goal of this study was to use real-world data sources that may be faster and more complete than self-reported data alone, and timelier than cancer registries, to ascertain breast cancer cases in the ongoing screening trial, the WISDOM Study.

METHODS

We developed a data warehouse procedural process (DWPP) to identify breast cancer cases from a subgroup of WISDOM participants (n = 11,314) who received breast-related care from a University of California Health Center in the period 2012-2021 by searching electronic health records (EHRs) in the University of California Data Warehouse (UCDW). Incident breast cancer diagnoses identified by the DWPP were compared with those identified by self-report via annual follow-up online questionnaires.

RESULTS

Our study identified 172 participants with confirmed breast cancer diagnoses in the period 2016-2021 by the following sources: 129 (75%) by both self-report and DWPP, 23 (13%) by DWPP alone, and 20 (12%) by self-report only. Among those with International Classification of Diseases 10th revision cancer diagnostic codes, no diagnosis was confirmed in 18% of participants.

CONCLUSION

For diagnoses that occurred ≥20 months before the January 1, 2022, UCDW data pull, WISDOM self-reported data via annual questionnaire achieved high accuracy (96%), as confirmed by the cancer registry. More rapid cancer ascertainment can be achieved by combining self-reported data with EHR data from a health system data warehouse registry, particularly to address self-reported questionnaire issues such as timing delays (ie, time lag between participant diagnoses and the submission of their self-reported questionnaire typically ranges from a month to a year) and lack of response. Although cancer registry reporting often is not as timely, it does not require verification as does the DWPP or self-report from annual questionnaires.

INTRODUCTION

Obtaining reliable, valid cancer measures is essential to conducting meaningful, actionable cancer research. Currently, the most commonly used sources for cancer diagnosis data include cancer registries, chart review, electronic health record (EHR) data warehouses, and self-reported questionnaires.^1,2 Each source has advantages and limitations with respect to timing, accuracy, and geographic reach of cancer diagnosis ascertainment (Fig S1, Data Supplement).

Cancer registries are the best source for complete, accurate ascertainment of new cancer cases and long-term diagnostic trends. Cancer registry reporting is considered the gold standard for ascertaining cancer diagnoses¹; however, the delay between diagnosis and reporting of cases by the California Cancer Registry (CCR) can be up to 2 years.^3,4 This long latency period required for case confirmation and elimination of duplicates is a significant limitation to using the CCR for ascertaining cancers when the goal is to measure real-time incidence of cancer in an on-going study cohort. Therefore, access to lower-latency data sources is important in pragmatic studies that need to continuously monitor cancer incidence to determine safety, such as the WISDOM Study.

The WISDOM Study, an ongoing breast cancer screening clinical trial (ClinicalTrials.gov Identifier: NCT02620852), requires biannual reporting to a Data Safety Monitoring Board (DSMB) for the purpose of monitoring study safety.⁵ The frequency of advanced breast cancer is low and thus timely and accurate measurement of breast cancer incidence is critical to meeting trial objectives. WISDOM is pragmatic, using participants self-report of breast cancer diagnoses through online annual follow-up breast health questionnaires (BHQs).⁵

This manuscript describes the result of using real-world data to identify potentially missing cancer cases by leveraging EHR data warehouse information. To our knowledge, this is the first large-scale study comparing ascertainment of breast cancer diagnosis through a clinical data warehouse combined with self-reported online questionnaires from a cohort of women enrolled without a history of breast cancer.

METHODS

Study Design and Population

This study used the University of California Data Warehouse (UCDW), which includes information on diagnoses, encounters, medications, procedures, and laboratory test results from EHR records of six University of California (UC) health systems since 2012.⁶ The medical disease codes are International Classification of Diseases 10th revision Clinical Modification (ICD-10-CM), and the medical services and procedural codes are Current Procedural Terminology (CPT) codes. All diagnoses in the UCDW are mapped to ICD-10-CM codes. The UCDW includes individuals who received breast-related care at a UC health system during the study period.

As shown in Figure 1, during our study period (September 2016-December 2021), there were 47,501 WISDOM Study participants. Of those, 17,955 indicated on their baseline questionnaire that they have previously received care at a UC Health facility, and we searched the UCDW for those who had any ICD-10-CM or CPT code to confirm that there had been some level of clinical service at a UC Health facility. The deterministic matching process was based on UC medical record number, if known, first name, last name, and date of birth, which we obtained from the WISDOM data set. We identified 15,526 participants who were in the UCDW. Of this group, we then identified 12,948 individuals who had a UC breast-related care record (at least one ICD-10-CM or CPT code relevant to breast care or breast cancer—Fig S2, Data Supplement). Among them, 11,314 individuals had a WISDOM screening assignment, which indicates that participants received a study intervention and were considered full study participants.

FIG 1.

WISDOM cohort selection to identify participants with at least one code specific to breast care in the UCDW. ^aAs self-reported in the BHQ. ^bScreening assignment means they have not received a screening assignment letter, the key WISDOM study intervention. Reasons for not receiving a screening assignment include no mammogram on file, no BHQ, or no genetics testing results. BHQ, Breast Health Questionnaire; CPT, Current Procedural Terminology; ICD-10-CM, International Classification of Diseases 10th Revision Clinical Modification; UC, University of California; UCDW, University of California Data Warehouse; WISDOM, Women Informed to Screen Based On Measures of risk.

The WISDOM Study is approved by the University of California, San Francisco institutional review board, and all participants provide informed consent to participate in the study.

Measures

Cancer Ascertainment: Self-Report With Confirmation

The primary method of discovering incident breast cancers was through the WISDOM Study BHQ; participants are invited to complete BHQs 1 year after their last completed BHQ was submitted. Some participants report their diagnosis to the study outside the context of the BHQ, by email or phone. When participants self-report a cancer diagnosis, WISDOM site coordinators call the participant to verify details about their diagnosis and then conduct chart review to fully confirm the breast cancer diagnosis and collect specific data on their breast cancer, including stage and receptor status (Figs S3 and S4, Data Supplement, case verification process details). When chart review or communication with a participant uncovers incorrect self-reporting of cancer, the database is corrected to reflect the participant’s cancer status according to their medical record. In this manuscript, chart review is defined as WISDOM site coordinators conducting a thorough manual chart review of all available breast pathology reports and clinician notes related to breast cancer to ascertain diagnosis status.

Cancer Ascertainment: Data Warehouse Procedural Process With Confirmation

Of the 11,314 individuals with a UC breast-related care record, we identified those with a diagnosis of breast cancer (ICD-10-CM codes C50.011-C50.909, indicating invasive breast cancer, and D05.01-D05.03, indicating ductal carcinoma in situ [DCIS]). Diagnoses codes were pulled from all sources (professional billing diagnoses, problem list, hospital problem list, hospital billing diagnoses, encounter diagnoses, admit diagnoses, hospital diagnoses, hospital problem list, and health issues). To confirm the accuracy of the UCDW findings, for those individuals who had not self-reported with confirmation, chart review was conducted for documentary evidence of the participant’s breast cancer diagnosis; if confirmed, specific breast cancer data were collected.

False-Positive Cases in Data Warehouse Procedural Process

We quantified the number of participants with EHR breast cancer diagnosis codes without a confirmed diagnosis by chart review to assess accuracy of the data warehouse procedural process (DWPP). In addition, we quantified the number of breast cancer diagnosis codes found in records for false-positive cases and compared them with the confirmed positive cases.

Self-Reported to WISDOM but No Cancer Code in the UCDW: False Negatives

We quantified the number of participants who self-reported a cancer diagnosis to WISDOM and did not have any C50* or D05.10-.9 code in the UCDW. Those who self-reported a cancer diagnosis were confirmed to be diagnosed with breast cancer by manual chart review, and those who received their breast cancer care at a UC were considered a false-negative case.

Validation by Comparison With the CCR

To further assess the accuracy of ascertaining cancer diagnoses through the DWPP and self-report, we searched for breast cancer diagnoses according to the CCR among our study cohort of 11,314 participants by matching on their first name, last name, date of birth, and home address. The CCR data file was received in June 2022, which included complete diagnoses from 2016 to June 2020.

RESULTS

Table 1 displays demographic characteristics for our study cohort (n = 11,314). The average age was 59.4 years (standard deviation = 10.0) with all self-reporting as female and a majority self-reporting as college graduates (89%) and non-Hispanic White (84%).

TABLE 1.

Descriptive Characteristics of WISDOM Participants Enrolled From September 2016 to December 2021 With at Least One Code Specific to Breast Care in the University of California Data Warehouse (n = 11,314)

Characteristic	Study Cohort
Age, years, mean (SD)	59.4 (10.0)
Self-report sex, %
Female	100.0
Male	0.0
Race and ethnicity, %
Non-Hispanic White	84.4
Non-Hispanic Black	2.9
Non-Hispanic Asian	7.8
Non-Hispanic AI/AN	0.1
Non-Hispanic NH/PI	0.2
Non-Hispanic two or more races	3.6
Hispanic	10.3
Unknown	1.0
Educational attainment, %
Some high school	0.1
High school graduate	1.6
Some college	15.4
College graduate	82.9
Study arm, %
Personalized screening	65.4
Annual screening	34.6

NOTE. UC health systems included UC Davis, UC Irvine, UC Los Angeles, UC Riverside, UC San Diego, and UC San Francisco.

Abbreviations: AI, American Indian; AN, Alaska Native; NH, Native Hawaiian; PI, Pacific Islander; SD, standard deviation; UC, University of California.

Confirmation of Breast Cancer Cases

Invasive Cancer (C50*)

As shown in Figure 2, among the 11,314 participants in our study cohort, 160 participants (Box A) had at least one ICD-10-CM C50* invasive breast cancer diagnostic code. Of them, 136 individuals (83%) were confirmed to have a breast cancer diagnosis through manual chart review. Of the 136, 114 (86%) individuals (Box B) had self-reported their diagnosis, and 22 (17%) individuals (Box C) were uncovered by the DWPP.

FIG 2.

Data Warehouse Procedural Process to identify breast cancer diagnoses in the UCDW. C50*, an ICD-10 code that codes for invasive breast cancer; CPT, Current Procedural Terminology; D01.01-03, ICD-10 code that codes for DCIS; DCIS, ductal carcinoma in situ; EHR, electronic health record; ICD-10-CM, International Classification of Diseases 10th Revision Clinical Modification; UCDW, University of California Data Warehouse; WISDOM, Women Informed to Screen Based On Measures of risk.

Twenty-four individuals (15% of the 160 with any C50* code according to the DWPP; Box D) did not appear to have a diagnosis documented in either a pathology report or a clinical note; thus, no breast cancer diagnosis was confirmed for these individuals.

Ductal Carcinoma In Situ (D05.1-D05.9)

Among the 11,314 participants in our study cohort, 27 individuals (Box E) had at least one D05.01-03 and no C50* code, which codes for a DCIS diagnosis. Of them, 16 individuals were confirmed by chart review to have DCIS: 15 (94%) individuals (Box F) had self-reported their diagnoses to WISDOM, and one (Box G) was uncovered by the DWPP. Thus, 59% of the 27 individuals who had a recorded D05.01-03 and no C50* code had a confirmed (via chart review) breast cancer diagnosis.

Eleven individuals (41% of the 27 with a D05 code and no C50* code; Box H) did not have a diagnosis documented in either a pathology report or a clinical note; thus, no DCIS diagnosis was confirmed for these participants.

Newly Identified Cases by DWPP That Had Not Been Self-Reported

WISDOM participants most commonly report their diagnosis through annual breast health update questionnaires (BHQs). Our analysis considered the timing of when the diagnosis was given, as some individuals had not yet sent the BHQ to report their diagnosis to WISDOM at the time when the DWPP was implemented. Twenty-three individuals had not self-reported their breast cancer diagnosis to WISDOM but had a breast cancer diagnosis via EHR chart review which was later confirmed (Fig 3). Of these 23, four had been diagnosed >1 year ago and had not filled out a BHQ since their diagnosis. Among the 19 participants who were diagnosed ≤1 year ago, five had not filled out their BHQ in over 18 months. We considered these nine participants (6% of total ascertained cases identified by the DWPP) unresponsive or lost to follow-up. Since 11 (48%) individuals had not sent their most recent BHQ before the data pull, it is unclear whether they would have reported the diagnosis or not given more time. Three ended up having a correct self-report, given that they reported a diagnosis to WISDOM via the BHQ within 1 month after the data pull.

FIG 3.

Newly identified breast cancer cases (invasive and DCIS) by data warehouse procedural process (UCDW Procedural Process) with a breast cancer diagnostic code that had not been self-reported to WISDOM. BHQ, Breast Health Questionnaire; DCIS, ductal carcinoma in situ; UCDW, University of California Data Warehouse.

We did not identify any cases where a participant responded to a BHQ and did not report their diagnoses.

Self-Reported Breast Cancer Diagnoses Not Found in the UCDW: False Negatives

As shown in Figure 4, among those who had at least one ICD-10-CM/CPT breast-specific code in their medical record, 20 individuals had self-reported a breast cancer diagnosis which was confirmed directly with the participant but did not have either a C50* or an D05.01-03 code in their UCDW record. Among these individuals, 19 (95%) did not appear to seek breast cancer care at a UC, and one participant, who received a second opinion at a UC, was likely missing from the UCDW because of miscoding. We consider this one participant a false negative.

FIG 4.

Self-reported cases without a UCDW breast cancer diagnostic code. UC, University of California; UCDW, University of California Data Warehouse.

Summary of Breast Cancer Diagnoses

Our study identified 172 participants with confirmed breast cancer diagnoses by the following sources: 129 (75%) by both self-report and DWPP, 20 (12%) by self-report only (Table 2), and 23 (13%) newly identified by DWPP alone (Table 2). We also examined the diagnoses during two time periods: before June 2020, during which cancer diagnoses were available from the CCR, and from June 2020 to December 2021.

TABLE 2.

Diagnosed Invasive and DCIS Cases Identified by Self-Report and DWPP (n = 11,314)

Invasive and DCIS Cancer Cases by Data Source	Time Lapse Since Diagnosesa
	Diagnoses ≥20 Months Old (September 2016-May 2020)	% Cases Ascertained in Comparison to the CCR	Diagnoses <20 Months Old (June 2020-December 2021)	% Cases Ascertained in the Study Period	All Diagnoses (September 2016-December 2021)	% Cases Ascertained in the Study Period
Self-reported also identified by DWPPb	69	85	60	66	129	75
Self-reported not identified by DWPPc	9	11	11	12	20	12
Newly identified by DWPPd	3	4	20	22	23	13
Total ascertained cases	81e	100	91	100	172	100

NOTE. CCR report for June 2020-December 2021 diagnoses was not available at the time of the study.

Abbreviations: CCR, California Cancer Registry; DCIS, ductal carcinoma in situ; DWPP, data warehouse procedural process; UCDW, University of California Data Warehouse.

^aJanuary 1, 2022, data warehouse data and February 1, 2022, WISDOM self-report data matched with WISDOM cohort.

^bCases previously self-reported and confirmed through WISDOM Study with a breast cancer diagnostic code in UCDW.

^cCases previously self-reported and confirmed through WISDOM study without a breast cancer diagnostic code in UCDW.

^dNew cases found through the DWPP that were not previously self-reported and confirmed through WISDOM study.

^eLinking pre-June 2020 cases reported by the CCR to our study sample identified 81 breast cancer diagnoses.

For validation, we linked pre-June 2020 cases reported by the CCR to our study sample and identified 81 breast cancer diagnoses. These 81 CCR cases, which represent a small subset of the total number of breast cancer incident cases ascertained by the CCR during the same time period, were identified by combining self-report and DWPP for diagnosis that occurred ≥20 months prior: 69 (85%) by both self-report and DWPP, nine (11%) by self-report only, and three (4%) newly identified by the DWPP alone. For the 81 confirmed cases by the CCR, the self-report rate for cancers diagnosed before June 2020 (≥20 months prior to the January 1, 2020 data pull) was 96%. Between June 2020 and December 2021, of the 91 cases with confirmed breast cancer, 71 (78%) breast cancer cases had been self-reported, and an additional 20 were newly identified by the DWPP.

DISCUSSION

Our data warehouse cancer ascertainment process, linking WISDOM Study participants with the UCDW, identified a meaningful number of breast cancer diagnoses (13%) missed by self-report. Cancer ascertainment with EHR data from the UCDW can be completed in 1-3 months after diagnosis, significantly faster than the 13-15 months required for completed self-report data from annual questionnaires and 2 years for CCR reporting. Self-reported data with confirmation alone provided quick ascertainment with relative accuracy; however, not all cases were found. Cancer ascertainment was improved by combining self-reported data with EHR data from the UCDW, particularly for missing cases because of self-reported questionnaire issues such as timing and lack of response; however, chart review was critical to identify false-positive codes. Accuracy of self-reported cancer diagnosis from annually distributed questionnaires improves over time, making the DWPP more essential for reporting recent diagnoses more accurately than older diagnoses. For diagnoses that occurred ≥20 months before the January 1, 2022, UCDW data pull, we found a high concordance of cancer cases identified through self-report with confirmation (96%) and our DWPP (89%) when compared with WISDOM Study UC Health center patient cases identified by the gold standard, the CCR. There were nonoverlapping cases identified by self-report and data warehouse EHR codes and combined were a 100% match with the WISDOM UC patient cases identified by the cancer registry. This suggests that our process was effective in identifying most breast cancer diagnoses among the WISDOM study cohort who received breast care at a UC Health center.

This targeted, procedural process is particularly helpful for a national study such as WISDOM because it would be impractical to do chart review on such a large number of participants. Additionally, it is not ideal for the WISDOM Study to wait for cancer registry reporting because registry latency (1-2 years) would create considerable delays for DSMB reporting. Our study demonstrates that breast cancer case ascertainment through combined self-report and a DWPP can enhance the ability to detect incident cancers more quickly, thereby identifying possible safety issues in large pragmatic clinical trials that specify safety end points of cancer detection. This combined process provides earlier ascertainment than that obtained by relying on cancer registry alone.

Although WISDOM self-report was relatively high, our DWPP uncovered a substantial number of cases that had not been identified by self-report. Differences in reporting between cases identified through self-report and those identified by our DWPP were primarily due to questionnaire delays or lack of response to the BHQ (estimated BHQ response rate ranges between 56%-67%), underscoring the importance of our process for early ascertainment. A high percentage (48%) would not have been able to self-report because of the time lag between their diagnoses and the receipt and submission of the BHQ. Our process allowed us to address these issues by finding these cases up to a year earlier for those who did not receive their BHQ after diagnosis and for those unresponsive to the BHQ. Despite missed cases by self-report, we observed a high self-report rate in our study cohort for those who were diagnosed pre-June 2020 (96%). We attribute the strong cancer self-report rate to WISDOM Study reminders to complete the annual questionnaires and the study’s communication and community-building efforts. This high self-report rate indicates that those diagnosed with cancer are filling out their BHQ at a higher rate than the average WISDOM participant.

Although our process using EHR codes identified individuals with breast cancer, we also observed a relatively high false-positive rate with participants who had a C50* or D05.01-03 code in the data warehouse. This is consistent with previous studies reporting substantial false-positive rates.^1,2,7 Thus, EHR codes can be helpful for finding cancers missed by self-report but can introduce diagnostic errors.⁸ This process had a very low false-negative rate, which arguably is more important than a low false-positive rate with respect to accurately ascertaining all cancer cases for DSMB reporting. A high false-positive rate can be remedied through chart review; however, a high false-negative rate cannot be corrected until there is complete cancer registry reporting. To ascertain cancer accurately, it was critical to do chart review for confirmation of a diagnosis.

A limitation of our study was that we could not definitively conclude that a data warehouse cancer ascertainment process can capture all cancer cases. Missing data in EHR systems are likely to occur for patients who seek care at multiple health care systems.^9-11 Studies have observed that a significant percentage of patients with cancer (16%-25%) received cancer care from more than two institutions.^12-14 This highlights a potential benefit of integrating other regionally located data warehouses (ie, using a medical record resource such as Care Everywhere) to capture additional diagnoses. Additionally, non-Hispanic White women are overrepresented in our analytic sample (84.4% non-Hispanic White), which is less diverse the race and ethnicity mix in California and therefore limits the generalizability of our findings.

The strengths of the study include the large number of participants, the effective use of almost real-time, real-world EHR data, and chart review confirmation of all cases. In addition, because all participants consented to WISDOM accessing their EHR data, WISDOM has a higher likelihood of ascertaining cancer diagnoses from EHR data compared with studies that do not have EHR access for all participants. This study’s high ascertainment rate from EHR records allowed WISDOM to evaluate the accuracy of self-reported diagnoses before the availability of cancer registry reporting. Furthermore, the complimentary use of self-report allowed for identification of cases outside of UCDW’s capability, those participants who had received breast-related care in the UC Health system but were diagnosed and treated outside the system.

In conclusion, no data source or process is perfect for capturing every breast cancer; however, we have shown that a multipronged approach using complimentary data sources (self-report and a DWPP) can enhance the ability to accurately detect cases quicker and thus enable timely reporting of cancer cases in a large, pragmatic study of breast cancer screening such as WISDOM. An opportunity exists to reduce the latency of cancer case ascertainment in cancer registries by integrating them with institutional EHR systems or their data warehouse, as has been previously suggested.^13,15 Using multiple methods is superior to a single data source. We have presented a generalizable process. However, it should be noted that informatics-driven processes, such as DWPP, require customization on the basis of a deep understanding of the disease, participant perceptions related to their illness, and clinical workflows, all of which are inextricably linked to the use of the real-world data for research. Each data source has strengths and limitations regarding missingness, inaccuracies, and timeliness—the key is to continuously improve processes to optimize the use of data sources to fit the research objectives.

CONTEXT.

Key Objective

Faster ascertainment of cancer diagnoses in a clinical trial requires outcome monitoring by combining two data sources, clinical data warehouse data and self-reported online questionnaires.

Knowledge Generated

Accuracy of self-reported cancer diagnosis can be high (96%) within a highly engaged participant population; however, delays up to a year between diagnosis and submission of an annual questionnaire affect timely reporting. Studies can use electronic health record (EHR) diagnostic codes to reduce this time lag by uncovering recent cancer diagnoses among participants.

Relevance

Our findings support integration of cancer registries with institutional EHR systems to reduce cancer registry case ascertainment latency.