Using hospital medical record data to assess the accuracy of the SEER Los Angeles Cancer Surveillance Program for initial treatment of prostate cancer: a small pilot study

Abstract

Purpose

Treatment information from the Surveillance, Epidemiology, and End Result Program (SEER) cancer registries is increasingly being used for population-based cancer research; however, it may be incomplete for outpatient procedures and is not quality controlled. We sought to validate SEER information on initial treatment of prostate cancer by comparison to electronic medical record (EMR) review.

Methods

Patients diagnosed with prostate cancer between 1 January 2010 and 31 December 2014 in Los Angeles County who received treatment at our institution within 6 months of diagnosis were identified from the SEER registry. We reviewed the hospital EMR for these patients and identified initial treatment received within 6 months of diagnosis. We compared data reported to SEER data to our re-abstracted hospital EMR data (defined as the gold standard) to identify the completeness of SEER treatment data (sensitivity) and the accuracy of the SEER information (positive predictive value).

Results

Based on 266 eligible patients, SEER’s sensitivity in capturing initial treatment was 95.9% (118/123) for prostatectomy, 95.8% (69/72) for no treatment, 87.5% (21/24) for radiation therapy, 68.3% (28/41) for active surveillance or watchful waiting, and 50.0% (2/4) for cryosurgery. The SEER positive predictive value was 100% for radiation therapy and cryosurgery, 97.5% (118/121) for radical prostatectomy, 82.3% (28/34) for active surveillance or watchful waiting, and 78.4% (69/88) for no treatment.

Conclusion

The SEER data were highly sensitive and has a high positive predictive value for surgery and radiation therapy but underreported use of active surveillance. These results may assist researchers in understanding the strengths and weaknesses of using SEER prostate cancer treatment data.

Introduction

The Surveillance, Epidemiology, and End Results Program (SEER) was created in 1973 and collects information regarding “incidence and survival data from population-based cancer registries and now covers approximately 28% of the US population” [1]. Each year about 500,000 new cases are added to the SEER database and as of November 2016 there were 9,675,661 cases captured in the SEER database [2]. The SEER Program was initially designed to collect data regarding the incidence of various cancers and diagnostic variables including histology, staging, and patient demographics and has quality control measures in place to ensure the accuracy of these variables. However, although the SEER program also collects information on patients’ cancer treatment modalities, abstraction of outpatient records is often limited [3, 4] and quality control measures have not been used to ensure the accuracy of the treatment variables prior to 2015 [5]. Despite this, the SEER treatment related variables are increasingly being used for population-based cancer research and to measure whether hospitals and providers are following published guidelines in care delivery [6-10].

As the SEER Program continues to be used for both incidence measures and quality of care assessment, it is important to ensure that it is accurate and comprehensive, not only in regard to diagnostic variables, but also the treatment patients receive. The SEER Program has been shown in previous studies to have a greater than 90% agreement between SEER and Medicare claims in identifying a patient’s receipt of surgery and radiation therapy [11, 12]. However, recent studies have shown that there may be geographic variation in the way data are recorded [13], limitations in SEER’s ability to capture outpatient treatments [3, 4], and lower ascertainment of adjuvant treatments the further in time they are received from the primary treatment [14, 15].

The SEER Program is made up of multiple population-based cancer registries. The second largest of these registries is the Los Angeles Registry, also known as the Los Angeles County Cancer Surveillance Program (SEER-LA) which has been utilized in over 3,300 scientific publications [16, 17]. This study’s aim was to assess the SEER-LA database’s sensitivity and positive predictive value in capturing and accurately identifying all types of initial treatment for prostate cancer among patients with localized disease, using re-abstracted medical record information on patients seen at our facility as the gold standard.

Methods

Registry details

The abstraction of cancer medical record information at the University of Southern California Norris Comprehensive Cancer Center (Norris), a university-based private cancer hospital that is a National Cancer Institute (NCI) designated comprehensive cancer, is performed by hospital cancer registrars and reported to the SEER-LA registry. SEER mandates that hospital abstraction of cases be completed within 6 months of diagnosis and that the first treatment received be recorded regardless of how long after diagnosis it is received. After the Norris Hospital data have been reported to the SEER-LA registry, the cases are consolidated with data submitted by other hospitals where the patient has been seen to ensure that there are no duplicate cases. Data regarding treatment selection were not quality controlled prior to 2015 [5] and only Commission of Cancer (CoC) accredited hospitals are required to submit updated adjuvant treatment data as additional treatments are received.

Patient selection

We requested data from SEER-LA for patients who were diagnosed with prostate cancer between 1 January 2010 and 31 December 2014 (at any facility) and then received their initial treatment between 1 January 2010 and 30 June 2015 at Norris. CoC Class of case (CL) variables were used as a screening tool to identify patients who initiated treatment at Norris. The Class of case variable was not a required variable by the Centers for Disease Control and Prevention or NCI programs during the time period of this study but was available for all patients. The relevant CL codes were 10 (Initial diagnosis and part or all of first course of treatment or a decision not to treat done at the reporting facility), 12 (Initial diagnosis by staff physician and part of first course treatment was done at the reporting facility), 13 (Initial diagnosis and part of first course treatment was done at the reporting facility), 14 (Initial diagnosis and all first course treatment or a decision not to treat was done at the reporting facility), 20 (Initial diagnosis elsewhere and all or part of first course treatment was done at the reporting facility), 21 (Initial diagnosis elsewhere and part of treatment was done at the reporting facility), and 22 (Initial diagnosis elsewhere and all treatment was done at the reporting facility). The EMR of patients with a CL code indicating that treatment was initiated at Norris were reviewed to confirm that patient had not been previously treated elsewhere. Patients were excluded from the study if they had a Gleason score < 6 or were diagnosed at autopsy.

Treatment variables

SEER-LA registry

Using the data provided by the SEER-LA registry, we identified the type of initial treatment received for each patient. The types of treatment were categorized as: No treatment (did not receive any form of definitive treatment and no indication of being on active surveillance or watchful waiting), Active Surveillance or watchful waiting (AS), Radical Prostatectomy (RP), Cryosurgery, and Radiation therapy (any type). Specific variables used to identify these treatments were RX Summ—Treatment Status NAACCR item 1285, RX Summ—Radiation NAACCR item1360, and RX Summ—Surgery Type NAACCR item 1640. If the patient received radiation therapy or surgical management, we used the variables RX Date Surgery NAACCR item 1200, RX Date Radiation NAACCR item 1210, and Date Initial RX SEER NAACCR item 1260 to identify which treatment a patient received first.

EMR review

Using the patients identified by the SEER-LA registry, a blinded retrospective chart review was conducted at Norris using the EMR to abstract types of initial treatments received. The types of treatment were categorized as: No treatment (did not receive any form of definitive treatment and no indication of being on active surveillance or watchful waiting), AS, RP, Cryosurgery, and Radiation therapy (any type). The retrospective chart review was performed by a single reviewer for consistency in identifying the treatment received. Patients were only categorized as receiving AS if an active surveillance or watchful waiting protocol was explicitly stated within the EMR. RP and cryosurgery were identified using surgical procedure notes within the EMR. Patients who received radiation therapy were identified using the radiation therapy treatment summary note in the EMR that delineates each fraction of radiation therapy delivered over the course of treatment.

Quality control

If the EMR review identified information that was missing or inconsistent with SEER-LA information, the patient’s records were reviewed by a NCI SEER-LA quality control supervisor who confirmed whether it was an inaccuracy within the SEER-LA database, or if was not reported to SEER-LA. If an error was identified within the SEER-LA database (based on coding information contained in the abstract), it was corrected by the NCI SEER-LA quality control supervisor. We also assured that the data included from the EMR were based on treatment received within the first 6 months after diagnosis.

Statistical methods

Frequency distributions of the types of therapies received were compared for the Registry variables and the EMR variables. The re-abstracted EMR data were considered the “gold standard” and was compared to data reported to SEER-LA to determine the sensitivity and positive predictive value of the SEER-LA database in capturing the type of initial treatment received.

Results

We identified 268 patients from the SEER-LA database who received treatment at Norris within 6 months of diagnosis. Two of these patients were excluded: one patient with a Gleason score of 5 and one patient who was treated outside the US before presenting at Norris. Differences in the distribution of patient demographic and clinical characteristics for each treatment category (based on the EMR data) are shown in Table 1. The Norris EMR data identified 41 patients on AS, 24 with radiation therapy, 129 patients with surgery (including 123 with RP, two with cystoprostatectomy, and four with cryosurgery), and 72 patients as receiving no treatment. Patients with no treatment differed from AS patients: they were more likely to be 75+ years old, of Asian/other race/ethnicity, and have a higher PSA, Gleason score, and stage at diagnosis. Patients receiving radiation versus surgery were also more likely to be older and to have more aggressive clinical characteristics.

Table 1.

Clinical and demographic characteristics of prostate cancer patients treated at Norris Hospital between 2010 and 2015 by type of initial treatment, based on Hospital EMR data

Variables	Active surveillance		Radiation		Surgerya		No treatment
	(n = 41)	%	(n = 24)	%	(n = 129)	%	(n = 72)	%
Age at diagnosis
Range	49.5-87.1		43.6-86.5		45.7-82.3		42.7-92.1
Mean	67.7		67.2		64.8		68.2
Median	66.9		70		64.1		68.4
< 60	7	17.1	5	20.8	31	24.0	16	22.2
60-64	9	22.0	3	12.5	39	30.2	9	12.5
65-69	11	26.8	7	29.2	25	19.4	16	22.2
70-74	6	14.6	3	12.5	22	17.1	11	15.3
75 +	8	19.5	6	25.0	12	9.3	20	27.8
Ethnicity
Non-Latino White	29	70.7	16	66.7	70	54.3	47	65.3
Latino	5	12.2	3	12.5	40	31.0	4	5.6
Black	1	2.4	2	8.3	4	3.1	4	5.65
Asian/other	6	14.7	3	12.5	15	11.6	17	23.6
SESb
1 (lowest)	4	9.8	2	8.3	15	11.6	9	12.5
2	4	9.8	5	20.8	22	17.1	12	16.7
3	3	7.3	7	29.2	19	14.7	10	13.9
4	10	24.4	4	16.7	35	27.1	12	16.7
5 (highest)	19	46.3	6	25.0	36	27.9	28	38.8
Unknown	1	2.4	0	0.0	2	1.6	1	1.4
Marital status
Married	32	78.0%	18	75.0%	106	82.2	49	68.1
Unmarried	9	22.0	6	25.0	23	17.8	23	31.9
PSA at diagnosisc
Range	0.63-20.0		1.6-38.3		0.4-139.5		0.2-149
Mean	5.7		10.7		14.7		32.3
Median	4.8		7.89		7.2		10.75
< 5	25	61.0	7	29.2	38	29.5	23	31.9
5-9	12	29.3	7	29.2	40	31.0	13	18.1
10-14	2	4.9	4	16.7	16	12.4	7	9.7
15-19	1	2.4	0	0.0	14	10.8	3	4.2
20 +	1	2.4	6	25.0	21	16.3	26	36.1
Gleason score at diagnosis
6	37	90.2	1	4.2	31	24.0	19	26.4
7	4	9.8	6	25.0	48	37.2	9	12.5
8-10	0	0.0	14	58.3	50	38.8	33	45.8
Unknown	0	0.0	3	12.5	0	0.0	11	15.3
Clinical T stage
T1c	38	92.7	8	33.3	66	51.2	27	37.5
T2a	2	4.9	7	29.2	31	24.0	14	19.4
T2b	0	0.0	1	4.2	2	1.6	3	4.2
T2c	0	0.0	0	0.0	13	10.0	5	6.9
T3a	0	0.0	1	4.2	4	3.1	3	4.2
T4	0	0.0	4	16.7	0	0.0	12	16.7
Not stated	1	2.4	3	12.5	13	10.0	8	11.1

^aThe category surgery includes radical prostatectomy, cystoprostatectomy, and cryosurgery

^bSES was determined using the SEER variable for socioeconomic status that groups patients into a quintile with one being the lowest SES and five being the highest SES. All other data presented in this chart was obtained through EMR chart review

^cPSA at diagnosis was defined as a PSA value obtained 1 year before diagnosis or 3 months after diagnosis but before treatment was initiated

Sensitivity of SEER-LA registry data

The SEER-LA database’s sensitivity in capturing the initial treatment received was 95.9% (118/123) for RP, 95.8% (69/72) for no treatment, 87.5% (21/24) for radiation therapy, 68.3% (28/41) for AS, and 50.0% (2/4) for cryosurgery (Table 2).

Table 2.

Number of prostate cases by type of initial therapy based on EMR review and SEER-LA registry database, and sensitivity and positive predictive value of SEER-LA data compared to EMR review

	EMR	SEER-LA	SEER-LA accu- rate (vs. EMR)	Sensitivity of SEER-LA (%)	Positive predictive value of SEER-LA (%)
Radiation therapy	24	21	21	87.5	100
Prostatectomy	123	121	118	95.9	97.5
Cystoprostatectomy	2	0	N/A	N/A	N/A
Cryosurgery	4	2	2	50.0	100
Active surveillance	41	34	28	68.3	82.3
No treatment	72	88	69	95.8	78.4

EMR electronic medical record, SEER-LA Surveillance, Epidemiology, and End Results Program Los Angeles County Cancer Surveillance Program

Sensitivity = (number of patients SEER-LA accurately identified as receiving a given treatment)/(number of patients identified by the EMR as receiving a given treatment)

Positive predictive value = (number of patients SEER-LA accurately identified as receiving a given treatment)/(number of patients SEER-LA identified as receiving a given treatment)

In general, the discrepancies in treatment information between the EMR and SEER-LA data were largely due to missing information on definitive treatment in the SEER-LA database. Of the three patients not reported by SEER-LA as receiving radiation therapy, two patients were recorded as receiving no treatment and one patient was recorded as receiving AS. Of the five patients not reported by SEER-LA as receiving RP, two were recorded as receiving AS and three were recorded as receiving no treatment. Similarly, the two patients not reported by SEER-LA as receiving cryosurgery were recorded as receiving no treatment. Among 13 AS cases identified based on EMR abstracting but not identified by SEER-LA, almost all (12/13) were coded by SEER-LA as having received no treatment and one was recorded as receiving a RP (which the patient did receive at the conclusion of active surveillance). In contrast, there were three patients coded by the EMR as receiving no treatment that were coded as receiving AS by SEER-LA (Table 3).

Table 3.

Initial treatments for prostate cancer identified by SEER-LA versus EMR

	EMR radiation	EMR RP	EMR CP	EMR AS	EMR no treatment	EMR cryosur- gery
SEER-LA radiation	21	0	0	0	0	0
SEER-LA RP	0	118	2	1	0	0
SEER-LA CP	0	0	0	0	0	0
SEER-LA AS	1	2	0	28	3	0
SEER-LA no treatment	2	3	0	12	69	2
SEER-LA cryosurgery	0	0	0	0	0	2

EMR electronic medical record, SEER-LA surveillance, epidemiology, and end results program Los Angeles County cancer surveillance program, RT radiation therapy, RP radical prostatectomy, AS active surveillance, CP cystoprostatectomy

Initial treatment received: positive predictive value (PPV) of SEER-LA registry data

SEER-LA identified 21 patients as receiving radiation therapy, 121 patients as receiving RP, two patients as receiving cryosurgery, 34 patients as receiving AS, and 88 patients as receiving no treatment. The PPV of the SEER-LA database for these treatments was 100% for radiation therapy and cryosurgery, 97.5% (118/121) for RP, 82.3% (28/34) for AS, and 78.4% (69/88) for no treatment (Table 2).

Of the three patients who were inaccurately identified by SEER-LA as receiving a RP as their initial treatment, two had a cystoprostatectomy and one received AS before having a RP based on the EMR review. Of the six patients who were inaccurately identified by SEER-LA as being on AS, three did not receive any treatment and were not being followed by a watchful waiting approach, one patient had radiation therapy after a delay in treatment, and two had a RP after a delay in treatment. Of the 19 patients SEER-LA inaccurately identified as receiving no treatment, 12 had AS, two had radiation therapy, two had cryosurgery, and three had a RP (Table 2).

Discussion

We found SEER-LA data to have a high sensitivity in capturing patients who received a RP, radiation therapy, and patients who received no treatment. SEER-LA also had a high PPV in reporting patients who received radiation therapy, RP, or cryosurgery. However, the SEER-LA program underreported AS and coded several of these cases as receiving no treatment, perhaps because of the inability to find specific information on this management strategy in hospital abstracts which is typically found in the assessment and plan section of physician notes.

Misclassification of cystoprostatectomy

The two patients who had a cystoprostatectomy, but recorded in SEER-LA as having a RP, had incidentally discovered prostate cancer following cystoprostatectomy. Studies have shown that between 14 and 50% of patients who undergo radical cystoprostatectomy are found to have incidentally discovered prostate cancer [18-24]. Studies focusing on treatment of localized prostate cancer may want to exclude any patient who received a cystoprostatectomy or who has comorbid bladder cancer due to their different clinical circumstances. This would likely have a minimal impact on the number of patients included in the study while making it more representative of the intended study population.

Underascertainment and inaccuracies in recording active surveillance

The SEER-LA Program’s limitations in capturing patients on AS was responsible for 67.9% (19/28) of patients’ first treatments being misclassified. With the help of a quality control supervisor from SEER-LA, we identified that 10/13 of the patients who were not coded as receiving AS actually had abstracted data available on the patients’ AS protocol within SEER-LA abstractor notes. Thus, raising the awareness of SEER-LA abstractors to the importance of coding AS may be helpful. We were unable to determine why five patients were recorded in SEER-LA as receiving AS when there was no record of it in the EMR. It is possible that patients had previously received AS at another facility, and thus the PPV for AS and other treatments should be considered the lowest PPV possible with the true PPV potentially being higher. However, for each of these patients, we ensured that, according to their class of case code, they had received treatment at Norris. These discrepancies could be due to variability in the abstractor’s definition of AS, as reflected in research articles which have used different criteria to identify patients on AS within the SEER database [25-28]. This may suggest the importance of standardizing the definition of AS.

Radiation therapy and surgical treatments

We identified three patients who received radiation therapy and five patients who received surgical management that were not captured by the SEER-LA database; however, there were notes from abstractors indicating planned or recommended surgery or radiation therapy. In each case, the data were abstracted prior to the patient receiving the recommended or planned treatment. Since SEER-LA does not have a mechanism or requirement that hospitals update treatment information, these additions to the record would not have been included in the registry database.

Impact of time of abstraction

Knowing that most hospitals are not required to submit updated treatment information increases the significance of the date of abstraction, which is supposed to be completed within 6 months of diagnosis. Treatments received after the initial abstraction may not be recorded within SEER. Also, abstraction of treatment is largely based on inpatient hospital records. Thus, identification of some treatment regimens, such as AS, which may be recorded in physician notes or outpatient records may not be available at the time of abstraction and this may explain why there was low sensitivity for AS within SEER-LA. Other treatments, which may be recommended, but not received at the time of the record abstraction, may also be underreported. This coincides with other studies which have shown that delays in treatment contribute to underascertainment of treatments within the SEER database [14, 29].

Limitations

This study has several limitations. One limitation of this study is that it the small sample size and restriction to one hospital. Since all patients in this study were from a single institution, it is possible that the underascertainment and inaccuracies identified could be specific to Norris rather than systemic limitations within the SEER-LA Program. Additionally, the Summary Treatment variables used to identify treatment received include treatment received at any facility. CL codes were used in this study to ensure that patients initiated treatment at Norris but the CL codes have not been studied for accuracy. The EMR of each patient with a CL code indicating that initial treatment was received at Norris was reviewed to confirm that the patient had not previously been treated at another facility. However, patients may have been treated at another facility after being seen at Norris and patients with a CL variable indicating that they initiated treatment at another facility may have been inappropriately excluded if the CL variable was miscoded.

Conclusions

SEER-LA remains a source of high-quality data for researchers to use when performing population-based research. In our study, it demonstrated a high degree of sensitivity and high PPV in capturing and identifying patients who had radiation therapy or RP as their initial treatment for prostate cancer. However, its limitations were in reporting AS. As this is a single institution study, further investigation into the accuracy of recording AS within SEER-LA is needed at other institutions.