Abstract
Background
Surveillance, Epidemiology, and End Results (SEER) cancer registries provide accurate information on cancer surgery and radiation, but the validity of registry data on chemotherapy and hormone therapy for breast cancer has not been well studied. We validated the registry data for chemotherapy and hormone therapy against an independent medical chart review.
Methods
We identified 1,228 women diagnosed with breast cancer at age > =65 in 1993–1999 in the New Mexico SEER Tumor Registry and completed medical chart reviews.
Results
Overall, there was moderate agreement between these two databases on chemotherapy that was received within 6 months of diagnosis. The observed agreement was 96.0%, with a kappa of 0.72 (95% confidence interval: 0.64–0.79). The sensitivity of the registry data for chemotherapy was 70.7% and the specificity was 98.2%. The positive predictive value of the registry data for chemotherapy was 77.8%. The sensitivity of the registry data for hormone therapy was 59.7%, and the specificity was 89.5%. The observed agreement for hormone therapy was 80.0%, with a kappa of 0.52 (0.46–0.57).
Conclusion
Agreement on chemotherapy and hormone therapy between the New Mexico SEER Tumor Registry and chart reviews was moderate. The preferred approach would be to combine data from different sources to obtain more complete information.
Keywords: Breast cancer, Chemotherapy, Hormone therapy, Tumor registry, SEER, Chart review
1. Introduction
The National Cancer Institute's Surveillance, Epidemiology, and End Results (SEER) tumor registries provide excellent information for estimating cancer incidence, mortality, and treatment trends; for identifying unusual changes over time; and for promoting studies designed to identify factors amenable to cancer control interventions [1–3]. The SEER data have been extensively used to study the pattern and quality of radiation therapy and surgery for breast cancer [4–13], including a number of studies that specifically involved validation of information on cancer radiation therapy and surgery in SEER registries [14–17]. Little research has been conducted on the use of cancer chemotherapy using SEER data [18–24] because the SEER program does not release the information on chemotherapy in the Public Use Database due to concern about whether this information is complete. The validity of the SEER data for chemotherapy and hormone therapy has not been well studied.
Mariotto and colleagues reported that the SEER data on chemotherapy was slightly underreported, and this underreporting was constant over time from 1975 to 1999 with respect to findings from the National Cancer Institute's study on the Patterns of Care (POC) [18]. Harlan et al. [21] reported that the sensitivity of the SEER data for chemotherapy for breast cancer was 78% compared to the POC data collected in 1991 and 1995. Because there was only a small number of cases in the POC study, no reliable information was reported on the validity of information on chemotherapy for breast cancer for each individual SEER registry. Because more cancer care has moved to outpatient or community settings, researchers face challenges in the completeness of the treatment information collected from tumor registries that are mainly based on hospital records [24–26]. Tumor registry data should be enhanced by promoting collection of data from all places of care and by linkage to other population-based data such as nationwide Medicare insurance data [26].
We conducted an independent medical chart review specifically on chemotherapy and hormone therapy for 1,228 women diagnosed with breast cancer at age 65 or older from 1993 to 1999 in the state of New Mexico. This independent medical chart review was used as a “gold standard” to validate existing data from the New Mexico Tumor Registry on chemotherapy and adjuvant hormone therapy that was originally collected as part of the SEER effort. This report presents the findings of this external validation.
2. Methods
We used two independent data sources in this study: the existing New Mexico SEER Tumor Registry data, and the current medical chart review data.
The New Mexico Tumor Registry is a state-wide, population-based tumor registry established in 1966, and has been part of the SEER program since 1973 [1–3]. The registry ascertains all newly diagnosed (incident) cancer cases from multiple reporting sources such as hospitals, outpatient clinics, laboratories, private medical practitioners, nursing/convalescent homes/hospices, autopsy reports, and death certificates [1–3,27]. New Mexico residents who are diagnosed and treated at facilities outside the state are identified through data exchange with surrounding state registries in Colorado, Arizona, Utah, and Texas, as well as information from the New Mexico Bureau of Vital Records and Health Statistics and pathology laboratories and hospitals that operate close to New Mexico Borders [28]. The information on chemotherapy and hormone therapy was collected in all SEER registries, although it has not been released in the SEER Public Use Data Set because of concern about whether it is complete. The SEER program has a well-structured data collection system, and provides details on how therapy information was collected [27]. In brief, after patients with cancer were notified to the registry from the hospitals and pathology laboratories, and after case screening and ascertainment procedures, the SEER abstractors reviewed medical charts for information on the first course of therapy (surgery, radiotherapy, chemotherapy, and hormone therapy) [27]. In our study, we obtained the information on chemotherapy and hormone therapy for cases diagnosed with breast cancer from 1993 to 1999 from the New Mexico Tumor Registry that was already collected in the registry database as part of the SEER effort before our validation study started in 2001.
Because the major goals of our original proposal were to compare an independent medical chart review with Medicare claims and tumor registry for chemotherapy, we only studied women diagnosed with breast cancer at age 65 or older and linked with their Medicare claims files. The data abstractors were then instructed to complete the medical chart reviews for these cases, but were blinded to the status of the currently existing tumor registry data on cancer treatment.
The data collection form was designed and tested on first 20 cases. After a few minor modifications, the abstract form was finalized and comprised of three pages. The first page, which was to be kept all the time in the New Mexico Tumor Registry for confidentiality, included case identification (ID) number, name, social security number, name and address of the diagnosing facility, and if different, the names and addresses of the facility that provided surgery, radiation therapy, chemotherapy, and hormone therapy. The second page contains ID number, date of birth, date of breast cancer diagnosis, age at diagnosis, chemotherapy treatment (yes or no), type of chemotherapy, and date of therapy. Finally, the abstractor recorded where the chemotherapy information was obtained from, covering one or more data sources of medical charts documenting this therapy such as from hospitals, oncologists' offices, radiology departments, or other physicians' offices. The third page collected information on hormone therapy for breast cancer. The data format on hormone therapy were similar to that collected on chemotherapy (described above). The data collection was conducted in March 2001 through February 2003.
The Institutional Review Boards of the University of Texas Medical Branch and the University of New Mexico, and the Committee for the Protection of Human Subjects at the University of Texas Health Science Center at Houston approved this study.
We sought to complete data abstraction forms for a target sample of 1,241 subjects. This sample size was determined to achieve a 0.05 level of precision and an estimated sensitivity at 0.9. We identified 3,282 women from the New Mexico Tumor Registry who were diagnosed with stage I–IV breast cancer at age 65 years or older between 1993 and 1999. Of these women, 1,733 were successfully linked with their Medicare files as part of our previous study on the external validation of Medicare claims for chemotherapy [29]. Excluded were women who did not have full coverage of both Medicare Part A and Part B, or who were members of health maintenance organizations in the year of diagnosis.
The data abstractor sorted the 1,733 eligible cases according to the hospitals where a breast cancer diagnosis was made. Although these cases were not randomly selected for medical chart reviews, the data abstractor worked through the list of cases according to the order of the identification number, and aimed to review medical charts for as many cases as possible during her visit to the hospital or to the oncologist's office across the state of New Mexico. If the medical records were not available during the visit, the data abstractor attempted to complete the abstraction form at the next visit, and so on until we reached the target sample size of 1,241. Of these 1,241 subjects, 13 had incomplete abstraction forms because of missing items, leaving 1,228 cases for the final analysis. We compared the 1,228 cases with chart reviews to the 492 eligible cases without chart reviews, and found no significant difference in the distribution of age between the two groups. However, those without chart reviews had a significantly higher proportion (77.6%) of having been diagnosed earlier in our study periods, that is, 1993–1996, compared to 41.8% among those with chart reviews.
The categorization of receipt of chemotherapy in this analysis was either “yes” or “no” from medical charts and the original tumor registry, regardless of number of records of chemotherapy in medical charts. The sensitivity and specificity of the New Mexico Tumor Registry data for chemotherapy was evaluated using the medical chart review as the reference. The sensitivity of the tumor registry for chemotherapy was defined as the percentage of cases receiving chemotherapy according to medical chart reviews that were also identified by the tumor registry as receiving chemotherapy. Specificity was defined as the percentage of cases identified by medical records as not receiving chemotherapy that were identified by the tumor registry as also not receiving chemotherapy. The sensitivity and specificity of the tumor registry for hormone therapy were similarly defined.
A simple kappa statistic and 95% confidence intervals were calculated to quantify the degree of agreement on the receipt of chemotherapy as well as hormone therapy between the two databases by adjusting for chance agreement. The kappa statistic greater than 0.75 represents excellent agreement beyond chance and values between 0.40–0.75 represent fair to good agreement beyond chance [30]. The concordant rate or observed agreement rate (i.e., number of cases agreed on the receipt of chemotherapy or hormone therapy over the total number of cases) were also calculated. The above analyses were repeated using different time periods of chemotherapy or hormone therapy that was given after diagnosis and recorded in medical chart reviews, for example, within 6 months or after 6 months of diagnosis. Because SEER registries usually collect primary treatment data within 4 months and adjuvant therapy (chemotherapy or hormone therapy) within 6 months [31,32], our main comparisons between the registry data and chart review were analyzed using data collected within 6 months of diagnosis. The analyses were also stratified for patient age, tumor stage, receipt of hormone therapy, year of diagnosis, and whether patient was alive or died at the time of chart review. The multivariable logistic regression was also performed for the risk of being discordant using PROC LOGISTIC. All computer programming and analyses were completed using the SAS system (version 8.2) [33].
3. Results
Table 1 presents comparisons between independent medical chart reviews and existing tumor registry data on chemotherapy for breast cancer that was received within 6 months of diagnosis. Of the 99 cases that received chemotherapy within 6 months of diagnosis according to medical chart reviews, 100% were reviewed from hospital records, 32% had records from at least two different sites, 19% from three sites, and 1% from all four sites (hospital, oncologist's office, radiation department, and other physician's office). The agreement rate between these two databases was 96.0%, with a kappa of 0.72 (0.64–0.79) (Table 1). The sensitivity of the tumor registry data for chemotherapy was 70.7% and the specificity was 98.2%. The positive predictive value of the tumor registry data for chemotherapy was 77.8%. When reanalyzed our data including cases with therapy received after 6 months of diagnosis, 108 subjects aged 65 or older received chemotherapy according to the medical chart reviews. The general agreement between medical chart reviews and the tumor registry data on chemotherapy was 95.4%, with a kappa of 0.69 (0.62–0.77).
Table 1.
Comparison between current medical chart review and the existing tumor registry data in New Mexico on chemotherapy administration that was received within 6 months of diagnosis
| Chemotherapy categories (row %) [column %] | Tumor registry | ||||
|---|---|---|---|---|---|
| No chemotherapy | Chemotherapy received | Total | Agreement rate and kappa (95% CI) | ||
| Chart review | No chemotherapy | 1,109 (98.2) [97.5] | 20 (1.8) [22.2] | 1,129 | 96.0% |
| Chemotherapy received | 29 (29.3) [2.6] | 70 (70.7) [77.8] | 99 | 0.72 (0.64–0.79) | |
| Total | 1,138 | 90 | 1,228 | ||
Table 2 presents comparisons between the medical chart reviews and the existing tumor registry data on adjuvant hormone therapy (mostly tamoxifen) for women with breast cancer in New Mexico. Medical chart reviews captured many more patients who received hormone therapy than did the tumor registry. Compared to the medical chart review data on hormone therapy that was recorded within 6 months of diagnosis, the sensitivity of tumor registry data for hormone therapy was 59.7%, and the specificity was 89.5%. When including those with hormone therapy received after 6 months of diagnosis, 421 (34.3%) of women diagnosed with breast cancer at age 65 or older received hormone therapy according to the medical chart reviews, compared to 322 (26.2%) subjects by the New Mexico Tumor Registry. The general agreement was 79.5% and the kappa was 0.52 (0.47–0.57).
Table 2.
Comparison between current medical chart review and the previous tumor registry in New Mexico on adjuvant hormone therapy that was received within 6 months of diagnosis
| Hormone therapy categories (row %) [column %] | Tumor registry | ||||
|---|---|---|---|---|---|
| No hormone therapy | Hormone therapy received | Total | Agreement rate and kappa (95% CI) | ||
| Chart review | No hormone therapy | 748 (89.5) [82.6] | 88 (10.5) [27.3] | 836 | 80.0% |
| Hormone therapy received | 158 (40.3) [17.4] | 234 (59.7) [72.7] | 392 | 0.52 (0.46–0.57) | |
| Total | 906 | 322 | 1,228 | ||
Tables 3 through 5 present comparisons between two databases on the receipt of chemotherapy and hormone therapy, stratified by patient age, tumor stage, hormone therapy or chemotherapy (according to chart reviews), year of diagnosis, and the sources of referrals from which the information had been obtained. The sensitivity and specificity of tumor registry data for chemotherapy varied by these factors, but generally had high specificity and low sensitivity rates across the various strata (Table 3). For example, sensitivity for women aged 65–69 (88.2%) was higher than those aged 70–74 (53.5%) or those aged 75 or older (77.3%), and was slightly higher in cases who were still alive (72.7%) than cases who died (66.7%) at the time of medical chart review. Sensitivity was also higher in cases with early-stage cancer, diagnosed in earlier years, or having no hormone therapy. Similar variations in sensitivity and specificity for hormone therapy were also observed, but the overall sensitivity and specificity of the registry data for hormone therapy were lower.
Table 3.
Sensitivity and specificity of tumor registry data for chemotherapy and hormone therapy within 6 months of diagnosis compared with medical chart review, stratified by other factors
| Chemotherapy | Hormone Therapy | ||||
|---|---|---|---|---|---|
| Characteristics from SEER registry | Number of patients | Sensitivity (%)a | Specificity (%)b | Sensitivity (%)a | Specificity (%)b |
| All patients | 1,228 | 70.7 | 98.2 | 59.7 | 89.5 |
| Age | |||||
| 65–69 | 305 | 88.2 | 98.2 | 66.0 | 92.1 |
| 70–74 | 365 | 53.5 | 98.1 | 62.2 | 83.3 |
| 75+ | 558 | 77.3 | 98.3 | 54.1 | 92.0 |
| Sources of charts documenting therapy | |||||
| Oncologist's office | |||||
| Yes | 308 | 66.7 | 98.5 | 57.1 | 85.7 |
| No | 920 | 73.0 | 98.1 | 61.2 | 90.4 |
| Radiation Department | |||||
| Yes | 212 | 55.6 | 99.0 | 64.3 | 90.6 |
| No | 1,016 | 74.1 | 98.1 | 58.4 | 89.3 |
| Other physician's office | |||||
| Yes | 251 | 42.9 | 99.2 | 44.7 | 92.2 |
| No | 977 | 75.3 | 98.0 | 63.8 | 88.8 |
| Tumor stage | |||||
| Early stage (I+II) | 961 | 75.0 | 98.6 | 60.7 | 90.6 |
| Late stage (III+IV) | 176 | 69.6 | 95.4 | 57.5 | 84.5 |
| Unstaged (unknown) | 91 | 40.0 | 98.8 | 55.2 | 85.5 |
| Hormone therapyc | |||||
| Yes | 421 | 64.9 | 98.4 | — | — |
| No | 807 | 74.2 | 98.1 | — | — |
| Chemotherapyc | |||||
| Yes | 108 | — | — | 54.8 | 81.8 |
| No | 1,120 | — | — | 60.1 | 90.3 |
| Year of diagnosis | |||||
| 1993–1996 | 513 | 81.0 | 98.4 | 61.3 | 85.4 |
| 1997–1999 | 715 | 68.0 | 98.1 | 58.8 | 92.7 |
| Status at time of review | |||||
| Alive | 867 | 72.7 | 98.6 | 55.4 | 90.0 |
| Died | 361 | 66.7 | 97.3 | 71.4 | 88.3 |
The sensitivity of tumor registry for therapy was defined as the percentage of cases receiving therapy according to medical chart reviews that are identified by tumor registry as receiving therapy.
Specificity was defined as percentage of cases identified by medical records as not receiving therapy that are identified by tumor registry data as also not receiving therapy.
Information on hormone and chemotherapy was from medical chart reviews.
Table 5.
Comparison between tumor registry and medical chart review on hormone therapy that was administered within 6 months of diagnosis: kappa, concordance rate, and adjusted odds ratios of discordance
| Tumor registry compared to medical chart review | ||||
|---|---|---|---|---|
| Characteristics from SEER registry | Number of patients | Simple kappa (95% confidence interval) | % of concordant cases (observed agreement) | Adjusted odds ratio of being discordant (95% confidence interval)a |
| All patients | 1,228 | 0.52 (0.46–0.57) | 80.0 | — |
| Age | ||||
| 65–69 | 305 | 0.61 (0.51–0.70) | 83.9 | 1.00 (reference) |
| 70–74 | 365 | 0.46 (0.16–0.56) | 76.4 | 1.66 (1.12–2.47) |
| 75+ | 558 | 0.50 (0.42–0.58) | 80.5 | 1.40 (0.95–2.05) |
| Sources of charts documenting therapy | ||||
| Oncologist's office | ||||
| Yes | 308 | 0.43 (0.34–0.53) | 82.6 | 1.98 (1.43–2.73) |
| No | 920 | 0.54 (0.47–0.60) | 72.1 | 1.00 (reference) |
| Radiation Department | ||||
| Yes | 212 | 0.57 (0.46–0.68) | 80.2 | 0.89 (0.60–1.32) |
| No | 1,016 | 0.50 (0.44–0.56) | 79.9 | 1.00 (reference) |
| Other physician's office | ||||
| Yes | 251 | 0.54 (0.49–0.60) | 76.1 | 1.27 (0.89–1.80) |
| No | 977 | 0.41 (0.29–0.53) | 81.0 | 1.00 (reference) |
| Tumor stage | ||||
| Early stage (I+II) | 961 | 0.54 (0.48–0.60) | 81.6 | 1.00 (reference) |
| Late stage (III+IV) | 176 | 0.43 (0.30–0.57) | 73.3 | 1.99 (1.31–3.02) |
| Unstaged (unknown) | 91 | 0.42 (0.22–0.62) | 75.8 | 1.92 (1.11–3.29) |
| Chemotherapyb | ||||
| Yes | 108 | 0.37 (0.17–0.56) | 74.1 | 1.21 (0.74–1.99) |
| No | 1,120 | 0.53 (0.48–0.58) | 80.5 | 1.00 (reference) |
| Year of diagnosis | ||||
| 1993–1996 | 513 | 0.47 (0.38–0.55) | 78.8 | 1.00 (reference) |
| 1997–1999 | 715 | 0.55 (0.49–0.62) | 80.8 | 0.70 (0.51–0.95) |
| Status at time of review | ||||
| Alive | 867 | 0.48 (0.42–0.55) | 78.5 | 1.00 (reference) |
| Died | 361 | 0.60 (0.51–0.69) | 83.4 | 0.56 (0.40–0.80) |
Odds ratios were derived from the logistic regression model, adjusted for the variables listed in the table.
Information on chemotherapy was from medical chart reviews.
The kappa statistic for the comparison between two databases on chemotherapy was excellent in patients aged 65–69 (0.85) (Table 4). The observed agreement rates on chemotherapy were generally over 92.0%, except for those subjects with stage III–IV cancer who had a significant greater risk of being discordant. The odds ratio of discordance for chemotherapy between the two databases was also significantly greater in those aged 70–74 than those younger patients (Table 4). Patients aged 70–74 were significantly more likely to have discordant information on hormone therapy between the databases than those aged 65–69, as were those with advanced tumor stages (Table 5). The kappa statistics showed poor to good agreement between the two databases on hormone therapy across various strata. Patients who died at the time of chart review seemed to be at a higher risk of being discordant for information on chemotherapy between the two databases, but were significantly less likely to be discordant for information on hormone therapy (Table 5).
Table 4.
Comparison between tumor registry and medical chart review on chemotherapy that was administered within 6 months of diagnosis: kappa, concordance rate, and adjusted odds ratios of discordance
| Tumor registry compared to medical chart review | ||||
|---|---|---|---|---|
| Characteristics from SEER registry | Number of patients | Simple kappa (95% confidence interval) | % of concordant cases (observed agreement) | Adjusted odds ratio of being discordant (95% confidence interval)a |
| All patients | 1,228 | 0.72 (0.64–0.79) | 96.0 | — |
| Age | ||||
| 65–69 | 305 | 0.85 (0.76–0.95) | 97.0 | 1.00 (reference) |
| 70–74 | 365 | 0.60 (0.46–0.74) | 92.9 | 2.28 (1.03–5.06) |
| 75+ | 558 | 0.70 (0.54–0.85) | 97.5 | 0.66 (0.27–1.60) |
| Sources of charts documenting therapy | ||||
| Oncologist's office | ||||
| Yes | 308 | 0.72 (0.59–0.85) | 94.8 | 1.30 (0.66–2.56) |
| No | 920 | 0.72 (0.62–0.81) | 96.4 | 1.00 (reference) |
| Radiation Department | ||||
| Yes | 212 | 0.64 (0.44–0.85) | 95.3 | 1.34 (0.63–2.83) |
| No | 1,016 | 0.73 (0.65–0.81) | 96.2 | 1.00 (reference) |
| Other physician's office | ||||
| Yes | 251 | 0.53 (0.27–0.78) | 96.0 | 1.15 (0.55–2.41) |
| No | 977 | 0.74 (0.69–0.82) | 96.0 | 1.00 (reference) |
| Tumor stage | ||||
| Early stage (I+II) | 961 | 0.73 (0.63-0.83) | 97.4 | 1.00 (reference) |
| Late stage (III+IV) | 176 | 0.69 (0.56–0.81) | 88.6 | 4.27 (2.17–8.41) |
| Unstaged (unknown) | 91 | 0.48 (0.04–0.91) | 95.6 | 1.49 (0.48–4.62) |
| Hormone therapyb | ||||
| Yes | 421 | 0.69 (0.56–0.82) | 95.5 | 0.94 (0.50–1.77) |
| No | 807 | 0.73 (0.64-0.82) | 96.3 | 1.00 (reference) |
| Year of diagnosis | ||||
| 1993–1996 | 513 | 0.73 (0.58-0.87) | 97.7 | 1.00 (reference) |
| 1997–1999 | 715 | 0.71 (0.63-0.80) | 94.8 | 1.83 (0.90-3.73) |
| Status at time of review | ||||
| Alive | 867 | 0.75 (0.66–0.84) | 96.7 | 1.00 (reference) |
| Died | 361 | 0.66 (0.52–0.80) | 94.5 | 1.46 (0.76–2.79) |
Odds ratios were derived from the logistic regression model, adjusted for the variables listed in the table.
Information on hormone was from medical chart reviews.
4. Discussion
This study addressed the validity of the existing New Mexico SEER Tumor Registry data on chemotherapy and hormone therapy compared to an independent medical chart review. Overall, there was a moderate agreement between the two databases on chemotherapy use. The specificity of tumor registry for chemotherapy was high at 98.2%, but the sensitivity was low at 70.7%. Agreement between the two databases on hormone therapy was poor to moderate, with a sensitivity of 59.7% and specificity of 82.6%.
There are several reasons for the incompleteness of chemotherapy information in the New Mexico SEER tumor registry. First, SEER only requires data on treatments that were initiated within a few months of diagnosis [27]. Adjuvant chemotherapy and hormone therapy are mostly administered after initial primary surgery of breast cancer, so patients may not have been followed up long enough to obtain complete treatment information, particularly on adjuvant hormone therapy, which is often given after a first course of chemotherapy. This could be one of the reasons why the information on cancer surgery or radiation in the SEER registries was more complete, as we and several other investigators demonstrated [15–18]. Second, chemotherapy was often administered at outpatient clinics or at physician's office (usually in medical oncologists' offices). SEER does not require medical oncologists' offices to be checked for chemotherapy administration by the SEER data collectors [26,34,35]. It may be unrealistic for SEER to send registrars to physicians' office to abstract medical records or chemotherapy flow charts on every cancer case, especially in cases when cancer treatment is provided by multiple doctors [26]. However, there appeared to be no significant difference for chemotherapy between cases that were checked at an oncologist's office and those that were not. In contrast, for hormone therapy, discordance between the two databases were more likely when comparing cases that were checked at an oncologist's office with those who were not. Furthermore, although New Mexico residents who were diagnosed and treated at facilities outside the state can be identified through data exchange with surrounding state registries [27,28], recording of this treatment information in the registry could be significantly delayed. If patients received chemotherapy or hormone therapy in other nonsurrounding state medical facilities, these treatments would be likely to be missed in the tumor registry. In a recent study addressing the validity of the California Cancer Registry for breast cancer therapy compared to chart reviews, the agreement was 90% for chemotherapy and 50% for hormone therapy, and kappa was 0.62 (0.50–0.75) and 0.22 (0.16–0.29), respectively [13]. This study had lower agreement rates in comparison to our study, but it included women with breast cancer of all ages rather than just women aged 65 and older.
Quality and completeness of the current independent medical chart reviews are another important issue. Our two data abstractors are experienced and specially trained SEER data collectors. Patients' charts were abstracted at the hospital where the diagnosis was made. In addition, in half of all cases we abstracted charts from at least two different locations such as oncologists' offices or radiology departments. When the chart noted that the patient had been referred to an oncologist, arrangements were made to visit the appropriate facilities. However, not all patients with breast cancer were referred to oncologists, and two previous studies showed that about 50% of women aged 65 or older with breast cancer did not have consultations with medical oncologists after surgery [36,37]. Older women were substantially less likely to be referred to medical oncologists [36–40]. Even so, it is still possible that some medical charts that contained information on chemotherapy and hormone therapy were not located by our reviewers. If patients were deceased, their medical charts were often stored in off-site storage facilities. Retrieval of these charts was possible, but it took time and a fee was involved. Even if the charts were retrieved, frequently they had already been thinned or contained little to no data on the treatment. Although the sensitivity, kappa, and concordance rates for chemotherapy were lower in cases deceased at the time of chart review compared to those who were still alive, the risk of being discordant was not statistically significant. The possibility that medical charts were less likely to be located for the deceased cases was also not supported by the finding that there was a significantly lower risk of being discordant for hormone therapy in these cases (Table 5). Furthermore, the medical records themselves may have failed to record the fact that chemotherapy was administered, particularly in patients who received such a therapy at the out-of-state facilities. Previous studies showed that medical records often do not completely or accurately represent the care rendered to patients or certain aspects of their diseases, and some specific treatments may not be well documented, especially in the outpatient records [41].
Thus, it may be likely that both tumor registry and chart reviews missed cases with chemotherapy, which may be captured by contacting their oncology physicians or interviewing themselves. According to recommendations from several studies on the validity of information on cancer treatment by us and others [14–17,29,42–47], the preferred approach would be to combine the data from the registry, medical chart reviews, Medicare claims, and information from oncology physicians, whenever possible, because these data sources may compliment each other.
Our study has other limitations. First, we only studied women aged 65 or older who were diagnosed with breast cancer. Hence, the findings may not be applicable to younger women with breast cancer. Second, this study was limited to patients diagnosed with breast cancer. It is still unknown how valid the tumor registry data on chemotherapy is for patients diagnosed with other cancers. Finally, the findings in New Mexico may not be generalizable to other SEER registries or other tumor registries in the United States.
In conclusion, there was moderate to excellent agreement on the information for chemotherapy of breast cancer between the New Mexico Tumor Registry and the medical chart reviews. The agreement of the information on hormone therapy was poor to moderate between the two databases. Internal validity of the information on chemotherapy and hormone therapy from the tumor registry was good, which was supported by our recent reports using the New Mexico Tumor Registry data on the findings expected according to patient or tumor characteristics and clinical guidelines [20,46]. However, a possible underreporting should be noted when reporting information on cancer chemotherapy and hormone therapy using the SEER tumor registry data. The preferred approach would be to combine several large databases to obtain more complete information on cancer chemotherapy and hormone therapy.
Acknowledgments
This study was supported by a grant from the National Cancer Institute (R01-CA090626). We thank Anna Davidson, James Goodwin, and Dawn Hamilton for their help in initiating this study in New Mexico, and faculty and staff from the New Mexico Tumor Registry and numerous medical facilities across the state of New Mexico for their cooperation in this study.
References
- 1.Ries LAG, Kosary CL, Hankey BF, Miller BA, Clegg LX, Edwards BK, editors. SEER cancer statistics review 1973–1996 NIH Pub No 99-2789. Bethesda, MD: National Cancer Institute; 1999. [Google Scholar]
- 2.Ries LAG, Eisner MP, Kosary CL, Hankey BF, Miller BA, Clegg L, Mariotto A, Feuer EJ, Edwards BK, editors. SEER cancer statistics review, 1975–2001. Bethesda, MD: National Cancer Institute; 2004. http://seer.cancer.gov/csr/1975_2001/ [Google Scholar]
- 3.Weir HK, Thun MJ, Hankey BF, et al. Annual report to the nation on the status of cancer, 1975–2000, featuring the uses of surveillance data for cancer prevention and control. J Natl Cancer Inst. 2003;95(17):1276–99. doi: 10.1093/jnci/djg040. [DOI] [PubMed] [Google Scholar]
- 4.Lazovich DA, White E, Thomas DB, Moe RE. Underutilization of breast-conserving surgery and radiation therapy among women with stage I or II breast cancer. JAMA. 1991;266:3433–8. [PubMed] [Google Scholar]
- 5.Nattinger AB, Gottlieb MS, Veum J, Yahnke D, Goodwin JS. Geographic variation in the use of breast-conserving treatment for breast cancer. N Engl J Med. 1992;326:1102–7. doi: 10.1056/NEJM199204233261702. [DOI] [PubMed] [Google Scholar]
- 6.Farrow DC, Hunt WC, Samet JM. Geographic variation in the treatment of localized breast. N Engl J Med. 1992;326:1097–101. doi: 10.1056/NEJM199204233261701. [DOI] [PubMed] [Google Scholar]
- 7.Samet J, Hunt WC, Key C, Humble CG, Goodwin JS. Choice of cancer therapy varies with age of patient. JAMA. 1986;255:3385–90. [PubMed] [Google Scholar]
- 8.Lazovich D, White E, Thomas DB, Moe RE, Taplin S. Change in the use of breast-conserving surgery in western Washington after the 1990 NIH Consensus Development Conference. Arch Surg. 1997;132:418–23. doi: 10.1001/archsurg.1997.01430280092014. [DOI] [PubMed] [Google Scholar]
- 9.Du XL, Freeman JL, Goodwin JS. The declining use of axillary dissection in patients with early stage breast cancer. Breast Cancer Res Treat. 1999;53:137–44. doi: 10.1023/a:1006170811237. [DOI] [PubMed] [Google Scholar]
- 10.Du XL, Freeman JL, Freeman DH, Syblik DA, Goodwin JS. Temporal and regional variation in the use of breast conserving surgery and radiotherapy for older women with early stage breast cancer from 1983 to 1995. J Gerontol (Med Sci) 1999;54:474–8. doi: 10.1093/gerona/54.9.m474. [DOI] [PubMed] [Google Scholar]
- 11.Du XL, Freeman DH, Syblik DA. What drove changes in the use of breast conserving surgery for breast cancer since the early 1980s? The role of the clinical trial, celebrity action and an NIH consensus statement. Breast Cancer Res Treat. 2000;62:71–9. doi: 10.1023/a:1006414122201. [DOI] [PubMed] [Google Scholar]
- 12.Du XL, Freeman JL, Nattinger AB, Goodwin JS. Survival of women after breast conserving surgery for early stage breast cancer. Breast Cancer Res Treat. 2002;72:23–31. doi: 10.1023/a:1014908802632. [DOI] [PubMed] [Google Scholar]
- 13.Malin JL, Kahn KL, Adams J, Kwan L, Laouri M, Ganz PA. Validity of cancer registry data for measuring the quality of breast cancer care. J Natl Cancer Inst. 2002;94(11):835–44. doi: 10.1093/jnci/94.11.835. [DOI] [PubMed] [Google Scholar]
- 14.Du XL, Freeman JL, Goodwin JS. Information on radiation treatment in patients with breast cancer: the advantages of the linked Medicare and SEER data. J Clin Epidemiol. 1999;52:463–70. doi: 10.1016/s0895-4356(99)00011-6. [DOI] [PubMed] [Google Scholar]
- 15.Du XL, Freeman JL, Warren JL, et al. Accuracy and completeness of Medicare claims data for surgical treatment of breast cancer. Med Care. 2000;38:719–27. doi: 10.1097/00005650-200007000-00004. [DOI] [PubMed] [Google Scholar]
- 16.Cooper GS, Yuan Z, Stange KC, Dennis LK, Amini SB, Rimm AA. Agreement of Medicare claims and tumor registry data for assessment of cancer-related treatment. Med Care. 2000;38(4):411–21. doi: 10.1097/00005650-200004000-00008. [DOI] [PubMed] [Google Scholar]
- 17.Virnig BA, Warren JL, Cooper GS, Klabunde CN, Schussler N, Freeman J. Studying radiation therapy using SEER-Medicare-linked data. Med Care. 2002;40(8 Suppl):49–54. doi: 10.1097/00005650-200208001-00007. [DOI] [PubMed] [Google Scholar]
- 18.Mariotto A, Feuer EJ, Harlan LC, Wun LM, Johnson KA, Abrams J. Trends in use of adjuvant multi-agent chemotherapy and tamoxifen for breast cancer in the United States: 1975–1999. J Natl Cancer Inst. 2002;94:1626–34. doi: 10.1093/jnci/94.21.1626. [DOI] [PubMed] [Google Scholar]
- 19.Du XL. Re: Trends in use of adjuvant multi-agent chemotherapy and tamoxifen for breast cancer in the United States: 1975–1999. J Natl Cancer Inst. 2003;95(9):683–5. doi: 10.1093/jnci/95.9.683. [DOI] [PubMed] [Google Scholar]
- 20.Du XL, Key CR, Osborne C, Mahnken JD, Goodwin JS. Discrepancy between consensus recommendations and actual community use of adjuvant chemotherapy in women with breast cancer. Ann Intern Med. 2003;138:90–7. doi: 10.7326/0003-4819-138-2-200301210-00009. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Harlan LC, Clegg LX, Warren JL. Chemotherapy in women with breast cancer. Ann Inten Med. 2003;139:868. doi: 10.7326/0003-4819-139-10-200311180-00025. [DOI] [PubMed] [Google Scholar]
- 22.Du XL, Key CR. Chemotherapy in women with breast cancer. Ann Inten Med. 2003;139:869. doi: 10.7326/0003-4819-138-2-200301210-00009. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Harlan LC, Abrams J, Warren JL, Clegg L, Stevens J, Ballard-Barbash R. Adjuvant therapy for breast cancer: practice patterns of community physicians. J Clin Oncol. 2002;20:1809–17. doi: 10.1200/JCO.2002.07.052. [DOI] [PubMed] [Google Scholar]
- 24.Cress RD, Zaslavsky AM, West DW, Wolf RE, Felter MC, Ayanian JZ. Completeness of information on adjuvant therapies for colorectal cancer in population-based cancer registries. Med Care. 2003;41:1006–12. doi: 10.1097/01.MLR.0000083740.12949.88. [DOI] [PubMed] [Google Scholar]
- 25.Bickell NA, Chassin MR. Determining the quality of breast cancer care: do tumor registries measure up? Ann Intern Med. 2000;132(9):705–10. doi: 10.7326/0003-4819-132-9-200005020-00004. [DOI] [PubMed] [Google Scholar]
- 26.Warren JL, Harlan LC. Can cancer registry data be used to study cancer treatment? Med Care. 2003;41:1003–5. doi: 10.1097/01.MLR.0000086827.00805.B5. [DOI] [PubMed] [Google Scholar]
- 27.National Cancer Institute. The SEER program code manual, revised ed NIH Publication No 94-1999. Bethesda, MD: National Cancer Institute; 1994. [Google Scholar]
- 28.The New Mexico Tumor Registry. About the New Mexico Tumor Registry. [February 9, 2004]; Available from URL: http://hsc.unm.edu/epiccpro/nmtr.html.
- 29.Du XL, Key CR, Dickie L, Darling R, Geraci JM, Zhang D. External validation of Medicare claims for breast cancer chemotherapy compared to medical chart reviews. Med Care. doi: 10.1097/01.mlr.0000196978.34283.a6. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Fleiss JL. Statistical methods for rates and proportions. 2nd. New York: John Wiley & Sons; 1981. pp. 212–36. [Google Scholar]
- 31.National Cancer Institute. The SEER program code manual, revised ed NIH Publication No 94-1999. Bethesda, MD: National Cancer Institute; 1994. [Google Scholar]
- 32.Warren JL, Harlan LC, Fahey A, et al. Utility of the SEER-Medicare data to identify chemotherapy use. Med Care. 2002;40(Suppl):IV-55–61. doi: 10.1097/01.MLR.0000020944.17670.D7. [DOI] [PubMed] [Google Scholar]
- 33.Stokes ME, Davis CS, Koch GG. Categorical data analysis using the SAS System. Cary, NC: SAS Institute Inc.; 1997. [Google Scholar]
- 34.Potosky AL, Riley GF, Lubitz JD, Mentnech RM, Kessler LG. Potential for cancer related health services research using a linked Medicare-tumor registry database. Med Care. 1993;31:732–48. [PubMed] [Google Scholar]
- 35.Warren JL, Klabunde CN, Schrag D, Bach PB, Riley GF. Overview of the SEER-Medicare data: content, research applications, and generalizability to the United States elderly population. Med Care. 2002;40(8 Suppl):3–18. doi: 10.1097/01.MLR.0000020942.47004.03. [DOI] [PubMed] [Google Scholar]
- 36.Newcomb PA, Carbone PP. Cancer treatment and age: patient perspectives. J Natl Cancer Inst. 1993;85:1580–4. doi: 10.1093/jnci/85.19.1580. [DOI] [PubMed] [Google Scholar]
- 37.Hillner BE, McDonald MK, Penberthy L, et al. Measuring standards of care for early breast cancer in an insured population. J Clin Oncol. 1997;15:1401–8. doi: 10.1200/JCO.1997.15.4.1401. [DOI] [PubMed] [Google Scholar]
- 38.Siminoff LA, Zhang A, Saunders Sturm CM, Colabianchi N. Referral of breast cancer patients to medical oncologists after initial surgical management. Med Care. 2000;38:696–704. doi: 10.1097/00005650-200007000-00002. [DOI] [PubMed] [Google Scholar]
- 39.Keating NL, Landrum MB, Ayanian JZ, Winer EP, Guadagnoli E. Consultation with a medical oncologist before surgery and type of surgery among elderly women with early-stage breast cancer. J Clin Oncol. 2003;21:4532–9. doi: 10.1200/JCO.2003.05.131. [DOI] [PubMed] [Google Scholar]
- 40.Baldwin LM, Taplin SH, Friedman H, Moe R. Access to multidisciplinary cancer care: is it linked to the use of breast-conserving surgery with radiation for early-stage breast carcinoma? Cancer. 2004;100:701–9. doi: 10.1002/cncr.20030. [DOI] [PubMed] [Google Scholar]
- 41.Iezzoni LI. Data sources and implications: information from medical records and patients. In: Iezzoni LI, editor. Risk adjustment for measuring healthcare outcomes. Chicago, IL: Health Administration Press; 1997. pp. 243–78. [Google Scholar]
- 42.Du XL, Goodwin JS. Patterns of use of chemotherapy for breast cancer in older women: findings from Medicare claims data. J Clin Oncol. 2001;19:1455–61. doi: 10.1200/JCO.2001.19.5.1455. [DOI] [PubMed] [Google Scholar]
- 43.Du XL, Goodwin JS. Increase of chemotherapy use in older women with breast carcinoma from 1991 to 1996. Cancer. 2001;92:730–7. doi: 10.1002/1097-0142(20010815)92:4<730::aid-cncr1376>3.0.co;2-p. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Du XL, Osborne C, Goodwin JS. Population-based assessment of hospitalizations for toxicity from chemotherapy in older women with breast cancer. J Clin Oncol. 2002;20:4636–42. doi: 10.1200/JCO.2002.05.088. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Du XL, Jones DV, Zhang D. Effectiveness of adjuvant chemotherapy for node-positive operable breast cancer in older women. J Gerontol (Med Sci) 2005;60:1137–44. doi: 10.1093/gerona/60.9.1137. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Du XL, Key CR, Osborne C. Community-based assessment of adjuvant hormone therapy in women with breast cancer in 1991 through 1997. Breast J. 2004;10:433–9. doi: 10.1111/j.1075-122X.2004.21357.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Du XL, Chan W, Giordano S, Geraci JM, Delclos GL, Burau K, Fang S. Variation in modes of chemotherapy administration for breast cancer and association with hospitalization for chemotherapy-related toxicity. Cancer. 2005;104:913–24. doi: 10.1002/cncr.21271. [DOI] [PMC free article] [PubMed] [Google Scholar]