Quantifying Limitations in Chemotherapy Data in Administrative Health Databases: Implications for Measuring the Quality of Colorectal Cancer Care

Abstract

Reliable chemotherapy data are critical to evaluate the quality of care for patients with colorectal cancer who are treated with curative intent. In Canada, limitations in the availability and completeness of chemotherapy data exist in many administrative health databases. In this paper, we discuss these limitations and present findings from a chart review in Nova Scotia that quantifies the completeness of chemotherapy capture in existing databases. The results demonstrate that even basic information on cancer treatment in administrative databases can be insufficient to perform the types of analyses that most decision-makers require for quality-of-care measurement.

Colorectal cancer (CRC) is the second most prevalent cancer among men and women in Canada, reflecting both high incidence and relatively favourable (63%) five-year survival rates (Canadian Cancer Society 2011). There is evidence that access to timely and appropriate health services along the CRC disease trajectory results in improved outcomes (Berrino et al. 2007; Cheung et al. 2009; Gatta et al. 2000; Hershman et al. 2006). The Canadian Institutes of Health Research/Cancer Care Nova Scotia Team in Access to Colorectal Cancer Services is using linked administrative data to measure access, quality, inequity and outcomes related to CRC care in Nova Scotia, focusing on transition points along the care continuum (Urquhart and Grunfeld 2010).

Invasive CRC includes stage I, II, III and IV disease, with lower-stage disease having much higher probability of long-term survival (and cure). Stage IV disease involves spread of the cancer to tissues or organs outside the primary disease site, and is associated with poor five-year relative survival (National Cancer Institute 2011). When the goal of treatment is curative, as is normally the case for stages I to III, clinical practice guidelines (CPGs) consistently recommend that adjuvant chemotherapy be considered for all patients with stage II/III rectal and stage III colon cancer following surgical resection of disease based on evidence of chemotherapy benefit (i.e., improved survival) in these populations (Figueredo et al. 1997, 2003; National Institutes of Health 1990). Accurate and reliable chemotherapy data are therefore critical to evaluate quality of care appropriately for patients with CRC treated with curative intent (Urquhart and Grunfeld 2009).

In the United States, researchers have demonstrated low sensitivity of chemotherapy data in cancer registries (Bickell and Chassin 2000; Malin et al. 2002). Limitations in the availability and completeness of chemotherapy data exist throughout Canada (Canadian Partnership Against Cancer 2009) and include difficulties in obtaining population-based chemotherapy as well as the absence of important clinical information (e.g., dose reductions, reasons for declining therapy) (Rayson et al. 2010). The completeness of chemotherapy data in Canadian administrative health databases has not been established.

In this paper, we discuss the limitations surrounding chemotherapy data capture for population-based cancer health services research and surveillance initiatives, and examine whether patients for whom CPGs recommend adjuvant chemotherapy actually received chemotherapy drugs when there was no indication of receipt in their administrative data files. As Canadian organizations begin reporting on chemotherapy measures using administrative data (Canadian Partnership Against Cancer 2010; Cancer Care Ontario 2009), we aim to provide empirical data on the completeness of these databases in one province to (a) improve our understanding of the potential strengths and limitations of measuring chemotherapy utilization from administrative data and (b) enhance our capacity to improve data capture in the future.

Methods

Administrative data

The Nova Scotia Cancer Registry (NSCR) identified all individuals diagnosed with CRC in Nova Scotia between January 1, 2001 and December 31, 2005. This population-based cohort was staged through chart review using the Collaborative Stage Data Collection System (Collaborative Stage Work Group 2010) and anonymously linked at the patient level to 14 administrative health databases via a unique study ID (Urquhart and Grunfeld 2010). The databases included Oncology Patient Information System (OPIS), Medical Services Insurance Physician Services (MSIPS) database, Canadian Institute for Health Information (CIHI) Discharge Abstracts Database, Seniors Pharmacare, Palliative Care Program database, Vital Statistics and Canadian census data. This linkage provided a comprehensive data source to examine health services utilization across the continuum of care. To assess chemotherapy receipt, we used linked data from OPIS, the administrative/scheduling database at both of the tertiary cancer centres in Nova Scotia, and MSIPS, which contains billings data for fee-for-service physician services and shadow billings for alternatively paid physicians. After examining these data, we observed that a large proportion of patients who met CPG recommendations for chemotherapy did not appear to have received it, and that large variations in chemotherapy receipt existed across regions (Rayson et al. 2010). These observations led to concerns regarding completeness of chemotherapy capture in Nova Scotia—based databases.

Chart review

To quantify the completeness of chemotherapy capture, we conducted an extensive, provincewide chart review to determine whether patients received adjuvant chemotherapy within six months of surgical resection despite no indication of receipt in administrative files. We identified all patients with curatively resected stage II/III rectal and stage IIB/III colon cancer in the cohort who (a) received a medical oncology consultation and (b) did not receive chemotherapy according to the linked administrative data. Patients with stage IIB colon cancer were also included because adjuvant chemotherapy is increasingly recommended for this group owing to the high risk of recurrence (BC Cancer Agency 2010; Jonker et al. 2008). Patients with stage IV cancer were excluded because CPGs do not recommend adjuvant chemotherapy for these patients, as it provides no long-term survival benefit. We did not review charts of patients for whom the administrative data indicated chemotherapy receipt. An authorized individual at the NSCR, which held the unique study ID key, identified the charts for the review and sent a chart list directly to health records departments at the applicable institutions. One experienced chart abstractor (a nurse specializing in gastroenterology) reviewed the charts. Both the administrative data linkages and chart review were approved by the relevant institutional ethics review boards. Descriptive data were analyzed using SAS v.8. A chi-square analysis was performed to test differences in completeness of administrative data by place of residence.

Results

Between January 1, 2001 and December 31, 2005, 3,501 patients were diagnosed with CRC. Charts were reviewed for 371 patients. Of these, 137 (36.9%) received adjuvant chemotherapy within six months of surgery, despite no indication of receipt in the linked administrative data. A medical oncologist located at one of the province's two cancer centres prescribed the chemotherapy for 136 (99.3%) of these patients. The completeness of the administrative data differed based on patient residence (p<0.0001): for patients residing in a health district with a tertiary cancer centre, 21 of the 143 charts reviewed (14.7%) indicated chemotherapy receipt; for patients residing in a health district without a cancer centre, 116 of the 228 charts (50.1%) indicated chemotherapy receipt. The rate of guideline-concordant chemotherapy receipt for this population-based cohort, using the combined administrative and chart review data, is reported elsewhere (Rayson et al. 2011).

Discussion

Large administrative health databases have the unique potential to provide robust population-based measures of quality across the care continuum for specific conditions (Grunfeld et al. 2006). Despite linking 14 administrative databases to the cohort, however, we were unable to reliably capture many CRC-related adjuvant chemotherapy administrations. The results of our chart review confirmed that chemotherapy data are incomplete in administrative databases in Nova Scotia: altogether, there was a 36.9% underestimate of chemotherapy receipt from these data sources. The underestimate was greater for health districts without a tertiary cancer centre than for those with one (50.1% vs. 14.7%; p<0.0001).

We hypothesize one reason for our findings is that patients are choosing, for accessibility or other reasons, to receive chemotherapy in outpatient settings at community hospitals, where chemotherapy drugs may be administered by nursing staff without a physician billing to record the visit/procedure (i.e., without an associated claim in MSIPS). The difference we observed by place of residence may be due to patients in districts with a cancer centre being more likely to receive chemotherapy at the nearby centre (with the administration data then reliably captured in OPIS), and patients residing in districts without cancer centres more likely to receive chemotherapy in outpatient settings at their local hospitals. This may be particularly true for CRC: the chemotherapy regimen is of relatively low intensity, is generally well tolerated by patients and therefore is more likely to be provided in community hospitals. Another possible explanation for our findings is incomplete “shadow billings”: chemotherapy administered by an alternatively paid physician may not be captured in the MSIPS database if the physician did not reliably submit shadow billings to the provincial health insurance program. However, this explanation is unlikely to account for the large underestimate we observed in health districts lacking tertiary cancer centres, because few physicians in those districts were participating in alternative funding plans during our study period.

Importantly, these findings demonstrate that relying on cancer registries and other administrative databases can be inadequate for population-based assessments of adjuvant chemotherapy for CRC patients. In its 2010 System Performance Report, the Canadian Partnership Against Cancer measured indicators related to CRC screening, diagnosis and treatment, including CPG-concordant adjuvant chemotherapy rates for resected stage III colon cancer patients. While this indicator was reported for five provinces (British Columbia, Alberta, Manitoba, Ontario and Newfoundland and Labrador), data limitations were noted: for example, the data did not include all forms of chemotherapy (Ontario) or were not population-based (British Columbia). Moreover, although Nova Scotia reported on a “proxy” measure for this indicator, the data included only patients residing in the two health districts with tertiary cancer centres. In this study, we demonstrated that even this assessment can lead to an underestimation in receipt of chemotherapy of 14.7%. Such limitations in chemotherapy capture need to be considered as efforts to develop pan-Canadian cancer control indicators move towards collecting and reporting data on systemic therapy utilization rates. The extent of the problem for other types of cancer is unknown.

Few studies have reported population-based chemotherapy data, possibly because of the difficulty associated with obtaining these data for entire jurisdictions. Numerous provincial cancer agencies have recently undertaken considerable effort to improve chemotherapy data capture. Cancer Care Ontario (CCO), for example, uses CIHI's National Ambulatory Care Reporting System to capture data on ambulatory chemotherapy services in Ontario, and has implemented a computerized physician order entry system to capture detailed chemotherapy data not available through other administrative databases. Despite these efforts, however, it remains difficult to report chemotherapy receipt for the entire population: CCO's Cancer System Quality Index reports CPG-concordant chemotherapy receipt for resected stage III colon cancer patients, yet, owing to limitations in data capture and reporting, the index restricts the analysis to patients referred to a cancer centre and for whom valid stage data were available in the CCO databases, resulting in only 38% of incident cases in Ontario being included in the most recent analysis (Cancer Care Ontario 2009). According to our analysis, this approach may still result in a 15% underestimate of chemotherapy receipt. In Alberta, researchers recently reported population-based data on receipt of CPG-recommended therapy for stage III colon cancer (Winget et al. 2010). Population-based data were obtained by first extracting treatment information from the Alberta Cancer Registry and then from the cancer electronic medical record (EMR) when adjuvant chemotherapy data were missing or incomplete in the registry. The extent to which chemotherapy data were missing in the cancer registry was not reported.

The capture of chemotherapy administrations is not the only limitation of current administrative databases. The absence of specific adjuvant therapy data in Nova Scotia databases — such as number of cycles received, schedule/frequency, dose reductions and reasons for not recommending or for declining chemotherapy — negatively affect the assessment of chemotherapy uptake and the evaluation of adherence to CPGs for the administration of systemic therapy for cancer. Accordingly, given current data capabilities, researchers and surveillance programs in many provinces may be limited in their ability to undertake population-based analyses of access and quality in systemic therapy. The application of new information systems, such as CCO's computerized physician order entry system, has the potential to greatly improve efforts to assess quality of systemic therapy by providing data that are not available through other sources. The widespread implementation of EMRs should also enhance our capacity to undertake such analyses in future studies.

There are several limitations regarding our data. First, we reviewed charts only for patients who received a medical oncology consultation but did not receive chemotherapy, according to our linked data. We did not review charts for patients who did not receive a medical oncology consultation and did not receive chemotherapy, because knowledge of local clinical practice led to the assumption that patients who did not see a medical oncologist were unlikely to receive chemotherapy. Nonetheless, we recognize that a small number of patients may have received chemotherapy without a medical oncologist consultation. Second, given the administrative data holdings, we were confident that a procedure code for chemotherapy administration was a reliable indicator of chemotherapy receipt. This assumption is supported by others, who have demonstrated high specificity of chemotherapy data in administrative databases (Bickell and Chassin 2000; Malin et al. 2002). However, we cannot comment on whether the recommended number of chemotherapy cycles was completed nor whether dose reductions occurred. Third, the chart review selection criteria preclude our ability to compute sensitivity and specificity statistics with regard to chemotherapy in administrative databases and limit the generalizability of our findings to patients outside our sample (e.g., patients without a medical oncology consultation). Another limitation of this study is the age of the data: this study was conducted on individuals who were diagnosed from 2001 to 2005, with data capture for adjuvant chemotherapy extending into December 2006. However, since then, few changes have occurred in the specific administrative data sources used in Nova Scotia to acquire chemotherapy data.

The limitations of administrative databases are also relevant to this study. Although these databases have been demonstrated to be valid and reliable for measuring health services utilization in Canada (CIHI 2010; Grunfeld et al. 2006; Wyse et al. 2011), the data are collected for administrative reasons and not for research or surveillance purposes. One potentially relevant limitation involves shadow billing. While Nova Scotia physicians participating in alternative funding plans are contractually mandated to shadow bill, during the period of this study, there was no mechanism in place to regularly monitor the completeness, accuracy and reliability of shadow billing, nor were there incentives or disincentives for physicians with respect to shadow billing (North South Group 2005). Additional limitations of administrative data include quality of documentation in the physician note (So et al. 2010), reliability of coding across individuals and institutions, and administrative errors during coding or data cleaning/management.

Conclusion

Population-based chemotherapy data are critical to measure and monitor the quality of care provided to patients with CRC, particularly to examine issues related to time intervals for treatment and adherence to CPGs. By determining whether patients actually received chemotherapy when there was no indication of receipt in their administrative files, this study demonstrates that even basic information on cancer treatment in administrative databases can be insufficient to perform the types of analyses that most decision-makers require for quality-of-care measurement. The intensive resource requirements for undertaking population-based chart reviews underscore the need for systematic improvements in data capture mechanisms and reporting practices in order to advance our ability to measure, evaluate and improve quality in cancer care. Such improvement is particularly imperative as national and provincial organizations push to develop a pan-Canadian performance reporting initiative.