Towards using administrative databases to measure population-based indicators of quality of end-of-life care: testing the methodology

Abstract

This study is concerned with methods to measure population-based indicators of quality end-of-life care. Using a retrospective cohort approach, we assessed the feasibility, validity and reliability of using administrative databases to measure quality indicators of end-of-life care in two Canadian provinces. The study sample consisted of all females who died of breast cancer between 1 January 1998 and 31 December 2002, in Nova Scotia or Ontario, Canada. From an initial list of 19 quality indicators selected from the literature, seven were determined to be fully measurable in both provinces. An additional seven indicators in one province and three in the other province were partially measurable. Tests comparing administrative and chart data show a high level of agreement with inter-rater reliability, confirming consistency in the chart abstraction process. Using administrative data is an efficient, population-based method to monitor quality of care which can compliment other methods, such as qualitative and purposefully collected clinical data.

Introduction

This study is concerned with methods to measure population-based indicators of quality end-of-life care. Defining, measuring and monitoring quality is a facet of health services research that is growing in importance.^1,2 Much has been done towards developing a methodology to measure quality of care,^2,3 quality of care for cancer patients,^4,5 and, in particular, quality of care for breast cancer patients.^6–8 However, developments in measuring quality end-of-life care have been slower than other phases of the cancer care continuum.^9,10

Quality of care has been defined as the ‘degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge’.¹¹ The challenge remains, however, to define key indicators of quality of care, to link those indicators with health outcomes,^12–16 and to establish reliable, valid, and timely measurements of the indicators.⁷

Quality indicators are tools designed to measure quality of care,^13–15,17 and, thus, help enhance quality,¹⁵ by identifying areas needing improvement.¹⁸ They have been defined as: ‘a measurable element of practice performance for which there is evidence or consensus that it can be used to assess quality and hence change the quality of care provided’.¹⁵ Quality indicators can measure any step along the process-to-outcome continuum.¹⁹ Kahn et al.,^4,7,8 and Lorenz et al.,²⁰ have conducted extensive reviews on quality of care measurement in cancer. There are a number of considerations in developing quality indicators, including acceptability, feasibility, validity and reliability.¹² In this study on end-of-life quality indicators, we focus on feasibility, validity and reliability. We consider acceptability in a separate study.²¹

Administrative health databases have the unique potential of providing population-based, unbiased, efficient measures of quality of care,^22,23 especially in countries where population-based data are available.²⁴ In order to exploit this valuable resource towards the important goal of population-based measures of quality end-of-life care,^18,24,25 it is important to determine which indicators can be measured through administrative databases and which cannot be measured.^9,26 This study aimed to begin the process of determining which indicators can and cannot be measured by studying administrative databases for two provinces in Canada.

The overall objective of this study was to assess the feasibility of using routinely collected administrative data to measure population-based indicators of quality end-of-life care. Adapting from Hofer et al.,¹⁹ our specific objectives were to: (1) classify candidate quality indicators for end-of-life cancer care; (2) establish the feasibility of measuring them through two population-based administrative databases in two provinces; and (3) establish their content validity and reliability through detailed chart review.

Methods

Study design and subjects

We conducted a retrospective cohort study, selecting all females who died of breast cancer between 1 January 1998 and 31 December 2002 in Nova Scotia (NS) or Ontario, Canada. The end-of-life time period was defined as the last six months of life, and was measured by working backwards from the time of death.^24,27,28 A retrospective approach is an efficient tool to monitor end-of-life care across jurisdictions, demographic groups and time periods.²⁹

In order to select females who died of breast cancer over the five-year study period, the cause of death is an important piece of information. Administrative databases are particularly strong tools for this first step in the analysis.³⁰ To select those whose cause of death was breast cancer, information was obtained from Vital Statistics files, which, in Canada, are administered by the provinces using national standards with registration required under various provincial Statistics Acts. ‘Due to legal reporting requirements, registration of births, deaths, stillborns and marriages is considered to be virtually complete’ (p xiii).³¹

From the Vital Statistics files linked to the cancer registry in each province, two inclusion criteria were used to identify the study subjects. First, using the diagnostic codes from the Vital Statistics files, we selected those females whose cause of death was listed as breast cancer, and whose cancer diagnosis was known prior to their date of death. As a second criterion, we included those with a cause of death listed as disseminated disease, and who had a previous breast primary diagnosis. Thus, we excluded females who had breast cancer, but died of other causes, or died at an early stage of cancer from treatment effects.³² By using these criteria, we included only females whose death was due to advanced breast cancer, giving us a homogeneous group of subjects.

Ethical approval for the study was obtained from the Capital District Health Authority, NS, and the Ottawa Hospital Research Ethics Board, Ontario. Confidentiality was maintained according to the policies of the Population Health Research Unit (PHRU) in NS,³³ and the Institute for Clinical Evaluative Sciences (ICES) in Ontario.³⁴

Quality indicators and analyses

Feasibility

Following Earle et al.,²³ a preliminary list of quality of care indicators were selected after a thorough literature review. From this list of 71 candidate quality indicators, an expert panel determined that it was potentially feasible to measure 19 using administrative databases currently available in the two provinces and linkable to the study sample. The panel consisted of experienced end-of-life researchers from various disciplines including oncologists, palliative care physicians, epidemiologists, data analysts, health system managers and a statistician. All 19 indicators are shown in Table 1, with a description of all calculated statistics in Table 2. Each indicator has been assigned a number and, for clarity, the same number is used for that indicator throughout this paper. It was not the aim of this study to develop a comprehensive list of indicators, but rather to test the feasibility of measuring indicators from administrative databases. From this list of 19, we compared data needed to calculate the individual indicator statistics to data fields available in the provincial administrative databases, to determine which indicators are currently measurable.

Table 1.

Classification of quality indicators according to measurability using administrative databases

Measurable quality indicators
Nova Scotia	Ontario
1. Interval between last chemotherapy and death	1. Interval between last chemotherapy and death
2. Site of death	2. Site of death
3. Frequency of ER visits	3. Frequency of ER visits
4.1. Hospital days near the end of life	4.1. Hospital days near the end of life
4.2. ICU days near the end of life	4.2. ICU days near the end of life
5. Continuity of care	5. Continuity of care
6. Time and location of care	6. Time and location of care
7. Adverse events	7. Adverse events
Partially measurable quality indicators
8. Enrollment in palliative care near death
9. Access to palliative care
10. Radiotherapy for uncontrolled bone pain for bony metastases
11. Potent antiemetic for emetogenic chemotherapy	11. Potent antiemetic for emetogenic chemotherapy
12. Multidisciplinary care	12. Multidisciplinary care
13. Access to care	13. Access to care
14. Interval between new chemotherapy and death
Not currently measurable quality indicators
	8. Enrollment in palliative care near death
	9. Access to palliative care
	10. Radiotherapy for uncontrolled bone pain for bony metastases
	14. Interval between new chemotherapy and death
15. Community-centred services	15. Community-centred services
16. Reimbursement	16. Reimbursement
17. Advance care directives	17. Advance care directives
18. Assessment of financial/care-giving resources	18. Assessment of financial/care-giving resources
19. Regular palliative care	19. Regular palliative care

Table 2.

Description and calculation of quality indicators

Quality indicator	Description and calculation
1. Interval between last chemotherapy and death	Short interval may indicate poor quality care. Calculations: No. of cases receiving chemotherapy in last 14 days divided by all cases receiving chemotherapy in the last six months Days between the last chemotherapy and death averaged across all cases receiving chemotherapy in the last six months No. of cases receiving chemotherapy in last six months divided by entire cohort No. of cases receiving chemotherapy in last 14 days divided by entire cohort
2. Site of death	Death in hospital rather than at home may indicate poor quality care. Calculation: No. of cases dying in hospital divided by all cases in cohort
3. Frequency of emergency room (ER) visits	High number of emergency room visits near death may indicate poor quality care. Calculations: No. of cases with more than 1 ER visit in the last 30 days divided by entire cohort No. of ER visits in the last 30 days averaged across entire cohort Sum of ER visits across entire cohort divided by the sum of available days (ie, days out of hospital) across entire cohort
4.1 Hospital days near the end of life (includes any inpatient days)	Hospital stays in the terminal period of cancer may indicate poor quality care. Calculations: No. of cases with more than one hospital admission in the last 30 days divided by entire cohort No. of cases with more than 14 days in the hospital in the last 30 days divided by entire cohort Inpatient days in the last 30 days averaged across entire cohort Hospital admissions in the last 30 days averaged across entire cohort
4.2 Hospital days near the end of life (includes Intensive Care Unit (ICU) days only)	Hospital stays in the terminal period of cancer may indicate poor quality care. Calculations: No. of cases with one or more ICU admissions in the last 30 days divided by entire cohort
5. Continuity of care provided by General Practitioners (GPs) and non-GPs	Co-ordination of care by one provider and co-ordination of records may indicate good quality care. Calculations: Modified Modified Continuity Index (MMCI)45=1−(No. of ambulatory providers/[No. of ambulatory visits+/0.1])1(1/[No. of ambulatory visits+/0.1]) MMCI for GPs only – for those with > =3 GP visits MMCI for non-GPs only – for those with > =3 non-GP visits GP visits in last six months averaged across all cases with at least three GP visits Total number of GPs in last six months averaged across all cases with at least three GP visits Non-GP visits in last six months averaged across all cases with at least three non-GP visits Total number of non-GPs in last six months averaged across all cases with at least three non-GP visits
6. Time and location of care	The ideal is assumed to be clinician visits that are more frequent as death approaches. Calculation: sum of home and office visits for the entire cohort in last two weeks divided by the sum of home and office visits for the entire cohort in last six months
7. Adverse events	High proportion of adverse events may indicate poor control of symptoms and poor quality care. Examples may include falls, avoidable infections, bedsores or injuries. Calculation: number of cases who experienced a bedsore, infection, fall, or injury in the last six months divided by the entire cohort
8. Enrollment in palliative care near death	A high proportion may indicate poor quality care. Palliative care should be accessible to all patients and families with a cancer diagnosis, in a timely manner, throughout the entire duration of their disease. Calculation: No. of cases enrolled in palliative care within three days prior to death divided by all cases enrolled in palliative care
9. Access to palliative care	Palliative care will be accessible to all patients and families with a cancer diagnosis, in a timely manner, throughout the entire duration of their disease. A high proportion may indicate good quality care. Calculation: No. of cases enrolled in palliative care divided by all cases
10. Radiotherapy for uncontrolled bone pain for bony metastases	Providing patients with radiotherapy for bony metastases improves pain management and improves quality of life. A high proportion may indicate good quality care. Calculation: No. of cases who received palliative radiation to the bones divided by the entire cohort (palliative is defined as having an intent code listed as palliative or a dose level less than or equal to 3000 cGy)
11. Potent antiemetic for emetogenic chemotherapy	Potent anti-emetic therapy for highly emetogenic chemotherapy treatments greatly controls nausea and vomiting. A high proportion may indicate good quality care. Calculation: No. of cases 65 years of age and older who had a prescription for a potent antiemetic divided by all cases 65 years of age and older
12. Multidisciplinary care	Multidisciplinary care is considered an element of quality cancer care as it provides patients with access to practitioners with different perspectives and skill sets in order to achieve holistic care. Examples may include nursing services, social work, specialised medical services. A high proportion may indicate poor quality care. Calculations: Sum of GP visits for the entire cohort divided by the sum of physician visits for the entire cohort Sum of non-GP visits for the entire cohort divided by the sum of physician visits for the entire cohort
13. Access to care	Fair and equitable access to care for patient and family, regardless of financial considerations, indicates good quality care. A high proportion may indicate poor quality care. Calculation: No. of cases living in a rural area divided by entire cohort
14. Interval between new chemotherapy and death	Short interval between new chemotherapy regimen and death may indicate poor quality care. Calculation: No. of cases starting a new chemotherapy regimen in the last 30 days divided by all cases

Validity and reliability

Face validity,¹⁵ of these indicators has been established by the preliminary work of Earle et al.,²³ and Teno et al.²⁶ To assess content validity and reliability, we obtained a random sample of the study subjects who received their care at a tertiary cancer treatment facility and its host hospital in each province: the QEII Health Sciences Centre in NS – the sole tertiary care treatment facility in the province, serving a population of 950 thousand; and the Ottawa Hospital in Ontario – the tertiary care treatment facility, serving a population of 1.5 million in Eastern Ontario. Electronic data abstraction forms were developed in MS Access and pilot tested. The patient charts of this random sample were abstracted by staff trained and contracted for this study. Content validity was assessed by determining whether, for each of the indicators, the data fields in the administrative databases were complete and accurate using patient charts as the gold standard.

The Kappa statistic was used to measure agreement for categorical variables. This method provides a measure of agreement which accounts for agreement that could occur due to chance.^35,36 There is little purpose in using a measure with a Kappa score of <0.4, while a score of 0.75 or higher represents excellent agreement.

The intraclass correlation coefficient (ICC) was calculated to measure agreement for continuous variables.³⁷ This statistic avoids arbitrary grouping of information into categories, and utilises all information from a continuous variable. The ICC measures three components of variance: the between-subject variance, the between-data source variance, and the residual variance. Unlike the Kappa statistic, the ICC has no generally accepted thresholds for acceptable agreement. As noted by Saito et al.³⁸ ‘However, in reliability studies, there is no consensus on the value of permissible ICC’ (p 1548). Nonetheless, calculated ICC statistics can be used as a relative measure of agreement.

Statistics for each indicator were calculated for both provinces. As the objective of the study was to test the use of administrative databases, the focus here is not on analysing the results or the determinants of any particular indicator. Rather, results are presented to check the plausibility of the calculated statistics. The required data may be present in the databases, but when computed, the indicator results may be so extreme as to be implausible. Thus, point estimate statistics are presented and were reviewed in each province as a further measure of validity.

Reliability was assessed by blinded, duplicate abstraction of charts, and inter-rater reliability was measured.^15,19

Data sources

For subject selection, all females who died of breast cancer between 1 January 1998 and 31 December 2002 were identified from the two provincial cancer registries, which link cancer cases to the provincial Vital Statistics files.

Probabilistic linkage using personal identifier information, such as health card number, name, date of birth and address, was used to link females who died of breast cancer to the provincial cancer registry in each province. Anonymised identifiers then allowed for deterministic linking to additional databases to calculate the statistics representing the indicators. In NS, the databases were: physician claims data, hospital discharge abstracts, seniors’ pharmacare, Oncology Patient Information System (OPIS), and palliative care program database. In Ontario, the databases were: physician claims data, hospital discharge abstracts, seniors’ pharmacare, and National Ambulatory Care Reporting System. All databases were linked at the individual person level. Analysis was performed using the statistical software package SAS.

Patient chart information for the random sample of patients in each province was abstracted for the purposes of validation, as described above. Abstracted hospital and cancer centre chart data were then linked to the linked administrative databases for the randomly selected individuals and compared.

Results

In terms of population size, Ontario is the largest of 10 provinces in Canada, while NS is seventh; Ontario contains a smaller proportion of the population >65 years of age (12.8 versus 14.2%).³⁹ Economically, Ontario has the second largest Gross Domestic Product per capita, while NS ranks ninth.⁴⁰

Using our selection criteria, 864 females died of breast cancer in NS, and 9261 in Ontario, over the five-year study period. The mean age at death was 68.2 and 67.5 years in NS and Ontario, respectively.

Using our list of 19 potentially measurable indicators, the various available databases were checked for information that would enable us to calculate statistics for each indicator. Table 1 outlines the indicators, which we group as measurable, partially measurable and not currently measurable. Seven indicators are fully measurable in both provinces.

An additional seven in NS and three in Ontario are partially measurable (Table 1). A partially measurable result can have a variety of meanings in this context. For example, enrollment in palliative care programs and prescriptions for potent antiemetics for emetogenic chemotherapy can be measured, but information is only available for a subset of the population. Whether radiation therapy to the bones was administered is available, but we do not have an indication of whether an individual had bone pain. Access to care is difficult to measure directly with administrative databases, but previous literature,^41,42 has indicated that rural residents may not have as many services available to them as urban residents. As such, we can calculate the proportion of our study subjects who are rural verses urban.

Finally, Table 1 shows those indicators which are not currently measurable through available databases that can be linked to our cohort at an individual level. In Ontario, there are nine such indicators, and five in NS.

Table 2 gives a detailed description of the statistics used for each indicator. Six of the indicators (indicators 1–4, 8 and 14) are based directly on statistics used by Earle et al.⁴³ Five (indicators 7, 9–11 and 13) are simple proportions showing prevalence across the population. Indicators 6 and 12 are described in Johnston et al.⁴⁴ Finally, continuity of care (indicator 5) is represented by the Modified Modified Continuity Index (MMCI),⁴⁵ which is shown together with its components. We calculated a separate MMCI for general practitioners and specialists, as the end-of-life care provided by each of these groups is likely to be very distinct.

Validity

Tables 3 and 4 show the agreement between administrative data and chart abstraction data for each indicator for a random sample of study subjects. Most measures show a match of >80% for both provinces. Since the actual agreement percentage does not account for chance agreement, we calculated the Kappa statistic. Eleven of 13 Kappa scores measured are >0.4 in the NS data, with seven of those exceeding the 0.75 threshold indicating excellent agreement. For Ontario data, five of eight Kappa values are >0.4 with two >0.75. Table 4 shows the highest level of agreement among continuous variables for NS for days in hospital. For Ontario, both days in hospital and days between last chemotherapy and death show the highest level of agreement.

Table 3.

Comparison of data from administrative databases with data from chart abstraction data for categorical variables

Quality indicator	Nova Scotia (n = 108)		Ontario (n = 90)
	Actual agreement	Kappa	Actual agreement	Kappa
1. (a) Any chemotherapy in last 14 days (only those who had chemo)	0.98	0.79	0.95	0.57
1. (c) Any chemotherapy	0.94	0.86	0.95	0.89
2. Died in reference hospital	0.98	0.96	0.88	0.76
3. More than one ERa visit in last 30 days	0.94	0.37	0.87	−0.04
4.1 (a) More than one hospital admission in last 30 days	0.97	0.79	0.77	0.15
4.1 (b) More than 14 days in hospital in last 30 days	0.98	0.94	0.84	0.55
4.2 Admissions to the ICU in last 30 days	NA	NA	NA	NA
7. Any adverse eventb	0.46	0.04	0.63	0.12
9. Enrolled in palliative care	0.91	0.70	–	–
10. Any radiotherapy to the bones in last 180 days	0.97	0.93	–	–
11. Potent anti-emetic prescribed in last 180 days (those > =65 years, n=62)	0.95	0.70	0.84	0.52
14. Any new chemotherapy in last 30 days	0.97	0.56	–	–

^aThe billings data do not contain a specific hospital indicator. Since administrative data include visits to hospitals other than the reference hospital used for the chart abstraction, agreement statistics will be lower for this indicator.

^bAdverse events include falls, infections, bedsores or injuries.

NA, kappa cannot be calculated as there are less than two non-missing values.

Table 4.

Comparison of data from administrative databases with data from chart abstraction data for continuous variables

Quality indicator	Nova Scotia (n = 108) ICCa	Ontario (n = 90) ICCa
1. Average No. of days between last chemotherapy and death	0.47	0.72
3. Average No. of ERb visits in last 30 days	0.66	0.33
4.1 Average No. of days in hospital in last 30 days	0.98	0.72
4.2 Average No. of admissions to hospital in last 30 days	0.96	0.44

^aIntraclass correlation coefficient.

^bThe billings data do not contain a specific hospital indicator. Since administrative data include visits to hospitals other than the reference hospital used for the chart abstraction, agreement statistics will be lower for this indicator.

Note: Since these are continuous variables, actual agreement is not shown.

For the 14 indicators in NS and 10 in Ontario listed as measurable or partially measurable, results are presented in Table 5. As shown, several indicators can be represented by multiple statistics. Following Earle et al.,⁴³ averages indicate quality care across the entire population, whereas outlier percentages may be an indication of aggressive care.

Table 5.

Quality indicators in two provinces

Quality indicator	Statistic description	Nova Scotia (%)	Ontario (%)
1. Interval between last chemotherapy and death	(a) Any chemotherapy in the last 14 days (% of those receiving chemotherapy)	9.3%	15.7%
	(b) Average number of days between last chemotherapy and death (for those receiving chemotherapy)	71.1	60.3
	(c) Proportion receiving any chemo in the last six months	26.2%	41.0%
	(d) Proportion receiving any chemo in the last 14 days (% of whole cohort)	2.4%	6.4%
2. Site of death	Died in the hospital	63.4%	52.9%
3. Frequency of ER visits	(a) More than one ER visit in the last month	5.6%	6.9%
	(b) Average number of ER visits in the last month	0.41	0.44
	(c) Average number of ER visits per available day in the last month	0.02	0.02
4.1 Hospital days near the end of life (includes any inpatient days)	(a) More than one hospital admission in the last month	11.7%	15.6%
	(b) More than 14 days in the hospital in the last month	29.1%	23.9%
	(c) Average number of hospital days in the last month	9.9	9.8
	(d) Average number of hospital admissions in the last month	0.70	0.86
4.2 Hospital days near the end of life (includes ICU days only)	Percentage with at least one ICU admission in the last month	2.1%	4.1%
5. Continuity of care	(a) Modified Modified Continuity Index45
	(i) General Practitioners (GPs) only	0.79	0.73
	(ii) Non-GPs only	0.59	0.79
	(b) Average number of GP visits (excluding < 3 visits)	7.07	8.80
	(c) Average number of GPs (excluding <3 visits)	2.18	2.66
	(d) Average number of non-GP visits (excluding <3 visits)	10.28	57.29
	(e) Average number of non-GPs (excluding <3 visits)	4.37	10.67
6. Time and location of care	Percentage of home and office visits which occurred in the last two weeks	10.5%	22.7%
7. Adverse events	Bedsore, infection, fall, or injury	9.3%	12.6%
8. Enrollment in palliative care within three days prior to death	Enrolled in palliative care program within three days prior to death (for those enrolled in palliative care) (data only available for part of the province)	4.2%	NA
9. Access to palliative care	Enrolled in palliative care program (data only available for part of the province)	78.3%	NA
10. Radiotherapy for uncontrolled bone pain for bony metastases	Received palliative radiation to the bones	21.9%	NA
11. Potent antiemetic for emetogenic chemotherapy	Prescription for potent antiemetic in the last six months (those 65 years of age and older)	5.9%	10.5%
12. Multidisciplinary care	(i) Percentage of physician visits by GPs	60.7%	30.3%
	(ii) Percentage of physician visits by non-GPs	39.3%	69.7%
13. Access to care	Rurala residence	39.8%	15.1%
14. Interval between new chemotherapy and death	Had new chemotherapy in the last 30 days of life	2.6%	NA

^aThe definition of rural differs between Nova Scotia and Ontario.

NA, data not currently available to calculate the statistic.

Reliability

Inter-rater reliability shows high agreement in both NS and Ontario. Kappa statistics are >0.4 threshold for all indicators except one in both provinces. In addition, >53% of the variables have a Kappa score of >0.75 (data not shown). These results suggest a high degree of consistency in the implementation of our chart abstraction system. A comparison between the two provinces does show that, while agreement levels between raters were acceptable in both provinces, overall reliability numbers are higher in NS. Results point to the importance of rigorous instruction for abstractors when chart information is considered the gold standard.

Discussion

The objective of this study was to test the use of administrative databases to measure quality of care near the end of life. The results show that using these databases for this purpose is feasible, valid and reliable. In this study, seven indicators of quality care were found to be fully measurable in both NS and Ontario. With another seven indicators in NS and three in Ontario, being partially measurable, we can further our understanding of the services provided. Although aspects of the health care systems by province differ, common methods in data collection within the national setting allows for reasonable comparisons across jurisdictions.

Administrative databases can be an efficient and flexible tool for measuring and monitoring quality of care for cancer patients near the end of life. These data are continuously collected suggesting consistency over time, and are population-based, meaning point estimates will reflect what actually occurred without sampling variability. Even smaller jurisdictions can have a sufficient number of observations to draw reliable conclusions. Furthermore, persons who may not be served by a particular institution or agency are included; access to the care that these persons receive may be very important. Administrative databases also allow for variation in the way statistics are calculated, which may make them more acceptable to stakeholders or practitioners in a particular jurisdiction. Using a retrospective study design, administrative databases can be extracted, examined and reformulated relatively quickly using alternative measurement computations arising from dialogue among researchers, care providers, program managers and persons directly affected by end-of-life care decisions.²¹

Prospective studies are an alternative method where patients are identified as terminally ill, and subsequent care is analysed. It has been suggested,⁴⁶ that end-of-life quality care research only use prospective studies as the interest is in those who are dying. Certainly, both methodologies have strengths and weaknesses, and results will differ. As Earle and Ayanian argue,²⁹ both designs can be considered complimentary, each providing a unique insight into quality care research.

In this study, we examined the quality of care at the end of life for those who died of breast cancer. It is important to distinguish between those dying as a result of breast cancer and those with cancer who died of a cause other than breast cancer.³² To this end, administrative databases allow for linkages between various data sources to capture a homogeneous study population, and registration requirements for Vital Statistics in Canada result in a reliable and comprehensive study subject selection. While females with breast cancer who died of other causes are also an important group to study, the disparities in the cause of death are more likely to lead to variations in care unrelated to breast cancer. The methods used in this study are sensitive to this cohort selection process.

Chart validation is an important component of this analysis. There are limits, however, on the extent of available chart information. For example, we measured continuity of care by an index which uses the total number of physicians and the total number of visits for each individual. This includes visits to the office, home, and acute care facilities other than the study hospital. Obtaining access to all charts from physician offices, as well as from all hospitals where the patient may have received care, is not feasible. In this instance, linked administrative databases may yield the more valid measure.

An adverse event, defined as a bedsore, a fall, an infection or injury, is listed in Table 1 as a measurable indicator. As Table 3 indicates, however, agreement between chart and administrative databases, as measured by the Kappa, is below the recommended threshold in both provinces. The hospital abstract database was used in both provinces to calculate the proportion experiencing an adverse event. This database uses an extremely comprehensive data collection technique,⁴⁷ yet chart abstraction showed a much higher occurrence of adverse events for both provinces. In addition, the level of inter-rater agreement is low in both provinces. This suggests selected types of information may be difficult to obtain accurately. Results for this indicator should be viewed with caution, and any new indicators developed should undergo a validation process prior to their use to identify any concerns or limitations that may emerge.

Limitations of using administrative databases have been highlighted elsewhere.^22,48,49 For this study, physician billings databases are a key data source in both provinces. Increasingly in Canada, physicians are paid through methods other than fee-for-service. Salaried physicians are to submit ‘shadow billings’, that are claims for procedures that were performed but not directly linked to payment. An increasing rate of expected shadow billings brings into question completeness surrounding using these databases for research purposes. ICES⁴⁹ reports on the switch away from fee-for-service to sessional payment schemes in administrative databases for emergency room physicians (ICES; p 6).⁴⁹ To the extent that there are differences in the proportion of physicians under a fee-for-service system across time or across jurisdictions could result in misleading conclusions. To date, only a small proportion of physicians are paid through a system other than fee-for-service. However, further analysis may be required to estimate the size of the bias and, if necessary, add linkage to additional administrative data sources.

Potential data sources and methods of eliciting data must also be considered. Most readily available data sources have been designed for administrative use and not to measure quality of care. If they are to be used for capturing quality indicators, their strengths and weaknesses must be understood.⁹ Some potential difficulties include lack of linkages between data sources, usefulness of the actual type of data found in the existing data sources, the absence of common patient identifiers between data sources, and difficulty in aggregating data.¹⁰

Most importantly, and particularly with respect to end-of-life care, most databases will not capture some of the information considered important by patients, families and providers, such as psychosocial care, multi-disciplinary treatment, pain and symptom management, spiritual well-being and advanced directives.^23,50 Other domains that cannot be captured through administrative databases are empathy, dignity,¹⁶ and communication.⁵¹ Additionally, measurement and analysis of these indicators is not instantaneous. There are aspects of care that require continuous monitoring and immediate adjustment. Finally, policy-makers and researchers must recognise the underlying reasons behind changes in these indicators over time or differences across jurisdictions, to ensure the goals of the particular health care system are being maintained. It is not the intent of this work to suggest administrative databases are a comprehensive source of information, but rather that they can be used in conjunction with other methods for an overall profile of care. Clearly, there are still gaps to fill, but measuring quality of care will continue to evolve and will probably always need to rely on multiple data sources, such as qualitative and purposefully collected clinical data.