A Validation Study of the Canadian Organ Replacement Register

Summary

Background and objectives

Accurate and complete documentation of patient characteristics and comorbidities in renal registers is essential to control bias in the comparison of outcomes across groups of patients or dialysis facilities. The objectives of this study were to assess the quality of data collected in the Canadian Organ Replacement Register (CORR) compared with the patient's medical charts.

Design, setting, participants, & measurements

This cohort study of a representative sample of adult, incident patients registered in CORR in 2005 to 2006 examined the prevalence, sensitivity, specificity, positive and negative predictive values, and κ of comorbid conditions and agreement in coding of patient demographics and primary renal disease between CORR and the patient's medical record. The effect of coding variation on patient survival was evaluated.

Results

Medical records on 1125 patients were reviewed. Agreement exceeded 97% for health card number, date of birth, and sex and 71% (range 46.6 to 89.1%) for the primary renal disease. Comorbid conditions were under-reported in CORR. Sensitivities ranged from 0.89 (95% confidence interval 0.80, 0.92) for hypertension to 0.47 (0.38, 0.55) for peripheral vascular disease. Specificity was >0.93 for all comorbidities except hypertension. Hazard ratios for death were similar whether calculated using data from CORR or the medical record.

Conclusions

Comorbid conditions are under-reported in CORR; however, the associated risks of mortality were similar whether using the CORR data or the medical record data, suggesting that CORR data can be used in clinical research with minimal concern for bias.

Introduction

Administrative data are increasingly being used to study outcomes of patients with end-stage renal disease (ESRD) requiring dialysis or transplantation (1,2) and as tools for benchmarking clinical outcomes within and between countries (3). Comorbid conditions are prevalent in this population and are associated with an increased risk of poor outcomes (4). Accurate and complete documentation is necessary to describe burden of illness, which can then be used to adjust for case-mix severity, reimbursement rates, and potential confounding in epidemiologic and clinical studies.

The Canadian Organ Replacement Register (CORR) is the national information system for renal and extrarenal organ failure and transplantation in Canada. Several recent publications using CORR data have adjusted the outcome of mortality and morbidity for differences in patient demographics, comorbid conditions, and cause of renal failure (5,6). Hence, the validity of these outcomes hinges upon the accuracy and completeness of the data in CORR.

The objectives of this study were to assess the quality of coding of demographics, primary renal disease, and comorbid conditions of patients at the start of chronic dialysis treatment compared with the patient's medical chart (reference standard). The effect of coding variation on estimates of patient survival was then evaluated. In addition, the factors contributing to discrepancies in the CORR data were identified.

Materials and Methods

CORR records the incidence, prevalence, and outcome of all chronic dialysis and solid organ transplant patients in Canada. Dialysis nurses and nephrologists (and at some sites unit clerks) document voluntary patient demographic information, the primary renal diagnosis, and existing comorbid conditions at the time of initial dialysis in a standardized CORR form (included in Supplemental Appendix 1) within 3 months of starting dialysis (7).

This study included all adult patients who started hemodialysis or peritoneal dialysis as their first form of renal replacement therapy between January 1, 2005 and December 31, 2006. Patients from dialysis units that reported <50 incident patients to CORR in 2005 and 2006 and patients who had died as of August 2007 were excluded from the sampling frame. We excluded those who had died at the time of the recoding exercise because of the difficulty in obtaining the dialysis chart for a deceased patient. A two-stage probability sample was used to select patients that first considered their geographic location and the number of incident dialysis patients registered in CORR. The second stage sampled patient records from the selected facilities, this time with consideration of the comorbid conditions that were reported to CORR.

To prepare for data collection, ten study coders underwent training in the data extraction process. Study coders were provided clear coding rules for data collection. The definition of comorbid conditions defined for CORR and this study are listed in Supplemental Appendix 2. Study coders were tested after the training session to ensure consistency and completeness when reviewing and interpreting medical record documentation. A selection of charts were independently reviewed by two study coders (inter-rater reliability) with a high (>95%) agreement.

Data collection took place during two time frames: in November and December 2007 and in September and October 2008. To collect the study data, study coders, who were blinded to the CORR data, reviewed the medical records (dialysis, nephrologist, and hospital charts). The age, sex, race, primary renal disease, and comorbid conditions were entered into a Canadian Institute for Health Information (CIHI) software application, and the CORR data were then unblinded and discrepancies between the two data sources were identified. Reasons for the discrepancies were recorded. The primary renal disease was aggregated into one of eight types of conditions, as listed in Table 2. The comorbid conditions listed in Table 3 were coded as “yes” for present, “no” for not present, and “unknown” when the presence was uncertain.

Table 2.

Agreement between CORR and the medical record for the primary renal disease

Type of Renal Disease	Percent Agreement on the Primary Renal Disease (95% confidence interval)
Glomerulonephritis	82.8 (74.9, 90.7)
Pyelonephritis	70.8 (54.8, 86.8)
Drug-induced nephropathy	58.7 (28.7, 88.6)
Polycystic kidney disease	89.1 (77.0, 100.0)
Hypertension/other vascular	66.7 (56.5, 77.0)
Diabetes	78.3 (70.8, 85.8)
Etiology uncertain or unknown	46.6 (35.9, 57.4)
Other	64.2 (48.6, 79.8)
Overall	70.9 (66.2, 75.6)

Table 3.

Prevalence of comorbid factors in CORR data and in the medical record before and after the start of dialysis

	Prevalence (%)
	CORR Data	Medical Record before Start of Dialysis	Medical Record Postdialysis Initiationc
Angina	18.2	16.6	17.2
Myocardial infarctiona	19.2	23.3	23.5
Coronary artery bypass grafts	12.3	13.7	13.8
Any cardiovascular diseaseb	29.1	33.2	33.9
History of pulmonary edemaa	21.5	26.7	26.8
Cerebrovascular diseasea	11.9	14.6	15.0
Peripheral vascular diseasea	11.7	18.1	18.1
Diabetes mellitusa	41.8	47.1	46.5
Malignancy before first dialysis treatmenta	10.3	14.0	14.3
Chronic obstructive lung disease	10.3	11.6	12.0
Receiving medication for hypertensiona	85.1	90.2	89.9
Other serious illness	9.0	12.3	12.5
Current smoker	12.1	14.7	15.3d

^aNet change in prevalence is statistically significant (P < 0.05).

^bIncludes angina, myocardial infarction, or coronary artery bypass.

^cUp to 3 months of documentation generated after initial dialysis.

^dThe difference between the predialysis and postdialysis study prevalence values is statistically significant (P < 0.05).

Data Analyses

Descriptive statistics were used to describe demographics, primary renal disease, and comorbid conditions. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for each comorbid condition, accepting the medical chart data as the reference standard for the purpose of calculating these statistics. The κ statistic was also used to assess agreement between the two data sources for the presence of the variables of interest, considering that chart data may not be accepted as the gold standard. Study weights and bootstrap weights were applied to the sampled records to allow for proper estimation of variance.

Adjusting Estimates to Account for Confounding Variables

Estimates of sensitivity, specificity, PPV, and NPV were adjusted for gender, age, treatment modality, and number of comorbid conditions using logistic regression models to identify patient variables most susceptible to under-reporting. Ninety-five percent confidence intervals (CIs) were based on the bootstrap weights used for sample design.

Hazard ratios were calculated using the CORR data and the data from the medical record to study the effect of these differences on mortality estimates. These estimates should not be interpreted outside of this study because of the immortal time bias and difference in follow-up. The unadjusted survival estimates for selected comorbid conditions and primary renal diseases were calculated using Cox proportional hazard models. The survival analysis included the time from the dialysis start to an end date of death, loss to follow-up, kidney function recovery, kidney transplant, or end of follow-up (December 31, 2007). Robust standard errors of the hazard ratio estimates were calculated to incorporate inter-regional variations. This was done to avoid their underestimation because there were significant regional reporting variations.

Results

A total of 10,197 dialysis patients were registered in CORR between 2005 and 2006. An additional 38 patients were included from December 2004 because of late submission of data. Exclusions including death before 2007 (1537 patients), facilities too small (858 patients), and four units refusing or unable to participate (549 patients) reduced the reference population to 7291 patients, of which 1351 patient records were randomly selected from 39 dialysis units. One-hundred and forty patient charts from four dialysis units were not recoded because of logistical challenges, and 86 patient charts were not recoded because the patient was transferred or deceased and the chart documentation was unavailable onsite. The remaining 1125 patient charts were recoded (response rate of 83%). Patient characteristics were similar between the 2005 and 2006 incident CORR population and the study sample as reported in Table 1.

Table 1.

Characteristics of patients who initiated dialysis between January 1, 2005 and December 31, 2006 in CORR and the study population

	Patients Registered to CORR		Patients Selected in This Study
	Male	Female	Male	Female
n (%)	6068 (59)	4129 (41)	775 (57)	576 (43)
Hemodialysis, n (%)	4977 (82)	3327 (81)	651 (84)	471 (82)
Peritoneal dialysis, n (%)	1091 (18)	802 (19)	124 (16)	105 (18)
Age in years, mean ± SD	64 ± 16	64 ± 16	63 ± 15	64 ± 15
Number of comorbid conditions, mean ± SD	3 ± 2	2 ± 2	3 ± 2	3 ± 2

Demographic Data

Agreement between CORR and the medical record exceeded 97% for health card number, date of birth, and sex. Lower agreement was observed for the patient's race (58%), for which discrepancies frequently were the result of a specific race reported in the CORR data but an unknown code reported in the medical record data.

Primary Renal Disease

The percent agreement between the CORR data and the medical record on the primary renal disease was 71% overall, although agreement rates varied based on the type of disease as described in Table 2. There was good agreement for polycystic kidney disease (89%), glomerulonephritis (83%), and diabetes (78%), whereas hypertension/other vascular renal disease had lower agreement rates (67%).

Comorbid Conditions

Comorbid conditions were generally more prevalent in the medical record, suggesting that comorbid conditions are under-reported in CORR data as described in Table 3. The only condition that was less prevalent in the medical record was angina. Further review of the postdialysis documentation had a minimal effect on the prevalence of comorbid conditions, as indicated in Table 3.

Sensitivity, specificity, PPV, and NPV results are presented in Table 4. Peripheral vascular disease, current smoker, and other serious illness were under-reported in CORR data. For example, of the patients who were documented to have peripheral vascular disease, only 47% (95% CI 0.38, 0.55) of them were correctly recorded as having this condition in the CORR data. High sensitivity was observed for hypertension (0.89; 95% CI 0.87, 0.92) and diabetes (0.86; 95% CI 0.82, 0.90). Low specificity was observed for hypertension (0.52; 95% CI 0.39, 0.64). No consistent effect on sensitivity was found after adjusting for age, gender, modality, diabetes, cardiovascular disease, and the number of comorbid conditions, although the sensitivity was often lower for patients with cardiovascular disease (data not shown).

Table 4.

Adjusted sensitivity, specificity, PPV, and NPV and unadjusted κ of comorbid conditions between CORR and medical record data

	Sensitivity (95% confidence interval)	Specificity (95% confidence interval)	PPV	NPV	κ
Angina	0.64 (0.56, 0.73)	0.91 (0.90, 0.92)	0.59 (0.52, 0.65)	0.93 (0.90, 0.95)	0.53
Myocardial infarction	0.62 (0.53, 0.70)	0.94 (0.91, 0.96)	0.75 (0.65, 0.85)	0.89 (0.86, 0.92)	0.59
Coronary artery bypass grafts/angioplasty	0.69 (0.61, 0.77)	0.97 (0.95, 0.99)	0.77 (0.64, 0.90)	0.95 (0.93, 0.97)	0.68
Cardiovascular diseasea	0.69 (0.62, 0.75)	0.91 (0.88, 0.93)	0.78 (0.72, 0.85)	0.85 (0.82, 0.89)	0.61
Recent history of pulmonary edema	0.62 (0.56, 0.68)	0.93 (0.91, 0.96)	0.77 (0.69, 0.85)	0.87 (0.85, 0.90)	0.59
Cerebrovascular disease	0.59 (0.48, 0.69)	0.96 (0.94, 0.98)	0.72 (0.61, 0.84)	0.93 (0.91, 0.95)	0.60
Peripheral vascular disease	0.47 (0.38, 0.55)	0.96 (0.95, 0.97)	0.72 (0.61, 0.83)	0.89 (0.86, 0.92)	0.50
Diabetes	0.86 (0.82, 0.90)	0.97 (0.96, 0.99)	0.97 (0.95, 0.99)	0.89 (0.85, 0.92)	0.84
Malignancy existing before first treatment	0.66 (0.57, 0.75)	0.99 (0.98, 0.99)	0.90 (0.85, 0.95)	0.95 (0.93, 0.97)	0.73
Chronic obstructive lung disease	0.60 (0.49, 0.70)	0.96 (0.94, 0.98)	0.67 (0.52, 0.81)	0.95 (0.93, 0.97)	0.59
Receiving medication for hypertension	0.89 (0.87, 0.92)	0.52 (0.39, 0.64)	0.94 (0.92, 0.97)	0.34 (0.25, 0.43)	0.33
Other serious illness	0.22 (0.14, 0.30)	0.93 (0.92, 0.94)	0.30 (0.24, 0.37)	0.89 (0.86, 0.93)	0.17
Current smoker	0.54 (0.46, 0.62)	0.95 (0.93, 0.97)	0.66 (0.55, 0.77)	0.92 (0.90, 0.94)	0.53

^aCardiovascular disease combines angina, myocardial infarction, and coronary artery bypass grafts/angioplasty into a single definition.

Hazard Ratio for Mortality

In general, hazard ratios for various primary renal diseases and comorbid conditions were similar whether these were calculated using the CORR data or data from the medical record. Hazard ratios remained <1 (indicating conditions that were protective of mortality) or remained >1 (indicating conditions that increase the risk of mortality). However, the extent of the risk for mortality sometimes changed in magnitude depending on the data source (i.e., CORR versus the medical record). Hazard ratios were underestimated for several conditions with CORR data as the data source as compared with the medical record as the data source. The only condition for which the hazard ratio was higher in CORR data as compared with the medical record was chronic obstructive lung disease. These results are summarized in Table 5. The hazard ratios were only calculated to study the effect of the coding differences on mortality estimates.

Table 5.

Unadjusted hazard ratios of mortality using CORR data and medical record data^a

	CORR Data Hazard Ratiob	Medical Record Hazard Ratio (95% confidence interval)
Primary renal disease
glomerulonephritis (reference)	1.0	1.0
pyelonephritis	1.13	0.51 (0.10, 2.73)
nephropathy, drug induced	1.53	1.78 (0.30, 10.65)
polycystic kidney disease	0.41	0.66 (0.26, 1.70)
hypertension/other vascular	1.32	1.54 (0.91, 2.60)
diabetes	1.18	1.44 (0.75, 2.76)
other	1.56	2.74 (1.46, 5.16)
etiology uncertain or unknown	1.51	1.39 (0.61, 3.19)
Comorbid conditions
angina	1.47	1.36 (1.01, 1.84)
myocardial infarction	1.54	1.60 (1.40, 1.82)
coronary artery bypass grafts/angioplasty	1.54	1.61 (1.14, 2.28)
cardiovascular diseasec	1.60	1.67 (1.33, 2.08)
recent history of pulmonary edema	1.86	1.79 (1.53, 2.10)
cerebrovascular disease	1.58	1.64 (0.87, 3.10)
Peripheral vascular disease	1.51	2.22 (1.76, 2.80)d
diabetes mellitus (type 1 and 2)	1.20	1.66 (1.41, 1.96)d
malignancy existing before first treatment	1.80	2.99 (2.36, 3.78)d
chronic obstructive lung disease	1.72	1.20 (1.00, 1.43)d
receiving medication for hypertension	0.68	1.34 (0.80, 2.24)d
other serious illness	1.45	1.55 (1.10, 2.17)
current smoker	0.94	0.88 (0.58, 1.35)
Number of comorbid conditions
0 or 1	0.63	0.42 (0.16, 1.09)
2 or 3 (reference)	Reference	Reference
≥4	1.55	1.77 (1.47, 2.13)

^aThese HR do not represent the entire CORR cohort because of exclusions of patients specific to this study.

^bCIs for the CORR data hazard ratios are not included because these are the actual ratios calculated when using the full sampling frame for this CORR study, and the purpose of this comparison was to test if the CORR data hazard ratio (in this case our population parameter) was significantly different from the hazard ratio calculated when using the study data. Because the latter of these is an estimate that was generated using a sample, it includes a 95% CI.

^cCardiovascular disease is defined as having any of the following: angina, myocardial infarction, or coronary artery bypass graph/angioplasty. Vascular disease is defined as having peripheral vascular disease or cerebrovascular disease.

^dThe difference in hazard ratios is statistically significant (P < 0.05).

Reasons for Inconsistencies in Coding Comorbid Conditions

Study coders reported differences in the interpretation of physician clinical notes and incomplete chart documentation as reasons for discrepancies. The average discrepancy rate across all comorbid conditions was 54.2% and the use of “no” or “unknown” accounted for 43.7% of these. This specific finding reflects a coding directive provided to the study coders in which “no” would represent that they located a statement in the chart notes that confirmed the absence of the disease. Persons entering CORR data did not have this additional instruction because it was not explicitly stated in the CORR Instruction Manual. As a consequence, their interpretation for when to code “no” and when to code “unknown” was less defined. Full details on the coding discrepancies and recommendations for improvement are available in the Data Quality Study on the CORR (7).

Discussion

This rigorous validation study contributes significantly to the understanding of data quality in CORR, will assist in the interpretation and analysis of existing data, and will identify measures to improve data collection in the future. We found that patient demographics were generally coded with high reliability, although race was not easily discerned from the patient chart. The primary renal disease had 71% agreement between the medical record and CORR. Development of standard and consistent definitions for the cause of renal disease may improve this.

Comorbid conditions were generally more prevalent in the chart data compared with CORR data, suggesting a tendency to under-report to CORR. High specificity was observed for most comorbid conditions, showing that it was uncommon for patients to be falsely labeled with a condition. The presence of diabetes had a high sensitivity and specificity, making it a reliable risk modifier. One of the issues identified from this study was the lack of understanding when to code the comorbid condition as “unknown” (i.e., there is nothing in the chart to confirm the presence or absence of the condition) versus coding the comorbid condition as “no” when this is explicitly noted in the chart.

Separating the medical record into the time before and the time after the initiation of dialysis, there was no significant change in the prevalence of medical conditions that was found documented in the patient chart, with the exception for the smoking status of the patient. This supports the importance of resource planning and performing a thorough review of the medical record at the start of dialysis to capture accurate and complete comorbid conditions.

In general, the risk of death associated with various primary renal diseases and comorbid conditions were similar whether they were calculated using the CORR data or the medical record data, although the effect size was generally larger for the medical record data. This suggests that CORR data can be used to adjust for potential confounders, although a more complete adjustment would be achieved with complete recording of all data contained within the medical record. The survival estimates should not be interpreted outside of this study because of the immortal time bias and difference in follow-up.

Our results are similar to reports by others (8–11). The Choices for Healthy Outcomes in Caring for End-Stage Renal Disease (CHOICE) study reported on data collected on Centers for Medicare and Medicaid Services (CMS) form 2728 (comparable to the CORR data form) compared with the patient chart (11). They too reported a generally low sensitivity and high specificity for the comorbid conditions. In addition, despite the tendency to over-report, patients were better able to report the presence of comorbid conditions compared with data extracted from form 2728 (12).

We found less under-reporting in the CORR database than in the CHOICE study. Form 2728 is often completed by the social worker because the main purpose is to register the patient in the ESRD program so that he/she receives Medicare coverage. In contrast nephrologists, nurses, data clerks, and other allied health personnel complete CORR, which has the potential to allow for improved documentation. The U.S. Renal Data System also uses CMS form 2728 to collect information on comorbid conditions, but its distant validation project did not include comorbidities (13). The implication of using a data source in which there is under-reporting of comorbid conditions is that there is an incomplete adjustment for confounding, which could bias associations toward null findings.

Although it is well established that measures of comorbidity predict outcomes in dialysis populations (4), recent studies suggest that adjustment for comorbid conditions adds little to the variance in survival after adjustment for age, gender, dialysis modality, and primary renal disease (14). Others suggest that adjustment for age, diabetes, and cardiovascular and peripheral vascular disease is sufficient to adjust for mortality differences (4). This is supported by a recent report from the Dialysis Outcomes and Practice Pattern Study (15). After the 17 most prognostic comorbid conditions were accounted for, the addition of 28 more conditions added only 4% of the total explained variance in the survival model. Focusing on a core set of comorbidities that are most associated with patient outcomes will improve the completeness of the data collected and there is no loss of explanatory power (16).

Coders from dialysis units reported incomplete, unclear, or conflicting chart documentation and inadequate directives in the CORR Instruction Manual as barriers to coding comorbid conditions. Educating coding staff and nephrologists on the value and use of CORR data, improving the completeness and quality of chart documentation, clarifying the use of “unknown” and “no” for coding comorbid conditions, and improving the CORR Instruction Manual and CORR form may improve the accuracy of the data.

Other strategies identified to improve completeness of data in renal registries include mandatory reporting, financial or other incentives, linking to multiple data sets to populate missing data or to perform continuous validation, use of specially trained staff to collect data, standardization of a few comorbid conditions with proven associations, and routine regular data validation to continually improve the data collection process (16). The public reporting of patient outcomes with accountability back to the renal program would rapidly increase the need and willingness to collect the necessary data to explain or refute reported outcomes.

This study has several strengths, including a comprehensive validation of patient demographics, primary cause of renal disease, and comorbid conditions in a nationally representative sample of the Canadian dialysis population, which will assist users of the CORR in future analysis including international comparisons. Reasons for discrepancies were collected that provided insight for improvement of data reporting in CORR. The Standards for Reporting of Diagnostic Accuracy guidelines, including training and testing of coders with inter-rater reliability, was used to report this study (17).

Our study also has limitations. The medical chart was considered the gold standard; however, we were unable to account for charting errors that could occur during routine history taking and documentation. Additionally, these data reflect only the presence and not the extent of disease.

In conclusion, patient demographic data were accurately collected with some coding differences for primary renal disease and under-reporting for most comorbid conditions. Although sensitivities for comorbid conditions were moderate to low, the patient's risk of mortality, for various primary renal diseases and comorbid conditions, were similar whether these were calculated using the CORR data or study data, suggesting that the data can be used in clinical research without biasing study results.

Disclosures

None.