Pitfalls of Using Administrative Data Sets to Describe Clinical Outcomes in Sickle Cell Disease

Abstract

Background

Administrative data sets are increasingly being used to describe clinical care in sickle cell disease (SCD). We recently used such an administrative database to look at the frequency of acute chest syndrome (ACS) and the use of transfusion to treat this syndrome in California patients from 2005–2010. Our results revealed a surprisingly low rate of transfusion for this life-threatening situation.

Procedure

To validate these results, we compared California OSPHD (Office of Statewide Health Planning and Development) administrative data with medical record review of patients diagnosed with ACS identified by two pediatric and one adult hospital databases during 2009–2010.

Results

ACS or a related pulmonary process accounted for one-fifth of the inpatient hospital discharges associated with the diagnosis of SCD between 2005 and 2010. Only 47% of those discharges were associated with a transfusion. However, chart reviews found that hospital databases over-reported visits for ACS. OSHPD underreported transfusions compared to hospital data. The net effect was a markedly higher true rate of transfusion (40.7% vs. 70.2%).

Conclusions

These results point out the difficulties in using this administrative data base to describe clinical care for ACS given the variation in clinician recognition of this entity. OSPHD is widely used to inform health care policy in California and contributes to national databases. Our study suggests that using this administrative database to assess clinical care for SCD may lead to inaccurate assumptions about quality of care for SCD patients in California. Future studies on health services in SCD may require a different methodology.

INTRODUCTION

Studies show that 95–98% of pediatric patients with SCD in the United States survive to reach adulthood [1]. However, as patients transition to adult care, there is an increase in mortality [2]. The greatest use of the emergency room and rehospitalization rates occur in the group of SCD patients between 18–30 years of age [3,4]. This suggests a lack of appropriate resources for this group of patients. SCD is an orphan disease with an estimated 89,000 patients nationally [5]. Previous studies [1] have indicated that increased mortality in young adults may be due to deaths from acute chest syndrome (ACS), a potentially life-threatening pulmonary process treated with transfusion [6]. Unlike pediatric patients who are mainly seen at sickle cell centers and who may not necessarily require transfusion for an infiltrate [7]; the majority of adult patients are hospitalized outside of centers [8] where providers may have very little experience with SCD and may be unaware of the potential need to transfuse these patients for what looks like an infectious pneumonia. Recognition and appropriate treatment of ACS could represent a key indicator of clinical care for adult patients with SCD.

A national prospective study performed at hospitals with sickle cell centers and published in 2000 found that 73% of patients with SCD who developed ACS received transfusions [9]. Using hospital discharge data from California’s Office of Statewide Health Planning and Development (OSHPD), we found that only 46% of pediatric as well as adult patients in California who developed ACS or related pulmonary illnesses received transfusions [10]. These results could be interpreted to indicate that many patients with SCD admitted to hospitals in our state are not being transfused for ACS. However, there are significant methodological differences between our study and a national prospective study. In particular the severity of cases in the latter might be greater than in our study which is retrospective.

OSHPD is an administrative data set used primarily to track financial tracking and healthcare utilization. The database may provide readily available epidemiological data on SCD. However, administrative data sets are not designed to provide insights into clinical care, despite increasingly being used to do so [11,12]. They are not validated for SCD. Here we report findings from a study conducted to assess the accuracy of the diagnosis of ACS and the reporting of transfusion in this setting.

METHODS

OSHPD analysis

We initially performed a retrospective cohort study using OSHPD data to examine transfusion rates of patients with the diagnosis of SCD who were admitted with ACS or other respiratory conditions. Our analysis examined inpatient discharges for the period 2005–2010. Cases were selected by the presence of a sickle cell disease ICD-9 code in the primary or a secondary diagnosis (282.60–282.69, 282.41–282.42). These codes include Hemoglobin SC, Hemoglobin S β thal, as well as Hemoglobin SS. Further inclusion criteria included presence of ACS (517.3) or respiratory diagnoses including pneumonia, pulmonary edema, acute respiratory distress syndrome, and respiratory failure in any primary or secondary diagnosis (see Appendix). This study includes the latter codes because ACS may be diagnosed as another respiratory disorder. To examine those cases in which transfusion would be more likely to be indicated, we used the ICD-9 codes for hypoxia (799.0, 799.02) and intubation (96.0–96.72) as severity modifiers. Frequency of transfusion was described using the ICD-9 codes for transfusion of packed cells and exchange transfusion (99.0, 99.01, 99.03, and 99.04). For the comparison of OSHPD to the hospital billing data (validation study), we used only those OSHPD visits from the years 2009–2010.

Validation Study

In order to validate the rate of transfusion for ACS in the OSHPD data base, we reviewed all visits for ACS from three different hospitals. Cases were identified from the billing databases at each hospital for the years 2009 and 2010. Criteria for inclusion in the study included the ICD-9 code for ACS (517.3) as well as the diagnosis of sickle cell disease (see above) and an inpatient hospitalization. ACS was defined as an acute pulmonary infiltrate (not atelectasis) in a patient with SCD associated with fever, wheezing, tachypnea, hypoxia, or cough [9]. We did not include the other pulmonary diagnoses in this part of the study in order to simplify the data analysis, although this limits the sensitivity of the diagnosis of ACS.

Since the OSHPD database allows identification of visits by individual facilities, we were able to compare the number of cases we found in our billing data with the number of cases found in OSHPD for those two years for each hospital. We were also able to ascertain from OSHPD the number of transfusions associated with those ICD-9 codes for those two years for each facility. However, because we used a de-identified database, we were not able to link a specific case from the billing data to a visit in OSHPD. Therefore, we could only make a side-by-side comparison of total cases found in each database.

Three participating hospitals with large sickle cell populations contributed patients for chart review: Children’s Hospital Los Angeles (Hospital 1), a large tertiary care academic institution, and two community-based hospitals in Los Angeles County: Miller Children’s Hospital (Hospital 2) and Long Beach Memorial Hospital (Hospital 3, an adult facility). The institutional review board at each facility approved the study. There were a total of 162 cases among the three hospitals where the study was conducted. Each chart was reviewed by a hematologist with 100% validation of the SCD diagnosis in this chart review

The goals of the chart review were to: 1) confirm the diagnosis of ACS; 2) identify those patients who were transfused with packed red cells; 3) understand the reasons for any discrepancy between the diagnosis used to determine billing and the final diagnosis; and 4) review why patients were not transfused. Each identified case was reviewed by a clinician for clinical signs and symptoms of ACS. Those patients who did not have an infiltrate on chest X-ray were classified as not having ACS. These cases which were identified by billing data as ACS but were negative by clinical criteria were further classified as: a) diagnosis was entertained but not proven (“Ruled Out”); or b) “electronic medical record (EMR) issue,” when a rule-out diagnosis was not corrected in EMR or there was a failure of EMR to link records. Transfusions were identified from blood bank order sets, fluid input flow sheets, and/or physician and nursing daily notes.

We also examined why patients who had a valid diagnosis of ACS were not transfused. There were a number of cases where the patient was diagnosed with ACS but did not receive a transfusion because they were stable clinically and the providers deemed transfusion to be unnecessary given the clinical course. These patients were classified as “clinically well”. Other reasons for non-transfusion included patient refusal, chronically transfused as an outpatient, left against medical advice, and multiple alloantibodies.

As this was primarily a descriptive study, statistical analysis was limited to calculation of frequencies and rates. For the validation study, we also calculated positive predictive value (i.e., precision rate) of ACS diagnosis and sensitivity (i.e., identification rate) of transfusion procedures in OSHPD.

RESULTS

OSHPD results

From 2005–2010, the OSHPD database recorded a total of 41,899 inpatient discharges from California hospitals coded with a sickle cell diagnosis (Table I). 7,940 visits were coded with ACS or a related pulmonary diagnosis. Forty-seven percent of those visits were coded with a transfusion procedure code. Of the 2,744 visits coded specifically as ACS during this period, 52% were also coded with a transfusion. Adding the severity modifier code for hypoxia to ACS and related diagnoses resulted in an increase in the transfusion frequency to 57%. A list of diagnoses and procedures is found in the appendix.

Table I.

Frequency of ACS and transfusion OSHPD data, 2005–2010

Condition	# Patients	Transfused N (%)
ACS + related diagnoses	7940	3698 (47%)
ACS diagnosis only	2744	1414 (52%)
ACS and related + hypoxia	1125	641 (57%)
ACS and related + intubation	541	310 (57%)
ACS and related + death	207	102 (49%)

Validation Study

We restricted this analysis to the years 2009 and 2010. We found one fewer case of ACS in OSHPD than in the billing data at Hospital 1 and one extra case at Hospital 3; the number of cases in OSHPD and Hospital 2 billing data were equal (Table II). There were 66 transfusions recorded for ACS cases at the three hospitals in OSHPD and 85 cases associated with transfusion in the billing databases. Therefore, OSHPD under-reported transfusions during this period for these hospitals by 22%.

Table II.

True rates of ACS and transfusion (Tx) in OSHPD, 2009–2010

Data Source	Billing Data Cases	Verified ACS Cases	Verified ACS and Tx	OSHPD Tx	True Tx Rate	OSHPD Tx Rate
Hospital 1	110	83	66	57	79.5%	52.3%
Hospital 2	30	26	13	4	50.0%	12.9%
Hospital 3	22	12	6	5	50.0%	22.7%
Total	162	121	85	66	70.2%	40.7%

Table II reports the differences between ACS diagnoses found in OSHPD and the true rate of ACS from each hospital. At all three hospitals the number of verified ACS cases was less than the number of ACS cases in the billing data base. The true rate of transfusion was markedly higher at each institution than reported in OSHPD. Overall, the true overall rate of transfusion across the three institutions was 70.2%, compared to OSHPD’s reported transfusion rate of 40.7%.

Table III describes those patients verified as having ACS in the billing data but who were not transfused. “Clinically Well” indicates those patients who truly had ACS but were mild enough to not require transfusion therapy. At each hospital, we found multiple cases in which ACS was initially considered but eventually “ruled out” by the provider as not meeting criteria for this syndrome. This was most often due to a negative chest X-ray, but was occasionally due to another diagnosis, such as asthma. The cases shown as “EMR issue” were those in which the diagnosis of ACS was incorrectly perpetuated in an electronic medical record or where there was a failure to link inpatient and emergency room records such that the chart reviewers could not assess the source of the diagnosis. Finally, several patients had ACS but were not transfused for a variety of clinical reasons listed as “other” in Table III.

Table III.

Reasons for lack of transfusions in verified ACS cases.

Data Source	Clinically Well	Ruled Out	EMR Issue	Other*
Hospital 1 (N=44)	17	14	10	3
Hospital 2 (N=17)	10	5	0	2
Hospital 3 (N=16)	4	8	2	2
Total (N=77)	31	27	12	7

^*Other reasons: Patient refused transfusion; chronically transfused as outpatient; left hospital against medical advice during treatment; multiple alloantibodies.

We computed the positive predictive value for the diagnosis of ACS and the sensitivity for the procedure code for transfusion in the OSHPD data. The positive predictive value (the number of chart-verified ACS cases from all cases with an ACS diagnosis code in OSHPD), was 74.7% in our study. The sensitivity of transfusion codes refers to the number of transfusions correctly identified in OSHPD (66) out of all transfusions that actually took place as verified by chart review (85), and was 77.6% overall. Our methodology depended on ACS appearing in the OSHPD diagnosis list; therefore, we could not obtain estimates of true or false negative ACS cases in this database, and could not calculate specificity or negative predictive values.

DISCUSSION

The use of administrative data sets to describe clinical care in SCD is increasing[13]. These data sources are readily available and provide researchers with information on large numbers of patients for an illness that affects a relatively small number of individuals in this country. These data are well suited to address the epidemiology of this disease but may fare less well when used to address issues of quality of care [14]. In this paper, we address some of the practical issues concerning using an administrative dataset to assess clinical outcomes for patients with SCD, and show that the OSHPD data over estimates some conditions while under-reporting others. Since OSHPD is part of the Nationwide Inpatient Sample (NIS) of the Healthcare Cost and Utilization Project (HCUP), a well-known and widely used source of inpatient data, the issues addressed in this study also apply to the NIS as well as other similar administrative databases that derive from inpatient billing records.

Two previous studies using OSHPD and SCD focused on the diagnosis of stroke [15,16]. Both of these studies were strictly epidemiological in nature and did not attempt to assess quality of care for stroke. In the first paper, a small validation study showed that using an ICD-9 based strategy to identify stroke cases had a sensitivity of 74%, similar to our results. The second paper contained no validation study. In contrast, our study considered both the diagnosis code for ACS and the procedure code for transfusion, thus introducing two potential sources of error into the evaluation. OSHPD records up to 24 discharge diagnoses, including diagnoses which were present on admission. It is possible that the diagnosis of ACS could be missed if it was not listed in these groups, or if the patient died within a short period of time after admission. In our study we had very strong agreement between cases discovered in the hospital billing databases and OSHPD, with only 6 out of 162, or 3.7%, that were discordant.

We were not able to calculate the sensitivity of the diagnosis of ACS in OSHPD, though we recognize that another potential source of error in this type of study is the difficulty in making the diagnosis of ACS. The published definition [9] also meets coding criteria for a number of other diagnoses including bacterial pneumonia. Clinicians who are not familiar with caring for patients who have SCD might not recognize this syndrome and thus the coding would not reflect ACS. It is possible that our study missed cases that were truly ACS but not coded as such. In the cases we looked at, this would have been less likely to occur at the two pediatric sickle cell centers.

Coding for transfusion has previously been shown to be a concern. In a study in 2000, Segal et al. looked at the validity of inpatient billing records for red cell transfusion at a single institution [17]. This study found that records from the Blood Bank indicated 17% of transfused patients were not billed for this procedure. We found a similar percentage of patients (22%) who were transfused in the chart review but not reported to OSHPD. It was not within the scope of this study to determine the source of this error; however, if transfusion is not listed in the first 20 procedure codes, then it might be missed in the OSHPD data.

We found a primary factor contributing to OSHPD’s overestimation of ACS incidence relates to visits reported to OSHPD as including ACS because they were billed and considered for ACS diagnosis, but ultimately not found to have ACS. Coding guidelines specifically state that these types of cases should be billed for the presumptive diagnosis [18]. Previous studies that have investigated other, non-SCD diagnoses also reported that approximately 20–25% of cases are false positives due to inclusion of cases that were billed but not ultimately proven [19,20].

Finally, another source of error is the expanding use of electronic medical records. We found that there were two sources of error that originated as a direct result of using EMR. First, failure to update problem lists with correct final diagnoses can result in incorrect diagnoses being perpetuated in the record. Consequently, a patient can be coded for an illness that is not currently relevant, resulting in over-reporting the diagnosis. Second, if the inpatient EMR is not linked to the records for the emergency room or the outpatient clinic, it may be difficult to assess how the diagnosis was made or whether relevant procedures were performed.

The accuracy of a diagnosis or a procedure reported in OSHPD or other databases is vitally important as these data sources are routinely used to assess clinical outcomes and inform health policy. A recent study using the NIS investigated mortality in patients with the diagnosis of SCD [12]. The use of transfusion and the diagnosis of ACS were also examined as predictors of mortality within this study. These investigators found that having a transfusion, especially an exchange transfusion, as well as the diagnosis of ACS predicted an increased risk of mortality [12]. However, given the findings of our validation study, we would predict that incidence of ACS could be lower and the rate of transfusion could be higher than the NIS estimates, potentially changing the outcome of the NIS study. A recent study on the diagnosis of priapism for males with SCD may have similar challenges [11]. Finally, is impossible to validate the genetic diagnosis of SCD in a database, thus introducing another potential source of error.

In summary, we found that the use of a large statewide administrative database to describe clinical care in SCD offers an inaccurate picture of clinical practice, suggesting that the rate of transfusion for ACS in SCD was low when, in fact, the rate was much higher at the three institutions we studied. As a convenient means to look at large numbers of cases of SCD, the advantages of using these tools to consider health services for SCD is expanding. However, we suggest that these billing databases do not accurately reflect clinical outcomes for this orphan disease, and that caution should be used when interpreting studies that use similar databases to describe care in SCD.