Comparison of Administrative/Billing Data to Expected Protocol-Mandated Chemotherapy Exposure in Children with Acute Myeloid Leukemia: A Report from the Children’s Oncology Group

Abstract

Background

Recently investigators have used analysis of administrative/billing datasets to answer clinical and pharmacoepidemiology questions in pediatric oncology. However the accuracy of pharmacy data from administrative/billing datasets have not yet been evaluated. The primary objective of this study is to determine the concordance of Pediatric Health Information System (PHIS) administrative/billing chemotherapy data with Children’s Oncology Group (COG) protocol-mandated chemotherapy and to assess the implications of this level of concordance for further PHIS research.

Procedure

Data from 384 pediatric patients (1060 courses of chemotherapy) with acute myeloid leukemia treated on COG clinical trial AAML0531 was previously merged with PHIS data. PHIS chemotherapy administrative/billing data was reviewed for the first three courses of chemotherapy. Accuracy was assessed using three metrics: recognizability of chemotherapy pattern by course, chemotherapy administration pattern by individual medication, and concordance with the number of days of protocol-defined chemotherapy.

Results

The chemotherapy pattern was recognizable in 87.3% of courses when course-wide accuracy was assessed. Chemotherapy administration pattern varied by medication. Cytarabine had perfect concordance 70.9% of the time, daunorubicin had perfect concordance 77.4% of the time, and etoposide had perfect concordance 67.8% of the time.

Conclusions

The accuracy of chemotherapy administrative/billing data supports the continued use of PHIS data for epidemiology studies as long as investigators perform data quality control checks and evaluate each specific medication prior to undertaking definitive analyses.

INTRODUCTION

Acute myeloid leukemia (AML) is the second most common pediatric hematologic malignancy. Recent clinical trials have demonstrated improved outcomes with five-year overall survival rates of 50–60%.[1,2] While randomized clinical trials are the gold standard methodology for evaluating therapies for childhood cancer, such trials are unable to address all clinically important questions. Alternative methodologies, such as analysis of administrative/billing datasets, may complement clinical trial data by evaluating clinical and pharmacoepidemiology questions in pediatric AML.[3,4] The validity of these findings depends on the accuracy of chemotherapy pharmacy data. Thus far, the accuracy of pharmacy data from administrative/billing datasets has not been evaluated in pediatric oncology.

Data from 384 patients treated on Children’s Oncology Group (COG) clinical trial AAML0531 was recently merged with data from the Pediatric Health Information System (PHIS) database.[5] The AAML0531 trial randomized 1022 eligible patients to standard chemotherapy with or without gemtuzumab between August 14, 2006 and June 15, 2010. The PHIS database captures chemotherapy pharmacy billing data that includes medication name and route of administration on a day-by-day basis in 43 freestanding tertiary care children’s hospitals in the United States. These merged data provide a comprehensive dataset to compare administrative/billing pharmacy data to the expected chemotherapy mandated by a COG protocol.

The primary objective of this study is to determine the concordance of PHIS chemotherapy data with COG protocol-mandated chemotherapy and to assess the implications of this level of concordance for further PHIS research. Specifically, we hypothesized that PHIS administrative/billing data would be highly concordant with the COG protocol-mandated chemotherapy. Furthermore, we hypothesized that both the PHIS administrative/billing data and the COG protocol-mandated schedule would be highly concordant with chemotherapy administration as recorded in the medication administration record (MAR) at a single institution.

METHODS

Data Sources

The PHIS database is an administrative/billing dataset of clinical and financial information that includes 43 free-standing, tertiary-care children’s hospitals located in major metropolitan centers and represents 85% of free-standing children’s hospitals in the United States. The Children’s Oncology Group is the largest international pediatric cooperative oncology group. Details of the COG-PHIS merge for AAML0531 have been published previously.[5] In brief, a probabilistic merge was used to identify patients enrolled on the Children’s Oncology Group (COG) AAML0531 trial at PHIS sites. The probabilistic merge was based on center, date of birth, gender, and date of diagnosis. The merge was performed at the Children’s Hospital Association (CHA), which oversees PHIS, and de-identified merge results were then transferred to the research team at the Children’s Hospital of Philadelphia (CHOP) for chemotherapy pharmacy data extraction and comparison. Forty-one percent of patients (416 patients) enrolled on COG clinical trial AAML0531 were treated at PHIS institutions and 94% of these patients were identified in the merge.[5] This study was approved by the Institutional Review Board at CHOP.

The first three chemotherapy courses on AAML0531 were included in this study. Gemtuzumab was not included given that billing codes would not appear in PHIS due to its status as an experimental agent on the AAML0531 trial. Only courses of chemotherapy administered during treatment with protocol therapy were included.

Background Variables

Demographic information including gender, age, race, ethnicity and insurance status was available in the PHIS database and was compared to COG data on all patients enrolled on AAML0531. As was done on clinical trial AAML0531, age was categorized as <2, ≥2 and <11, and ≥11. Race categories were: white, black, Asian, other, and unknown. Ethnicity was coded as a dichotomous variable (Hispanic/non-Hispanic). Insurance status was divided into the following categories: private, public, self-pay, other, and unknown. The number of courses of therapy on AAML0531 given at each hospital was tabulated and analyzed as a continuous variable.

Accuracy of Chemotherapy Administration

Chemotherapy pharmacy data was reviewed by a pediatric oncologist (TPM) for each of the first three courses of chemotherapy for each patient in the merged dataset. Any uncertain categorizations were confirmed by a second pediatric oncologist (RA). Accuracy in this study was assessed using three metrics: overall pattern of chemotherapy doses by course, chemotherapy administration pattern by individual medication, and number of days of AAML0531-defined chemotherapy. Graphical representation of chemotherapy patterns is shown in Supplemental Figure 1.

Accuracy Based on Chemotherapy Pattern By Course

First, accuracy was assessed by determining if a course of chemotherapy was recognizable as the appropriate COG-protocol mandated regimen. The pattern of chemotherapy administration reported for each course was evaluated and assigned one of the following categories: recognizable, not recognizable, or no chemotherapy billed (Supplemental Figure 1). Recognizable was defined as a pattern where each chemotherapeutic had no more than one excess or missing day and where the temporal relationship was correct within one-day variability over the treatment course when compared to the COG-protocol. Additional doses of cytarabine were permitted due to the possibility of intrathecal doses of cytarabine in addition to the mandated intravenous doses.

Accuracy Based on Individual Chemotherapy Administration Pattern

Second, accuracy was assessed by evaluating the pattern of chemotherapy administration reported for each medication individually. The pattern in PHIS was compared to the prescribed regimen on COG protocol AAML0531, and each chemotherapeutic agent was assigned a category of accuracy. Pattern accuracy categories were as follows: perfect chemotherapy match, excess days of chemotherapy, missed days of chemotherapy, no chemotherapy, and patient deceased during scheduled chemotherapy. Categorization allowed for determination of percentages of excess or missed days of chemotherapy.

Accuracy Based on Percentage of Correct Days of Chemotherapy

Lastly, accuracy was evaluated as the percentage of correct days reported in PHIS compared to the expected number of days on the COG protocol. Each chemotherapy medication for each course was dichotomized as a perfect match (PHIS reported 100% of the expected number of days) or imperfect match (greater than or less than 100% of the expected number of days). Intrathecal and intravenous administration of cytarabine could not be differentiated using the available PHIS data. PHIS pharmacy data does not distinguish the number of doses per day; cytarabine is dosed twice daily and was considered a perfect match if 10 or 11 days of cytarabine were observed in course 1, 8 or 9 days of cytarabine were observed in course 2, or 5 or 6 days were observed in course 3.

Single Center Validation

For validation, chart review of merged patients was performed by reviewing the Medication Administration Record (MAR) for the patients treated at CHOP. For this abstraction, chemotherapy medication and administration date and time were recorded. Medication review was performed for cytarabine, daunorubicin and etoposide. Etopophos was also reviewed to determine if etopophos was substituted for missing days of etoposide. The days of chemotherapy administration recorded in the PHIS dataset were compared to what was documented in the MAR. The chemotherapy recorded as given in the MAR was also compared to what would have been expected per the COG protocol. Chemotherapy administered just after midnight in the MAR was considered a perfect match to the PHIS record or COG regimen for the prior day as the medication may have been billed by the pharmacy before midnight one calendar day, but administered the following day.

Statistical Analyses

Demographic data were summarized using standard descriptive statistics. Each course of chemotherapy was considered an independent event for all analyses because accuracy of administrative/billing data would not be expected to vary by admission at an institution. All patients who received chemotherapy were included, including those who died during chemotherapy. The percentage of courses with perfect chemotherapy based on the number of correct doses in the whole cohort was determined for each chemotherapy agent. The percentage of courses with perfect chemotherapy based on the number of correct days was determined for each chemotherapy agent for the cohort of patients who received at least one dose of chemotherapy. The percentage of courses with perfect chemotherapy pattern categorization was compared to the percentage of courses with a 100% correct number of days of chemotherapy using Spearman correlation. Logistic regression was used to determine if there were differences between courses in the percentage of acceptable chemotherapy administration reported. Generalized estimating equations (GEE) method was used to obtain the robust variance estimate to account for potential clustering by hospital. The percentage of each pattern of accuracy for the individual medications and the percentage of recognizable patterns of accuracy by course were tabulated. For the CHOP MAR data, percentages of courses with 100% match between the PHIS data and the MAR were determined for each type of chemotherapy. Percentages of courses with 100% match between the MAR and the expected COG regimen were determined for each type of chemotherapy. Univariate logistic regressions with GEE method were performed to evaluate demographic factors and the number of courses given at a hospital as potential predictors of accuracy. A p-value of 0.05 was considered statistically significant. All statistical analyses were performed using STATA version 12 (College Station, TX).

RESULTS

Demographic Data

Of the 1022 eligible patients enrolled on AAML0531, 416 were treated at 37 PHIS-associated hospitals and 384 had data available in PHIS. These 384 patients contributed 1060 courses of chemotherapy for evaluation.[5] The median age was 8.6 years (range 0 to 23.9) and 52.2% were female. The majority of patients were white (72.0%) and non-Hispanic (77.4%). Half of the patients (49.4%) had private insurance listed as their primary insurance. The range of courses per patient was 1 to 3. The range of courses per hospital site was 3 to 65, with a mean of 38.8 (standard deviation 16.4) and median of 36 (IQR: 28 to 54).

Accuracy Based on Chemotherapy Pattern By Course

When evaluating the first metric of accuracy, 80.9% of courses had a recognizable chemotherapy pattern, 11.8% of courses did not have a recognizable pattern, and 7.3% of courses had no chemotherapy reported in PHIS. Of those courses with chemotherapy reported, 87.3% had a recognizable chemotherapy pattern.

Accuracy Based on Individual Chemotherapy Administration Pattern

Table I shows patterns of chemotherapy administration reporting based on the results of assigning categories of accuracy by individual medications. For cytarabine, 13.3% of courses had excess chemotherapy reported and 10.3% of courses had missing doses. However the percentage of courses with excess cytarabine was 26.6% for course 1 and only 7.3% in course 2 and 4.1% in course 3. For daunorubicin, only 5.6% of courses showed excess doses of chemotherapy and 6.9% had missing doses. For etoposide, most of the inaccuracy was due to missing doses (18.9%); excess doses were reported in 4.8% of courses.

Table I.

Percentage of Chemotherapy Accuracy Patterns

	Perfect	Excess Doses	Missing Doses	Deceased during chemo	No Doses
Cytarabine (1060 courses)	68.2	13.3	10.3	0.4	7.8
Daunorubicin (739 courses)	77.3	5.6	6.9	0.5	9.7
Etoposide (1060 courses)	67.6	4.8	18.9	0.4	8.3

Accuracy Based on Percentage of Correct Days of Chemotherapy

Table II reports the percentage of time that chemotherapy was reported as administered concordantly with the prescribed COG regimen. There were 78 courses (40 patients) where no chemotherapy was reported in PHIS. There were 5 additional courses (4 patients) where no cytarabine was reported. There were 19 other courses (17 patients) where no daunorubicin was reported. There were 10 courses (10 patients) where no etoposide was reported as administered. Over all courses, daunorubicin was reported as administered perfectly the greatest percentage of the time (77.4%). Etoposide was reported as administered perfectly the least often (67.8%).

Table II.

Percentage of Perfect Chemotherapy Courses

	Overall	Course 1	Course 2	Course 3	p-value*
Cytarabine	70.9	59.3	76.1	78.8	<0.001
Daunorubicin	77.4	77.6	77.3	---	0.887
Etoposide	67.8	68.7	67.1	67.6	0.841

^*Logistic regression with GEE method to account for clustering by hospital.

Cytarabine was reported as administered perfectly 70.9% of the time. There were statistically significant differences between each course in the percent reported as correctly administered for cytarabine (59.3%, 76.1% and 78.8% respectively, p<0.001). The differences between courses 1 and 2 and between courses 1 and 3 were statistically significant (p <0.001 in both comparisons), but the difference between courses 2 and 3 was not statistically significant (p = 0.202). This indicates that the statistically significant difference between courses for cytarabine is due to course 1. When patients who are not reported to have received any chemotherapy in PHIS were excluded, the percentage of perfectly administered chemotherapy increased to 76.9%, 85.8%, and 74.0% for cytarabine, daunorubicin and etoposide respectively.

Figure 1 shows histograms of the numbers of missed or excess days compared to the COG protocol-determined correct number of days to illustrate the percentage of accurate days of medication for each type of chemotherapy. The majority of the time only one day was in excess or missed. Data are shown for course 1 for etoposide and daunorubicin as there were no differences between courses for these medications. For cytarabine, data are shown for courses 1 and 2 as there were differences between course 1 and courses 2 and 3. For course 1, the number of doses within 1 dose of the protocol mandated regimen was 82.9% 96.8%, 92.5% for cytarabine, daunorubicin, and etoposide respectively. Excess doses of cytarabine or etoposide were often seen as additional doses after the prescribed number of consecutive days of medication. For daunorubicin, excess doses were most commonly on days in between two prescribed doses.

Figure 1. Deviation from Protocol-Mandated Chemotherapy Exposure by Chemotherapy Agent.

A. Cytarabine Course 1 (356 Total Courses)

B. Cytarabine Course 2 (331 Total Courses)

C. Daunorubicin Course 1 (341 Total Courses)

D. Etoposide Course 1 (347 Total Courses)

Zero doses incorrect indicates a perfect match. Data are shown from Courses 1 and 2 for cytarabine and Course 1 only for daunorubicin and etoposide for patients who received at least 1 dose of chemotherapy.

The percentage of courses with perfect chemotherapy accuracy based on the percentage of days of correct chemotherapy reported in PHIS was compared to the percentage with perfect accuracy categorization for each medication. The Spearman correlation coefficient for cytarabine was 0.940 (p <0.001), for daunorubicin was 0.998 (p <0.001), and for etoposide was 0.996 (p<0.001).

Univariate analyses showed that no demographic features were predictors of accuracy of chemotherapy documentation in PHIS. However, there was a trend towards an increased percentage of perfect accuracy for hospitals with a greater number of courses of chemotherapy given for all three medications (cytarabine: OR 1.026, p = 0.007, 95% CI 1.007–1.045; daunorubicin: OR 1.012, p = 0.014, 95% CI 1.002–1.022; etoposide: OR 1.042, p <0.001, 95% CI 1.021–1.062), which indicates that there is hospital-level variation (Table III). Figure 2 shows the variability in the percentage of courses assigned to each match category by hospital for cytarabine. Graphs for etoposide and daunorubicin were similar (data not shown).

Table III.

Univariate Logistic Regression of Predictors of Percentage of Correct Days of Chemotherapy

	Variable	Odds Ratio	P-value	95% CI
Cytarabine	Courses Per Site	1.026	0.007	1.007–1.045
Daunorubicin	Courses Per Site	1.012	0.014	1.002–1.022
Etoposide	Courses Per Site	1.042	<0.001	1.021–1.062

Univariate logistic regression analyses were performed for demographic variables and the number of courses per hospital site. GEE method was used to account for clustering by hospital. Only the variables with statistically significant results for at least one chemotherapy medication are shown.

Figure 2. Center-level Variability of Match Categories For Cytarabine.

All courses of chemotherapy at the 37 sites in the database were included in this graph. Each bar represents a single institution: color indicates the percentage of courses for each match category. The numbers on the Y-axis indicate the number of courses of chemotherapy at that site.

Chart validation was performed at CHOP. Twenty patients (57 courses) enrolled on AAML0531 and identified in the PHIS dataset were treated at CHOP. The chemotherapy administration data matched the expected regimen based on the COG protocol 98.2% of the time for cytarabine, 89.5% of the time for daunorubicin and 91.2% of the time for etoposide. When comparing the MAR to the PHIS data, the CHOP chemotherapy administration record matched PHIS data 89.5% of the time for cytarabine, 86.0% of the time for daunorubicin and 82.5% of the time for etoposide.

DISCUSSION

To our knowledge, this is the first analysis of the accuracy of chemotherapy pharmacy data in an administrative/billing database compared to an expected COG protocol-based regimen. The pattern of chemotherapy administration was recognizable in 87.3% of courses where chemotherapy was reported in PHIS. Chemotherapy reporting in PHIS perfectly matched the COG specified daily regimen on an individual medication level for at least 74.0% of courses with at least one billed chemotherapeutic in PHIS. These results indicate that PHIS pharmacy data accurately represent a complicated regimen of protocol-mandated chemotherapy in a substantial majority of chemotherapy episodes, but also highlight that data quality control checks must be performed on these data.

Chart review at a local institution validated that chemotherapy administration recorded in the chart closely parallels the protocol-mandated regimen. At least 89.5% of all chemotherapeutics were accurate when the MAR was compared to the expected COG chemotherapy regimen. Furthermore, PHIS pharmacy data concur with the MAR in a substantial majority of chemotherapy episodes. Comparison of the MAR to PHIS data also showed good accuracy, with perfect matching rates at least 82.5% of the time. The lack of complete concordance between MAR and PHIS data may stem from chemotherapy delays due to toxicity and/or billing errors.[6]

While PHIS data are widely used, recent reports have raised concerns about the accuracy of other administrative/billing datasets and the effect of this inaccuracy on outcome assessments,[7–9] and literature regarding the accuracy of billing data in PHIS has been lacking. While work is ongoing to assess the potential impact of the observed inaccuracies in PHIS data on important clinical endpoints, several general conclusions about the use of PHIS data may be made based on these results.

First, the level of accuracy of the dataset determines its utility for a given type of study. For studies using administrative/billing data to identify patients with a particular malignancy, chemotherapy patterns must be recognizable. However, if the research aims to study cumulative exposure to a type of medication, the frequency of medications being reported as excess or as missing days is crucial.

Second, cytarabine had the greatest number of excess days while etoposide had the greatest number of missed days. A substantial fraction of excess cytarabine days can be attributed to the fact that intravenous and intrathecal cytarabine doses cannot be distinguished in PHIS. As expected based on AAML0531 treatment guidelines for CNS involvement,[10] there was a significant drop in the percentage of excess cytarabine doses between course 1 and subsequent courses. These results highlight the fact that medication administration route can potentially bias analyses of medication usage and must be explicitly considered during the dataset construction process. The larger numbers of excess cytarabine days may also be due to the greater number of planned total doses and twice-daily dosing. Based on the available data, the observed differences in etoposide and daunorubicin accuracy are more difficult to explain. Etopophos is sometimes substituted when a patient has an allergic reaction to etoposide. Although etopophos was not included in the PHIS dataset, the pattern of missing etoposide doses was not indicative of a switch to etopophos partway through a course, as missing days could occur at any time during the five protocol-mandated days. In addition, etopophos dosing was reviewed in the CHOP MAR and did not replace missing etoposide doses in the 57 courses reviewed at this single institution. The small number of total doses of daunorubicin given in each course and lack of specific patterns of excess or missing doses make it difficult to draw conclusions about the inaccuracy of daunorubicin billing data.

Third, patterns of inaccuracy vary by medication type. There were multiple instances in which all three medications were reported on two consecutive days (Supplemental Figure 1), which could be due to inadvertent duplication when chemotherapy was prescribed on one calendar day and administered the following day. In the cases of missing doses, no pattern of missing data was detected within each course. Missing doses could be due to physician prescribing preferences, protocol-mandated chemotherapy delay, or to billing omission errors.

Lastly, these results show that medication accuracy varies by hospital site. Data from individual centers may need to be removed from analyses when pharmacy data from those sites are missing or inaccurate. While this may decrease cohort size and raise concerns about generalizability, a more homogeneous and accurate sample set may enable detection of associations that would otherwise be obscured.

Chemotherapy administrative/billing data have been used to identify cohorts of patients with certain oncologic diagnoses within the PHIS database,[11,12] but this is the first study analyzing whether chemotherapy is accurately reported in an administrative/billing dataset. The primary limitation of this study is that the gold standard of chart abstraction of chemotherapy administration was not feasible at all sites. Thus, protocol-mandated chemotherapy regimen was used as a proxy for the gold standard. Chart abstraction of MAR data from one institution was performed in order to demonstrate that at least at one site the actual administration of chemotherapy as indicated in the MAR was not dramatically different from the protocol-mandated chemotherapy. Despite this limitation, these data strongly suggest that investigators should perform data quality control checks on PHIS pharmacy data and consider the type of study being performed prior to undertaking definitive analyses. Moreover, the results of these analyses may determine how medication exposure is categorized and whether sensitivity analyses are necessary. The importance of this variation in patterns of accuracy for future studies will depend on the research questions being addressed. Research studies using a binary (yes/no) exposure variable will likely be less sensitive to this potential source of error than analyses examining multiple medications relative to one another.

It is important to note that 87% of chemotherapy courses were recognizable and approximately 75% of complex, sequential chemotherapy exposures were perfectly captured in PHIS data when patients with completely absent chemotherapy data were excluded. This level of pharmacy data accuracy supports continued use of PHIS data for complex pharmacoepidemiology and clinical epidemiology studies once data quality checks have been performed. Additional analyses in datasets with multiple classes of medications are needed to more fully define the impact of this variation. Currently, efforts are underway to extend these analyses using hospital electronic medical record data across multiple medications. In addition, further analyses of accuracy using merged datasets including different types of medication data, such as administrative/billing data and electronic medical record data, should be performed. Such analyses will serve as the foundation for more comprehensive guidelines regarding the evaluation of pharmacy administrative/billing data accuracy.

Supplementary Material

Supp FigureS1. Supplemental Figure 1. Examples of Chemotherapy Patterns.

AraC = Cytarabine

Dauno = Daunorubicin

VP16 = Etoposide

Yes = Chemotherapy reported on a given day

No = Chemotherapy not reported on a given day