Exploring the Hazards of Scaling Up Clinical Data Analyses: A Drug Side Effect Discovery Case Report

Abstract

We assessed the scalability of pharmacological signal detection use case from a single-site CDW to a large aggregated clinical data warehouse (single-site database with 754,214 distinct patient IDs vs. multisite database with 49.8M). We aimed to explore whether a larger clinical dataset would provide clearer signals for secondary analyses such as detecting the known relationship between prednisone and weight. We found significant weight gain rate using the single-site data but not from using aggregated data (0.0104 kg/day, p<0.0001 vs. -0.050 kg/day, p<.0001). This rate was also found more consistently across 30 age and gender subgroups using the single-site data than in the aggregated data (26 vs. 18 significant weight gain findings). Contrary to our expectations, analyses of much larger aggregated clinical datasets did not yield stronger signals. Researchers must check the underlying model assumptions and account for greater heterogeneity when analyzing aggregated multisite data to ensure reliable findings.

Introduction

Implementation of electronic health records (EHRs) has enabled secondary analysis of clinical data for a variety of purposes^1,2. Traditional time-consuming and costly research methods such as randomized controlled trials (RCTs)^2-5 are being complemented with secondary analyses of clinical data generated via clinical practice as a core activity of burgeoning learning healthcare systems^6,7. As one important use case, 32% of novel therapeutics approved by the FDA were associated with a post-marketing safety event⁸. Thus, post-marketing discovery and surveillance for drug side effects (pharmacoepidemiology) using retrospective analyses of large clinical data sets has gained interest.

Though the current abundance of machine-readable clinical data sets enables such analyses, clinical data are prone to quality issues^9-13. These issues are sometimes considered part of the noise that may mask signals in EHR data. It is often argued that larger data sets will amplify the signals, allows a more reliable estimate for the noise and improves the statistical power for detecting associations^14-18. In the healthcare setting, this can be achieved in two ways. Healthcare systems can wait until their databases have enough patients to study a specific phenomenon, in which case, a more traditional research approach such as single site experimental study such as an RCT is developed and implemented. An alternative strategy, is for multiple healthcare systems to aggregate their data in clinical data warehouses (CDWs)^19,20 to increase statistical power²¹, making clinical signals more detectable by secondary analysis methods such as regression analysis, as long as the underlying assumptions of these models are met. Research efforts are currently geared toward building large CDWs,^22-27 yet we found few studies in the literature reporting the impact of EHR data aggregation^28,29 or providing solutions to its consequences³⁰ and we found none illustrating its impact on traditional analytical methods such as regression analysis.

In this article, we attempt to "rediscover" the association between prednisone, a commonly prescribed corticosteroid, and weight gain^31,32 with longitudinal linear regression methods using two databases: a single-site CDW³³ and a much larger aggregated CDW. We chose this association because it is well-accepted by clinicians³⁴ and is defined by relatively objective numerical data (i.e., drug administration and weight change over time). Further, weight measurement does not require complex equipment and is often recorded during routine clinical care for a wide variety of patients across care settings. Thus, investigating the relationship between prednisone and weight gain is, in a sense, the "best case" scenario for assessment of factors associated with drug-side effect. Our goal was to "rediscover" a known side effect without leveraging knowledge about the side effect. Thus, we approximated the process of monitoring for previously unknown side effects. The contribution of this work is to illustrate an existing challenge of scaling up analyses using large, aggregated CDWs.

Methods

Data Sources - We used longitudinal linear regression methods to analyze the known relationship between prednisone and weight gain using real EHR data extracted from a single-site CDW and an aggregated CDW database. The single-site data set was extracted from an outpatient clinic's EHR production database and contained 754,214 distinct patient IDs with data from April 2004 to January 2014. The aggregated data set, Cerner Health Facts (Cerner Corporation, Kansas City, MO) a HIPAA de-identified database, contained over 49.8 million distinct patient IDs with data from January 2000 to October 2014. We selected patients with at least one prednisone prescription, all their recorded weights and covariates such as age and gender. Descriptive statistics are shown in Table 1. No missing values were found for age, and gender variables in either data set. The weight variable was normally distributed. Because the distribution of drug exposure was not normal, we categorized exposure into a binary variable encoded as high (above mean) vs. low (below mean) exposure. We filtered out patients under 21 years of age and extreme outliers for weight (i.e., weight>400 kg). A second round of filtering was performed on the weight variable by excluding measurements that were more than three standard deviations away from the mean on both sides. This study has been approved by the Committee for the Protection of Human Subjects (the UTHSC-H IRB) under protocol HSC-SBMI-13-0549.³⁵

Table 1. Descriptive statistics for single and aggregated datasets. Note that the original aggregated database contains over sixty times more patients than the single-site database, whereas the extracted data has slightly over 18 times more patients.

	Single-Site	Aggregated
Number of Distinct Patient IDs in Database	754,214	49,826,219
Timespan Covered	Apr. 2004 - Jan. 2014	Jan. 2000 - Oct. 2014
Number of Patients Included (% of Database)	9,767 (1.29%)	169,944 (0.34%)
Number of Weight Measurements	93,617	2,278,953
Missing Drug Exposure	15.40%	20.40%
Average Age (in years) for 1st Prescription	54±15	63±17
Median Age (in years) for 1st Prescription	55	64
MeanSD Weight	84.222kg	82.524kg
MeanSD Prednisone Exposure	312697mg	7811,740mg

Statistical Analysis - We used summary statistics such as mean, median and extreme values (e.g., values more than three standard deviations away from the mean) to screen the data for outliers, missing values and erroneous data. We assessed normality of continuous variables based on histograms. To detect weight gain over time, we built a longitudinal linear regression model using generalized estimating equations (GEE) method. Our longitudinal regression model predicted the main outcome (weight) based on drug exposure (high/low), the number of days from the first prescription (time), the patient's age at the time of first prescription, the patient's gender. An autoregressive correlation structure was used to account for potential correlation between multiple measurements from the same individuals. Statistical significance was set at a=0.05. A model was built on weight as the dependent variable with independent variables including, time and a binary exposure group (cumulative prednisone dose below or above the mean) along with known covariates: gender and age. The observational time span (or time windowing) was varied around the time of prescription between 90 days before prescription and 360 days after prescription to improve model fit. We used the Quasi-likelihood under Independence Model Criterion (QIC)³⁶ as a quantitative measure of goodness-of-fit. QIC is a generalization of Akaike's information criterion which represents the goodness of fit of statistical models typically employed with GEE methods and informs model selection. We used SAS (version 9.4, SAS Institute Inc., Cary, NC) for statistical analysis.

Sensitivity Analysis - To assess the robustness of our results, we built regression models for multiple sub-populations contained in our data set using the same methods described above. For example, we built separate regression models for males only, females only, initial weight below or above average and for patients with ages below or above average at the time of prescription. These subgroup analyses aimed to identify which subgroups of the cohort contributed to the overall effect. If the effect varies by multiple subgroups, it indicates potential effect modifications that should be explored.

Results

We built longitudinal linear regression models to optimize model fit (i.e., reduce QIC) exploring time windows between 10 days prior and 120 days after prescription. The best fit was found for 7-90 days and 7-32 days for the single-site and aggregated data respectively (Table 2). This matches what is clinically expected: patients gain weight within the first three months of starting prednisone. These conditions returned the largest effect sizes for the relationship between prednisone and weight in both data sets: 0.0114 kg/day (p<0.0001) for the single-site data and - 0.050 kg/day (p<0.001) but no association with exposure to prednisone (p=0.847) based on the aggregated data.

Table 2. Results from Longitudinal Linear Regression models with weight as the dependent variable under varying parameter assumptions based on the single-site dataset (n1=93,617) and the aggregated dataset (n2=2,278,953).

		Overall Regression (After Prescription)				Upper Time Limit				Upper and Lower Time Limit
		Single-site		Aggregated		Single-site		Aggregated		Single-site		Aggregated
Parameter		Estimate	p-value	Estimate	p-value	Estimate	p-value	Estimate	p-value	Estimate	p-value	Estimate	p-value
Intercept		83.45	<.0001	93.8	<.0001	82.05	<.0001	92.6	<.0001	84.2715	<.0001	92.2	<.0001
Time (Days from Prescription)		0	0.9004	-0.0032	<.0001	-0.0048	0.0038	-0.061	<.0001	0.0104	<.0001	-0.050	<.0001
Age (in years)		-0.087	<.0001	-0.2399	<.0001	-0.0729	<.0001	-0.228	<.0001	-0.101	<.0001	-0.213	<.0001
Gender	M	14.43	<.0001	8.3245	<.0001	14.78	<.0001	8.60	<.0001	14.10	<.0001	7.81	<.0001
Gender (Ref.)	F
Exposure (Classified)	Hi	-0.56	0.389	0.149	.0002	-1.40	0.0266	0.146	<.0001	-1.64	0.0616	-2.51	0.847
Exposure (Ref.)	Lo
Days*Exposure (Class)	Hi	-0.0003	0.237	-0.00008	<.0001	0.0158	<.0001	-0.0194	<.0001	0.0156	0.0055	1.47	0.068
Days* Exposure (Ref.)	Lo
QIC Value		79,020		835,487		15,669		393,087		7,335		120,133

Overall, our longitudinal linear regression (i.e., all data after prescription) yielded no statistical significance for time (p=0.900), exposure (p=0.389) or the interaction between these two variables (p=0.237). The aggregated dataset showed significance for all variables, most probably due to the large size of the dataset. However, the model estimated that the weight gain was negative over time, though the amount of weight loss was not statistically significant. Model fit was measured using QIC, with lower values indicating better fit, but no specific reference ranges. Both models showed a very high QIC=70,020 (single site) and QIC=835,847 (aggregated) indicating very poor fit of the model to the data.

Setting an upper temporal boundary at 90 and 32 days from first prescription for each data set respectively brought all variables beyond the significance threshold (time: p<0.0001, exposure: p=0.0266, interaction between these two variables: p<0.0001), improving the fit (QIC=15,669) for the single site data set. The final coefficient estimate value for time was 0.0104 kg/day (~3.8 kg/year), which revealed a positive correlation between time and weight increase in patients prescribed prednisone. On the other hand, the aggregated data showed significance for all variables. The QIC value improved to QIC=393,087.

We found larger QIC values indicating a poorer fit for larger upper time limits (60 and 90 days). Setting a delayed lower time limit where the data considered for the regression started at 7 days after the first prescription improved fit (QIC=7335), preserving statistical significance for all variables except binary (high/low) exposure (p=0.0645) in the single-site dataset. Adding this lower time limit to the aggregated data set yielded significance for all variables except exposure (p=0.847) and the interaction between exposure and time (p=0.068). However, the estimate for the time variable was negative (-0.050 kg/day) indicating a slight decrease in weight over time. It is important to note that the minimum QIC found for the aggregated data set was much higher than the single-site data set (QIC=120,133 vs. 7335), suggesting that a larger, more complex data set may include much more variability and may be much harder to account for heterogeneity in variance by this model. In summary, we found weight gain with prednisone in the single-site data matching the expected outcome but slight weight loss with prednisone in the aggregated data.

Sensitivity Analysis - To assess the robustness of our findings, we performed sensitivity analysis by doing subgroup analyses (Figure 1). The single-site CDW showed a positive relationship between time and weight for all categories. Only age categories above 70 and patients with an initial weight below average showed no significance. These findings suggest homogeneity within the subgroups of this single-site data set with an upward trend detectable in multiple subgroups. On the other hand, the findings from the aggregated data were inconsistent. Regression analysis provided 11 negative estimated effects (only 5 statistically significant) and 4 positive estimated effects (none statistically significant). The largest effects were found for patients in their 20s (estimated effect=-0.514 kg/day, p=0.338), 50s (estimated effect=-0.426 kg/day, p=0.261), age below average (estimated effect=-0.292 kg/day, p=0.046) and male patients (estimated effect=-0.320 kg/day, p=0.006). These results suggest that this data set contains heterogeneous sub-populations that have very different weight changes over time.

Figure 1.

Relationship between weight and follow up time for sub-populations within the single-site and aggregated data sets. The single-site data set presents similar results for most sub populations, whereas the aggregated data set returned much more variable results.

We compared size of data set and sampling rate (i.e., measures per patient) for the two data sets (Table 3). Overall, the aggregated data had only a slightly lower sampling rate than the single-site data set (8.65 vs. 9.56 measurements/patient). In contrast, for each optimized time window presenting the best QIC fit (i.e., 7-90 days and 7-32 days for the single-site and aggregated datasets respectively and found that the aggregated data had roughly twice the sampling rate compared to the single site data. However, the aggregated data set contained multiple visit types such as outpatient, inpatient and emergency, whereas the single-site database contained outpatients only. Comparing outpatient data only, we found 2.96 vs. 9.65 measurements/patient for the single and aggregated data sets respectively and 1.26 vs. 1.84 measurements/patient respectively within the selected time window. In this case, the lower sampling rate may partly explain the lower p-values due to lower statistical power for the single-site data in both analyses (p=0.014 vs. p<0.0001). The lower measurement count per patient could be explained by the difference in time windows 7-90 days vs. 7-32 days. However, using the 7-90 day time window and outpatients only the CDW data set returned a smaller sample than the single-site data set (1.69 vs. 1.84 measurements/patient). Analysis of these data showed weight gain (estimate=0.0114 kg/day), yet not significant (p=0.487).

Table 3. Regression analysis results summary for each data set and analytical method with patient counts and total number of measurements. Black arrows indicate statistically significant findings, whereas gray arrows represent p-values greater than 0.05.

	Single-Site		Aggregated
Data set	Full Data Set	Days 7-90 (Strongest Signal)	Full Data Set	Days 7-32 (Strongest Signal)	Outpatients Only	Outpatients Days 7-32	Outpatients Days 7-90
Patient Count	9,854	4,812	169,056	34,194	37,890	7,008	12,055
Total Number of Weight Measurements	94,233	8,872	1,355,508	127,550	112,098	8,874	20,370
Average Number of Weight Measurements Per Patient	9.56	1.84	8.02	3.73	2.96	1.27	1.69
Statistical Estimate (kg/day)	0 (p=0.900)	0.0104 (p<0.0001)	-0.0032 (p<.0001)	-0.050 (p<.0001)	0.0005 (p=0.781)	-0.0280 (p=0.0014)	0.0114 (p=0.487)
Finding

Finally, we built a model by controlling for the site where the weight measure was recorded on the aggregated dataset; this is often done for multisite aggregated data. Controlling for site did not change our conclusions regarding rate of change in weight over time, finding very similar estimates to that in the final model from Table 2 (e.g., -0.0503, p<.0001 vs. -0.050, p<.0001 for the time variable). The dataset contained 146 different sites out of which 35 returned statistically significant estimates for the effect of time on weight; This revealed significant differences across sites. We then evaluated potential interaction between exposure and site terms (i.e., exposure*site), finding 63 statistically significant interaction terms out of 273, revealing that prednisone exposure depended on the clinical site in 23% of all sites. We also found very similar estimates (e.g., -0.0498, p<.0001 vs. - 0.050, p<.0001 for time) in this model that accounted for interactions, which led to the same weight loss conclusion. These additional results indicate a robust statistical difference across sites and a significant interaction between the site and exposure in this particular aggregated database. However, controlling only for the study site did not improve the model, change the relationship between weight and time or the conclusions regarding the effect of prednisone on weight in the larger aggregated dataset.

Discussion

We found that prednisone was associated with weight loss, weight gain or no effect depending on the analysis methods, data set and assumptions. The much larger aggregated data set seemed more heterogeneous, including multiple visit types (e.g., outpatient, inpatient, emergency department, etc.), highly variable sampling rate (i.e., number of weight measurements per patient), multiple sources of data that seemed to present statistically distinct results and an interaction between the exposure and the site. This counters the idea that a larger aggregated data set would present a stronger overall signal and yield more robust findings in all circumstances, as we did not consistently find the expected association between prednisone and weight gain using traditional statistical regression methods. Subgroups analyses also yielded more inconsistent results in the aggregated dataset compared to the single-site dataset. In summary, the aggregated dataset appeared more heterogeneous and presented more analytical challenges during analyses.

Many studies have explored issues related to EHR data quality and bias^2,9,37-42. However, the heterogeneous nature of CDWs has only been mentioned as a potential hazard^43-48 and few publications have shown or quantitated the impact of heterogeneous aggregated clinical data sets on analysis.^28-30 Data quality is often defined as "fitness for purpose"^49-51. However, "fitness for purpose" is difficult to define in the absence of a clear understanding of the data sources constituting a clinical dataset.^52,53 Without this knowledge, it is difficult to anticipate threats to "fitness for purpose", because the processes that produced the analytical data are unknown and potentially unknowable due to data merging and deidentification. In other words, it is much more difficult to assess whether an aggregated dataset is "fit-for-a-particular-purpose" than for a single-site dataset (e.g., a local CDW) where data production processes are known by the analytical team. Simply put, it is more challenging to assess whether aggregated data are "fit-for-a-particular-purpose" than for a single-site/local CDW where data production processes are known. This case report illustrates this limitation and our findings suggest that CDW heterogeneity and data quality may affect analytical outcomes that may not be mitigated by using a larger database.

Our results suggest that these data quality and potential biases can change analysis outcome and are not necessarily mitigated by using a larger database. This may be due to the fact that clinical data reflect workflow, convention and multiple other factors in addition to "biology".^38,54-57 The number of such confounders grows with the aggregation of data from multiple care contexts including institutions, settings (e.g., outpatient vs. inpatient, primary care vs. subspecialty, medical vs. surgical, etc.) and other contexts. We found that visit types (inpatient vs. outpatient vs. emergency department) was one of the strongest confounders, which is more intimately related to clinical workflow than the biology of the studied population. This is consistent with existing work.^56,57 We also found differences across sites and interactions that must be taken into account for reliable analysis⁵⁸. This hints at the potential existence of analytical challenges particular to aggregated CDW (i.e., heterogeneous clinical workflow-derived data collected into a single database) that go beyond pitfalls expected in analyses making use of data produced for research purposes (e.g., multi-site data). These challenges must be further defined, and their effects evaluated in future research.

Though more data generally means more statistical power and narrower confidence intervals^32,33, it is important to consider potential heterogeneities³⁴ in large multisite aggregated clinical data sets. Because patients from diverse institutions (i.e., diverse clinical workflows), receiving diverse modes of treatment are included, the data may not be comparable^14,27,35-37. This has been often cited in the statistical literature^64,65 but could be easily overlooked in applied clinical informatics data reuse. The complexity of data production processes and aggregated databases are a constant threat to appropriate data use⁶⁶. It may be possible to tease out this complexity by employing reasonable assumptions that rely on clinical and healthcare workflow knowledge³³. For example, Hripcsak et al.⁶⁷ were able to reproduce previous Pneumonia Outcomes Research Team (PORT) studies using EHR data, but only after eliminating the vast majority (up to 90%) of the patients. In addition, their analysis (like ours) also benefited from knowing the expected results.

Understanding the workflow that produced the data being analyzed is often challenging; particularly for multisite data aggregated across institutions and care settings. Incomplete understanding of how a data set was produced is, generally a hazard to analysis and interpretation of analytical results from large clinical data sets⁶⁶. Clinical workflow may be difficult to define for a particular clinic or patient type even at a single institution. For example, are patients weighed routinely on every visit or only when there is reason to suspect a change or a specific clinical question? Are prescriptions from all providers, some of whom may be from other institutions, reliably entered into the source EHR? Our findings suggest that this threat grows with clinical data aggregation. They also suggest caution when reusing large aggregated CDWs for secondary analyses and the use of automated "big data" approaches to find previously unrecognized side effects. Also, just like in RCT data analysis, there are other statistical threats such as Simpson's paradox^68,69 and sub-group heterogeneity^65,70 that must be considered under even if the research team has perfect knowledge of how the data were produced.

A strength of our study is that we used two relatively large, robust data sets. The single-institution data included over 750 thousand patients and the multi-site database included over 49.8 million patients; among the largest clinical data sets that exist in the US.

Our study has limitations that will be addressed in future work. First, the analytical methods selected were not comprehensive. However, we ran similar regression on both datasets for the sake of comparability and explored additional regressions on the aggregated dataset that uncovered its complexities. To better understand the reasons for our findings, we chose traditional statistical models rather than neural network-based machine learning approaches (e.g., deep learning methods) that produce "opaque" models⁷¹. Though these methods could potentially enable accurate analyses despite poor data quality and hetorogeneity,⁷² understanding their role in analysis scalability was beyond the scope of this initial work. Most analysis decisions were driven by the need to replicate of a previous analysis³³ where the effect was found for the single-site dataset. This was done to allow for comparable analytical results. For example, we used a longitudinal regression models that are based on GEE methods with a categorization of the prednisone exposure variable per the initial analysis setup. Other advanced methods and their adaptation to these aggregated datasets will be explored in future work. Second, we chose a particular drug-side effect association. Thus, our results may not apply to other associations, particularly side effects that do not evolve over time. For example, it may be easier to discover events such as myocardial infarctions⁵⁵ than trends over time. Third, we did not investigate all possible sub-populations and potential covariates (e.g., race and ethnicity), concurrent clinical events such as other clinical conditions, drugs taken or surgical procedures. Many additional variables could potentially improve accuracy, some of which may be difficult to accurately assess (e.g., compliance with medications). An alternative approach would be a case-control retrospective study design with propensity scoring to control for potential confounding effects. Although these issues are clearly important, our primary goal was to assess the impact of size of data set on the detection of drug side effect associations in large aggregated clinical data sets rather than to build an optimal model for this particular case or to discover new biology.

Conclusion

Analysis of larger clinical databases does not necessarily generate clearer overall signals, in particular if the large data set is aggregated from multiple heterogeneous data sources. Analyses that successfully detected a known association in a single-site dataset were unable to detect this same association in a much larger aggregated data set despite of leveraging significant knowledge about the side effect (e.g., expected timing relative to exposure). Analyses of large, aggregated, anonymized data sets require attention to additional details addressing their heterogeneity beyond basic analysis design point typically considered for smaller, single-site clinical datasets.