Corroborating evidence by exploring sources of bias in observational spinal cord injury studies

Observational studies investigating large real-life datasets are a valuable resource in clinical research. Understanding the imperfect nature of clinical data, statistical approaches factoring in known confounders are instrumental for rigorously addressing bias.¹ Our recent work identifying pneumonia and postoperative wound infections (Pn/Wi) as risk markers for impaired long-term functional recovery and survival after spinal cord injury (SCI)² was considered as a strong statistical analysis.³ However, some unexplored putative confounders in terms of nonrandom loss to follow-up, temporal changes in clinical practice, and exclusion criteria were discussed.³ In order to evaluate and objectivize for the probability of attrition and temporal and selection bias, we apply and discuss an array of analytical tools extending beyond the format of the original publication.²

Methods

The major analysis framework of the original study regarding FIM_motor as primary outcome after SCI constituted a linear mixed regression model after multiple imputation of missing values due to dropout and a sensitivity analysis in complete cases in order to quantify the potential effect of attrition and guide the discussion of the results. This strategy was augmented with models stratified for the American Spinal Cord Injury Association impairment scale (AIS) to circumvent possible interactions with the injury severity.² In the stratified models, the imputed and complete case analyses produced differing results in AIS B and C subgroups.² To address the question whether inaccuracy in the imputations may be a reason for such disparities rather than nonrandom dropout,³ we compare the sociodemographic and neurologic baseline of complete cases with cases lost to FIM_motor examinations at 1 or 5 years.

The study included patients enrolled between 1995 and 2005 and the incidence of infections as well as the management of care after SCI may have changed throughout this time frame.³ To reveal temporal differences in the rate of Pn/Wi and evaluate their potential to influence the FIM_motor analysis, we compare the Pn/Wi rate by periods of enrollment (1995–1999 vs 2000–2005) and include the enrollment period as an explanatory variable in the linear models.

The exclusion criteria of the study were chosen to prevent unknown comorbidity or concomitant injuries from confounding the analysis. However, exclusion of patients bares the risk of selection bias, which can be explored by comparing characteristics of the included population with those of the “excluded” group. Because the number of excluded patients is relatively small in our study and their functional baseline is frequently missing, an outcome analysis³ would be underpowered. Thus, we analyze the association of sociodemographic baseline characteristics with study exclusion as dependent variable using a logistic regression model. With regard to the exclusion criterion “age <16 or >75 years,” we adjust for age by adding a linear and squared term to the model as we suspected possible effects to be U-shaped with extreme values for small and high age.

The analyses were performed in analogy to the original study² using SPSS for Windows (version 24.0, IBM Corp., Armonk, NY).

Results

Attrition bias

Significant baseline differences between complete cases and cases lost to follow-up exist in the total sample and in all AIS strata, indicating a pattern of attrition not completely at random. Major differences are a lower proportion of Caucasians and patients working prior to injury in some AIS strata with missing FIM_motor. Other disparities are concerning age and neurologic or functional baseline (table).

Table.

Comparison of baseline characteristics between the groups “missing (FIM_motor) follow-up” vs “complete cases” at 1 year (upper panel) and 5 years after SCI (lower panel)

Temporal bias

The Pn/Wi rate significantly decreased over time, from 53.3% (327 of 614 patients) in the period 1995–1999 to 40.2% (237 of 589) in the period 2000–2005 (p < 0.001). Adjustment of the linear regression model for enrollment period resulted in a change of only 0.001 in the R² without relevant effects on the explanatory value of Pn/Wi regarding variance in FIM_motor.

Selection bias

The logistic model reveals that baseline differences between the study population (n = 1,203) and excluded patients (n = 224) in terms of sex, working status, and education are related to age. The model with linear and quadratic age effects predicts a small exclusion probability of 5%–6% for participants age 33–49 years and an increased risk of 16% for participants age 20 and 62 years (both linear and quadratic term p < 0.001). None of the other baseline characteristics remains significant after age adjustment.

Discussion

We corroborated the robustness of the original study design² by reanalyzing the dataset for additional sources of bias. The exploration for nonrandom attrition reveals baseline differences between datasets lost to follow-up and complete cases prevailingly in terms of working status and ethnicity explicable by disparities in health care access.⁴ The imputation and the complete case analysis as a sensitivity analysis in the original study differ significantly only in the AIS B and C strata but not in the AIS A stratum or the total sample.² This is most likely connected to clearly smaller sample sizes and higher variance of outcome in AIS B and C groups^2,5 leading to underpowered strata due to attrition as demonstrated in the explorative analysis. Thus, inaccuracy of the imputation seems unlikely. Regarding temporal bias, the observed decline of infection rates over time is in line with longitudinal trends in the incidence of health care–associated infections.⁶ The adjustment of the regression models for the enrollment period does not reveal temporal bias regarding outcome, similarly to a previous study.⁷ Finally, the risk for selection bias is addressed by multiple logistic regression models adjusted for the u-shaped distribution of age in the excluded population.

As a common limitation of longitudinal observational trials, unmeasured confounders may exist also in this trial. For example, specific premorbid conditions or the management practice of infections were not documented in the study and the collection of additional data was not possible. Thus, confounding effects related to these items were only addressed indirectly by exclusion criteria or assessment of temporal effects, respectively. In addition, feasible instrumental variables were not available to address unknown confounders beyond the sensitivity analysis. We are fully aware of the fact that multiple imputation of missing values cannot eliminate biases caused by unmeasured confounders. However, we are convinced that imputation should be preferred to a complete case analysis, which additionally contains sources of biases due to measured confounders.

Our work illustrates that the validity of large observational studies, particularly in rare and heterogeneous conditions such as SCI,⁵ can be enhanced by combining comprehensive description of the data with consequent application of statistical tools to explore sources of bias in order to effectively augment the best evidence available.

Author contributions

Marcel A. Kopp: study concept and design, statistical analysis and interpretation of data, drafting of the manuscript, critical revision of the manuscript for intellectual content. Peter Martus: study concept and design, statistical analysis and interpretation of data, critical revision of the manuscript for intellectual content. Ralf Watzlawick: critical revision of the manuscript for intellectual content. Yuying Chen: study concept and design, critical revision of the manuscript for intellectual content. Michael J. DeVivo: study concept and design, acquisition of data, critical revision of the manuscript for intellectual content. Jan M. Schwab: study concept and design, critical revision of the manuscript for intellectual content.

Study funding

The study received funding from the Wings for Life Spinal Cord Research Foundation, Austria (grant WfL-DE-006/12), the Era-Net-NEURON Program of the European Union (SILENCE #01 EW170A and SCI-Net #01EW1710), and the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR grant 90DP0011 and 90SI50200100). NIDILRR is a Center within the Administration for Community Living, US Department of Health and Human Services.

Disclosure

M. Kopp, P. Martus, R. Watzlawick, M. DeVivo, and Y. Chen report no disclosures relevant to the manuscript. J. Schwab received funding from the W.E. Hunt and C.M. Miller Endowment and is a Discovery Theme Initiative Scholar (OSU). Go to Neurology.org/N for full disclosures.