Seven Deadly Sins in Trauma Outcomes Research: An Epidemiologic Post-Mortem for Major Causes of Bias

Abstract

Background

Because randomized clinical trials (RCTs) in trauma outcomes research are expensive and complex, they have rarely been the basis for the clinical care of trauma patients. Most published findings are derived from retrospective and occasionally prospective observational studies that may be particularly susceptible to bias. The sources of bias include some common to other clinical domains, such as heterogeneous patient populations with competing and interdependent short- and long-term outcomes. Other sources of bias are unique to trauma, such as rapidly changing multi-system responses to injury that necessitate highly dynamic treatment regimes like blood product transfusion. The standard research design and analysis strategies applied in published observational studies are often inadequate to address these biases.

Methods

Drawing on recent experience in the design, data collection, monitoring and analysis of the 10-site observational PROMMTT study, seven common and sometimes overlapping biases are described through examples and resolution strategies.

Results

Sources of bias in trauma research include ignoring 1) variation in patients’ indications for treatment (indication bias), 2) the dependency of intervention delivery on patient survival (survival bias), 3) time-varying treatment, 4) time-dependent confounding, 5) non-uniform intervention effects over time, 6) non-random missing data mechanisms, and 7) imperfectly defined variables. This list is not exhaustive.

Conclusion

The mitigation strategies to overcome these threats to validity require epidemiologic and statistical vigilance. Minimizing the highlighted types of bias in trauma research will facilitate clinical translation of more accurate and reproducible findings and improve the evidence-base that clinicians apply in their care of injured patients.

BACKGROUND

As the seven deadly sins lure the unwitting from grace and harmony, so insidious sources of bias deny trauma outcomes research its scientific and public health impact. In trauma resuscitation research, the sources of bias can be so subtle that they elude even randomized clinical trials (RCTs). The challenges to valid trauma outcomes research include those common to other clinical domains,¹ such as heterogeneous patient populations with competing and interdependent short- and long-term outcomes. Others are relatively unique to trauma, such as chaotic, rapidly changing multi-system responses to injury that necessitate highly dynamic, often idiosyncratic treatment regimens (e.g., blood product transfusions). Standard research design and analysis strategies are often inadequate, and more appropriate methods may be complicated, unfamiliar and difficult to communicate effectively to non-statistical audiences. Overcoming threats to validity in trauma research also depends on the availability of sufficient epidemiologic and statistical as well as clinical and informatics resources. Drawing on our multi-disciplinary team’s experience in the design, data collection, monitoring and analysis from the 10-site PRospective Observational Multicenter Major Trauma Transfusion (PROMMTT) study,^{2, 3} we describe relevant examples of seven major and sometimes overlapping sources of bias in trauma outcomes research. In addition, we discuss mitigation strategies for each type of bias that could be used to increase the validity and impact of comparative effectiveness research in trauma.

7 DEADLY SINS IN TRAUMA OUTCOMES RESEARCH

Sin 1- Ignoring Indication Bias

The primary source of indication bias is the practice of classifying patients by the non-randomized intervention they received during the course of their medical treatment instead of the preceding conditions that necessitated the intervention. This practice is based on the naïve assumption that indications for the intervention are as balanced between study groups in an observational study as they would be in an ideal RCT. Under this assumption, observational study patients are classified by receipt of the intervention of interest (yes/no), just as RCT participants are, but without the prior screening with eligibility criteria used to enroll a more homogeneous patient population in an RCT.⁴ In an observational study, proportionately more patients classified as not having received the intervention would have been ineligible for RCT enrollment than participants classified as having received the intervention. Observational study patients who did not receive the intervention of interest typically have insufficient or even contra-indications for the intervention and are thus not comparable to the observational study patients who received the intervention. Therefore, categorizing by intervention in an observational study will yield biased effect estimates. In this context, indication bias is a type of selection bias.

For studies of the comparative effectiveness of different fluid infusion protocols in seriously injured trauma patients, there are no established or universally accepted measures to quantify blood loss or the severity of continuing hemorrhage.^5,6 To compensate for the lack of quantitative metrics for bleeding severity, a single binary surrogate, massive transfusion intervention (MT), became entrenched in the trauma literature.⁷ MT has been defined as receiving 10 or more units of red blood cells (RBCs) within 24 hours of admission.³ There is growing recognition of the pitfalls associated with the use of MT as a surrogate for bleeding severity, and the need to replace this poor proxy.⁸ A recent suggestion recommends the use of RBC transfusion rates (i.e. 3 units of RBCs per hour) to “eliminate survival bias.”⁸ However, until valid bleeding severity measures that precede the infusion of any resuscitation fluids become available, the potential for bias will persist.

In trauma research, MT is typically used as a surrogate either 1) as the outcome or “gold standard” in evaluating the diagnostic value of early predictors of seriously bleeding patients in need of massive transfusion,⁹ or 2) as a stratification variable to account for potential confounding or effect modification when comparing the effectiveness of different resuscitation protocols.^{3, 10–12} Each of these distinct purposes for MT introduces a slightly different type of bias. In predictive MT algorithm studies, many seriously bleeding trauma patients are effectively misclassified as not receiving (sometimes incorrectly called ‘not needing or requiring’) an MT only because they either died or achieved hemostasis before the threshold unit of RBCs could be transfused. This misclassification may be one reason for the relatively modest performance of predictive MT algorithms⁹ and the persistent variability in clinical practice regarding MT protocols.^{13, 14}

Stratification of a study population by MT/non-MT status is commonplace in observational studies, both retrospective¹⁵ and prospective³. Stratification by MT has been so accepted as standard trauma research practice that it has been used even in RCTs^{10, 11} leading in one case to early termination of the trial¹¹. However, MT stratification inevitably introduces indication and survival bias (see Sin 2 below) into data analyses. To be classified as MT, patients must survive long enough to receive the threshold number of RBC units within the specified post-admission timeframe. To be classified as non-MT, patients must either 1) survive the entire period never exceeding the blood product threshold (i.e., the least extreme subgroup), or 2) die early before the threshold unit of RBCs could be transfused (i.e., the most extreme subgroup). Combining these two opposite extremes of bleeding severity into the non-MT subgroup reveals how MT classification actually defeats the purpose of stratification: to form more homogeneous patient subgroups.

To reduce the bias in the gold standard used to assess performance of predictive MT algorithms, the gold standard definition should encompass the full spectrum of outcomes within 24 hours of admission among severely bleeding trauma patients. It may help to redefine the conventional MT surrogate for this gold standard outcome as “the patients who needed activation of the blood bank’s Multiple Transfusion or MT protocol.” Under this broader definition, a composite variable could be established to include all the different types of patients who were at increased risk of death within 24 hours of admission due to hemorrhage. For example, a more appropriate composite variable could include patients who, within 24 hours of admission: 1) died early from hemorrhage-related causes, 2) received several units of blood products within a short timeframe (e.g. ≥ 3 units within one hour) and achieved hemostasis only after surgical or other interventions, or 3) received at least the specified threshold units of blood product. This broader composite definition would be more capable of accurately identifying the set of earliest ascertainable patient characteristics that most strongly indicate the need to activate a blood bank’s MT protocol (i.e., the best performing predictive algorithms).

To reduce indication bias in analyses relating resuscitation interventions to survival and other outcomes, variables chosen for stratification or adjustment should be restricted to baseline (pre-intervention) patient characteristics only. These baseline variables should include the most important indications for the intervention that are known to significantly influence patients’ outcomes. In the highly dynamic trauma resuscitation research setting, accounting for the timing and sequence of patient injury characteristics, diagnostics, interventions, intermediate endpoints and final outcomes is critical. Special precautions must be taken to avoid stratifying or adjusting analyses on variables intermediate in a causal pathway between the intervention and the outcome of primary interest (i.e., mediators or “colliders”).^16–18 For example, blood transfusions after admission can be intermediate in the causal pathway between pre-hospital crystalloid infusions (as the intervention of interest) and mortality (as the outcome of interest). While blood transfusions could be examined as an intermediate endpoint and secondary outcome of interest in a study assessing the effect of pre-hospital crystalloids on mortality, blood transfusions would be inappropriate stratification or adjustment variables, even in RCTs. On the other hand, in any study assessing the effect of novel in-hospital blood transfusion protocols on mortality, pre-hospital infusion of crystalloid could be an important covariate for adjustment. Once the set of key patient characteristics and indications have been identified, propensity score matching can be added to the overall analysis strategy to reduce indication bias.^{4, 19} However, recent evidence suggests that propensity score matching may not be superior to standard multivariable regression modeling in reducing indication bias, and that both adjustment techniques are equally susceptible to residual or unmeasured confounding (see Sin 7 below).²⁰ Clearly, there is no satisfactory substitute for complete data capture and attention to the timing and sequence of major indications, interventions and outcomes in trauma research.²¹

Sin 2- Ignoring Survival bias

Survival bias is a selection bias that occurs when comparing patient groups in which patients may die before treatment is initiated.²² This selection bias is compounded when the treatment is not constant and therefore changes over time (e.g., transfusion of blood products, see Sin 3 below). Survival bias is common in clinical studies,²³ but is potentially worse in trauma studies due to early and high mortality rates. In other clinical domains, patients die over the course of weeks, months or years; however in trauma, patients die over the course of minutes, hours or days, increasing the likelihood for survival bias. Because of this early mortality, it is difficult to validly compare “treated” and “untreated” patient groups even in a cohort study such as PROMMTT. RCTs are usually designed to limit survival bias, however many patients may die before being randomized, resulting in a study population that is very different from the general trauma population as a whole and thus limiting generalizability. If not well-designed, secondary analyses of RCTs may also introduce survival bias.

Survival bias may result in “reverse causation” which occurs when an association between treatment and outcome is found and thought to be causal, when in fact the outcome could have preceded exposure to the treatment because of survival bias. In trauma transfusion studies, reverse causation can explain an observed association between high plasma or platelet ratios and survival because the longer patients survive, the more likely they will receive units of plasma and platelets. If this situation is not taken into account in the analysis, longer survival will appear to be associated with infusion of plasma and platelets even if longer survival actually “caused” the use of plasma and platelets.³ Appropriate analysis strategies can reduce survival bias and better estimate the true extent to which survival may be associated with a particular transfusion protocol.

Strategies to minimize survival bias include: 1) using appropriate time-dependent regression models (e.g., Cox proportional hazards models), and 2) modeling or analyzing data within appropriate time intervals, conditional on survival. Patient treatment groups defined by a cumulative experience over time (e.g., total amount of fluid infusions over 24 hours) should not be compared if any patients died during the time period. For example, comparing cumulative transfusion ratios at 24 hours when some patients have already died prior to 24 hours will result in survival bias. Stratification or modeling using variables such as the conventional MT definition that are defined by a survival bias (see Sin 1 above) will introduce survival bias into a study that had none. Several manuscripts in the trauma literature discuss these issues.^24–27

The following hypothetical example using PROMMTT data³ assumes 30-day survival and equal allocation to three different transfusion ratio groups. Logistic regression and Cox time-dependent models were adjusted for the same set of covariates: age, injury severity score, time interval at cohort entry (receipt of third unit), bleeding site (head, trunk, or limb), and sum of transfusion at cohort entry (for Cox regression) or 24 hours (for logistic regression). Adjusted summary mortality risk ratio estimates from the time-dependent Cox and logistic regression models are presented in Table 1. Despite adjusting for the same set of covariates, survival bias caused the logistic regression analysis to over-estimate the potential survival benefit for the 1:1:1 group by 13–16 fold compared with the estimates from Cox regression models that accounted for time-varying transfusion ratios. Survival bias was introduced by relating cumulative 24-hour transfusion ratios to mortality using logistic regression that ignores time of death and the occurrence of many early deaths before the higher transfusion ratios (plasma:RBC and platelet:RBC) or hemostasis could be attained. Thus, time-dependent data like transfusion ratios should be modeled using appropriate time-dependent regression models (e.g., Cox regression models).^{3, 24}

Table 1.

Cox time-dependent hazard ratios andlogistic regression odds ratios to estimate the magnitude of survival bias using data from PROMMTT

Plasma:Platelet:RBC ratio	6 hour mortality	24 hour mortality	30 day mortality
<1:1:2 ratio	Referent RR=1	Referent RR=1	Referent RR=1
~1:1:2 ratio	0.344/0.152 = 2.26	0.379/0.272 = 1.39	0.661/0.344 = 1.92
~1:1:1 ratio	0.099/0.006 = 16.50	0.145/0.009 = 16.11	0.289/0.023 = 12.56

Sin 3- Ignoring Time-varying Treatment

Variability in how patients receive blood products is dependent on patient needs, physician ordering practices, availability, and small differences in volumes of products. No two patients get products in the exact same number and sequence over time, even in a clinical trial. Therefore, the sum of products within time intervals cannot distinguish between patients who receive the same number of products but with different patterns over the same time period.

For example, Table 2 shows instantaneous and cumulative blood product ratios (plasma:platelets:RBC) provided to different hypothetical patients. All patients have the same cumulative ratio over 75 minutes, but the patterns of transfusion (instantaneous ratios) were very different among patients. Patient A received platelets early and nearly equal amounts of plasma and RBCs throughout the study. Patient B received all RBCs, plasma, and platelets at the end of the interval. Patient C received RBCs early, plasma later, and platelets at the end of the interval.

Table 2.

Comparison of instantaneous and cumulative blood product ratios (plasma:platelets:RBC) for three hypothetical trauma patients.

	Patient A		Patient B		Patient C
Time	Instant	Cumulative	Instant	Cumulative	Instant	Cumulative
15 m	1:6:1	1:6:1	0:0:0	0:0:0	0:0:6	0:0:6
30 m	2:0:2	3:6:3	0:0:0	0:0:0	0:0:0	0:0:5
45 m	1:0:1	4:6:4	0:0:0	0:0:0	2:0:0	2:0:5
60 m	1:0:1	5:6:5	0:0:0	0:0:0	2:0:0	4:0:5
75 m	1:0:1	6:6:6	6:6:6	6:6:6	2:6:1	6:6:6

These differences in treatment may reflect differences in indication and thus may introduce indication bias as well as survival bias if not appropriately accounted for in the analysis. These patient-specific differences should be incorporated into a strategy like time-dependent Cox modeling to minimize these sources of bias. Covariates can also be time varying and require appropriate strategies for valid analysis. For example, in PROMMTT, the cumulative count of blood product transfusions was included as a time varying covariate for each of the 14 time intervals in the time-dependent Cox regressions.³

Sin 4- Ignoring Time-Dependent Confounding

When interventions are time varying, any treatment can be influenced by the preceding treatment, and may also influence the need for subsequent treatment as well as the outcome. Such a situation may involve time-dependent confounding requiring appropriate consideration in the research design or data analysis. In the presence of serious time-dependent confounding, even time-dependent Cox models may be inadequate, and more sophisticated statistical approaches like marginal structural modeling may be necessary.²⁸ Using Cox time-dependent adjustment to compute an overall summary effect measure in the presence of serious time-dependent confounding can actually introduce a selection bias.

The use of statistical approaches like marginal structural modeling in the trauma resuscitation literature is rare. Aspects of trauma resuscitation that may require special adaptations of this approach include the compressed timeline over which combinations of multiple, rapid-paced interventions are administered sequentially and differentially depending upon rapid changes in patients’ conditions and the markedly decreasing risk of injury-related death with increasing time from admission. A recent articles from the critical care literature²⁹ provides an example of how the marginal structural modeling approach has been applied in a clinical intervention context similar to trauma.

Sin 5- Assuming uniform effects over time

The dramatic change in the risk of mortality over the extended course of hospitalization for severely injured trauma patients is well recognized.³⁰ The risk factors for early mortality (e.g., due to exsanguination or coagulopathy) differ significantly from those for later mortality (e.g., sepsis or Acute Respiratory Distress Syndrome (ARDS)) Many trauma research design and analysis approaches ignore the fact that relevant risk factors (e.g., shock) take time to contribute to late mortality. Only patients surviving the initial 24 hours after injury go on to risk death from sepsis, ARDS or multi-organ failure (MOF), and patients in shock may require early transfusion to survive long enough to be at risk of these late complications. In other words, trauma patients experience competing risks from different causes of mortality over time.³¹ To assess the extent to which varying transfusion protocols influence risk of later complications like ARDS, independently of the indications for transfusion and other interventions, would require an observational study with complex modeling, if not a well-designed RCT.

Interventions like transfusion may also have differential effects on early versus late mortality. These non-uniform effects over time reflect effect modification. Even sophisticated modeling for time-dependent confounding (e.g., marginal structural modeling) will not yield valid effect estimates in the presence of significant effect modification unless appropriate interactions are investigated.²⁸ Alternative methods include stratification by time in clinically meaningful intervals. It is essential to test first for significant heterogeneous effects across strata before performing any overall summary adjustment (i.e., standard covariate adjustment in linear or logistic regression).

An example from PROMMTT (Table 3) illustrates this issue of non-uniform effects over time. The three time-specific death rates each differ by an order of magnitude. Additionally, the primary causes of death differ markedly in each of the three time intervals. Due to these nonuniform (heterogeneous) risks of cause-specific mortality, interventions can be expected to have differential effects on survival and complications across time. This situation was also suggested by the PROMMTT Cox time-dependent analyses^{3, 31} in which effect estimates were significantly heterogeneous across time. Transfusion ratio differences were most strongly associated with deaths within 6 hours (i.e., predominantly hemorrhage-related deaths), but were less associated with death in the subsequent time intervals up to hospital day 30 when the non-hemorrhagic causes of death were more common. For this reason, the analyses were presented stratified by the three time intervals.

Table 3.

Death rates from the PROMMTT study

Time interval after ED admission	Deaths	Hours at Risk	Mortality Rate
Within 6 hours	88	3,590	0.0245
From >6 hours to 24 hours	34	14,039	0.0024
From >24 hours to 30 days	84	491,618	0.0002

Sin 6- Assuming missing values are missing at random

Handling missing data in trauma research is a well-recognized problem.^{32, 33} In statistical parlance, there are three distinct mechanisms underlying missing data values: Missing Completely at Random (MCAR), Missing at Random (MAR) and Missing Not at Random (MNAR).^{34, 35} When data sets containing missing values are analyzed, one of these three mechanisms is implied. Trauma outcomes research is somewhat unique in that primary outcome data (e.g., 24-hour or 30-day mortality) are rarely missing. However, important covariate data (e.g., temperature, blood pressure, Focused Assessment with Sonography for Trauma (FAST) exam, base deficit, Glasgow Coma Scale, etc.) are often missing.³⁶ Because covariate data are more often missing in the two opposite extremes of injury severity (i.e., patients dying early and patients with the least severe injuries), the missing covariate data should not be assumed to reflect an MCAR mechanism.

The typical practice in trauma resuscitation research is to perform complete case analysis, excluding patients with one or more missing covariate values, which automatically assumes the MCAR mechanism. But even a MAR mechanism may be an inappropriate assumption in trauma research. The MAR mechanism assumes that all important confounders and effect modifiers have been measured, and that sufficient surrogates for the missing covariates are available in the dataset. These conditions are usually not satisfied in trauma resuscitation data because sufficient surrogate covariates are seldom available. Multiple imputation is a statistical strategy that implies the alternate MAR mechanism. Missing data values that could be accurately imputed only by variables not measured in the data set fall under the MNAR or non-ignorable missing mechanism. Distinguishing between MAR and MNAR mechanisms for missing data can be difficult if not impossible.³⁷ Simulations and sensitivity analysis are sometimes used to address the potential for MNAR in analyses that apply multiple imputation.

Challenges due to missing values in trauma research are in need of innovative developments. Examining the bounds between best- and worst-case imputation scenarios could be helpful, as was shown in another manuscript published in this supplement.³⁸ Another promising direction is the development and improved documentation of standardized pre-hospital indicators of injury and bleeding severity that can be used in analyses of trauma data as important covariates and appropriate surrogate variables.³⁹

Sin 7- Imperfectly defined variables

Defining study variables to represent the real construct of interest is challenging in trauma research. Variables must be defined appropriately in the research design phase, measured without error in data collection, and coded appropriately to support the most valid analysis. Imperfectly defined variables are often used as surrogates to substitute for a difficult to measure or document factor of interest. Imperfectly defined variables are a particular problem in trauma research because the surrogate(s) may inadvertently introduce rather than minimize bias. For example, the surrogate may actually be a consequence rather than a strong correlate of the factor (e.g., MT when used as a surrogate for bleeding severity, see Sin 1).

Imperfectly defined variables may also cause residual confounding, a distortion that remains after adjustment for potential confounders. The causes of residual confounding include over-simplification of a variable in an analysis model (e.g., dichotomizing of continuous variables), imperfectly measured variables (e.g., injury severity scores that are censored by death and dependent upon survival to accurate diagnosis) used as covariates in a model, or unmeasured confounders that were not considered. Imperfectly defined variables can also result in switchover effects, when the unadjusted (or less adjusted) effect is in the direction opposite of the adjusted (or more adjusted) effect.

One example of switchover effect can be seen in a recent study examining the association between transport type (helicopter versus ground) and survival for adults with major trauma (Table 4).⁴⁰ In this study, the unadjusted odds ratio (OR) was less than 1, suggesting that helicopter transport reduced survival compared with ground transport. After adjustment the OR was greater than 1, suggesting that helicopter transport increased survival compared to ground transport. In this case, the switchover effect was likely due to the higher prevalence of severely injured patients among those transported by air compared to those transported via ground.

Table 4.

Odds ratios for the comparison of air to ground transport for survival of adults with major trauma from Galvagno, et al (2012)

	Odds Ratio	95% CI	P-value
Unadjusted	0.88	0.85–0.90	<0.001
Adjusted*	1.31	1.27–1.38	<0.001

^*Adjusted for systolic blood pressure, respiratory rate, heart rate, motor component of GCS, mechanism of injury derived from ICD-9-CM e-codes for primary and secondary diagnoses, age, sex, type of trauma (blunt vs. penetrating), and transport mode

Switchover effects are relatively unusual outside of trauma and typically occur when a covariate (patient/injury characteristic) is associated with the intervention of interest in a direction opposite its association with the outcome. Switchover effects can signal residual confounding, effect modification or even reverse causation and cause difficulties for analysts who do not know the direction of the true relative risk. There is no obvious solution for switchover effects, other than having available better definitions (including timing and sequence) of the variable(s) representing the construct of interest. For observational trauma research in general and for switchover situations in particular, an appropriate strategy includes cautious interpretation of results following a concerted effort to reduce all sources of bias. A system capable of promoting more RCTs in the trauma setting with independent data coordinating centers would be especially beneficial.

CONCLUSION

Recognition of the need to practice evidence-based medicine has increased in the past ten years. Unfortunately, the evidence for trauma care is based largely on retrospective data rather than well-designed prospective studies or RCTs, and the basis for clinical decision making in trauma care rests largely on conjecture rather than well-wrought evidence from a consistent body of literature. The quality of evidence from trauma research can be improved with multi-disciplinary teams representing clinical, epidemiologic, statistical, informatics and other expertise as needed to focus on optimal study design, operationalization of the study methods, data analysis, and interpretation. We advocate for more high-quality research addressing relevant clinical problems in trauma. More rigorous epidemiologic and statistical strategies can help provide trauma research teams the protective armor to avoid the seven deadly sins and increase the scientific and public health impact of comparative effectiveness research in trauma.