Reprint: Good laboratory practice: preventing introduction of bias at the bench

Abstract

As a research community, we have failed to show that drugs, which show substantial efficacy in animal models of cerebral ischemia, can also improve outcome in human stroke. Accumulating evidence suggests this may be due, at least in part, to problems in the design, conduct, and reporting of animal experiments which create a systematic bias resulting in the overstatement of neuroprotective efficacy. Here, we set out a series of measures to reduce bias in the design, conduct and reporting of animal experiments modeling human stroke.

Nearly 10 years after the first Stroke Therapy Academic Industry Roundtable (STAIR) participants established guidelines intended to support the translation of neuroprotective efficacy from bench to bedside (Altman et al, 2001), there is still no clinically effective neuroprotective drug for stroke. One interpretation of this observation is that measures outlined in STAIR I have failed to deliver the promised improvements in drug development. However, a dispassionate analysis of data presented over the last 10 years suggests that the ‘STAIR hypothesis’—that improvements in animal experimental design will lead to improvements in translational efficiency—has yet to be adequately tested. Adhering to standards of conducting and reporting of experiments to reduce the confounding effects of bias and ensure adequate statistical power, as outlined below, will increase the confidence with which we can assess new data and maximize our chances of developing effective therapies.

The original STAIR proposal was that by paying due attention to experimental bias, to the breadth of physiologic variables known to influence stroke outcome in patients, and by testing therapies in a range of model systems which might more faithfully reproduce the key facets of stroke pathophysiology, we would be able to translate what appeared to be clear evidence of neuroprotective efficacy in animals to the more heterogeneous circumstances of human stroke. Although we believe strongly that failure to adequately consider variables such as age, comorbidity, physiologic status, and timing of drug administration contribute to the disparity between the results of animal models and clinical trials, they have been reviewed elsewhere (Altman et al, 2001; Bath et al, 1998) and are not the subject of this article.

Analyses of data supporting the efficacy of various neuroprotective strategies (Begg et al, 1996; Crossley et al, 2008; Dirnagl, 2006) have revealed that although many researchers adhere closely to the ethos of these guidelines, as a community we do not. A simple checklist derived from the STAIR guidelines to provide an overview of the range of data available for 1,026 candidate therapies (Crossley et al, 2008) revealed that only a few came close to meeting the STAIR guidelines. A higher score against this checklist was accompanied by a marked reduction in effect size. This later trend could be seen clearly even within the data for individual drugs (Grotta, 1995). Moreover, studies which reported measures to avoid bias such as random allocation to treatment group, masked induction of ischemia, or the masked assessment of outcome (Macleod et al, 2005, 2008), gave a markedly lower estimate of efficacy. Despite this there has been some evidence of improvement in study quality, and the performance of animal stroke studies is substantially better than that for most other models of neurologic disease (Dirnagl, 2006). And yet, the majority of investigators still do not report whether they took measures to avoid bias.

Systematic reviews and meta-analyses of data from animal stroke studies suggest that these studies may be substantially distorted by experimental bias. Taken together, publications supporting the efficacy of NXY-059 include randomized data with allocation concealment and masked outcome assessment, but most individual publications do not report these measures. Analyses of those data suggest that at least half of the reported 44% improvement in outcome could be attributed to experimental bias, specifically a failure to randomize the allocation to experimental group, a failure to conceal treatment group allocation from the surgeon or a failure to blind the assessment of outcome (Macleod et al, in press). Similar observations have been made of the hypothermia literature, where nonrandomized studies and studies without masked outcome assessment appear to give a relative overstatement of efficacy of 27% and 19%, respectively (Macleod et al, 2005). Despite the widely recognized importance of these aspects of study design, analyses conducted by the collaborative approach to meta analysis and review of animal data from experimental stroke (CAMARADES) group suggest that only 36% of studies reported random allocation to treatment group, only 11% report allocation concealment, and only 29% reported the masked assessment of outcome (Dirnagl, 2006).

A related issue is the number of animals used in experiments. The probability of detecting a difference of a given size between groups is related to the number of animals in each group, the size of the difference and the variability in the outcome measure used. However, only 3% of studies identified in systematic reviews reported using a sample size calculation (Dirnagl, 2006). Importantly, if sample size calculations are based on falsely large estimates of effect size, studies will not be powered to detect real differences between treatment and control groups. Indeed, post hoc analysis suggests that most experimental stroke studies have only a one in three chance of detecting a 20% difference in outcome.

These problems are not unique to the preclinical study of stroke. Clinical stroke trials have had problems with inadequate sample size (O’Collins et al, 2006) and have also failed to report whether they took measures to avoid bias (Plint et al, 2006). Indeed Cochrane’s observation that ‘when humans have to make observations there is always the possibility of bias’ (Sena et al, 2007) was a lynchpin of the CONSORT (Consolidated Standards of Reporting Trials) initiative to improve the reporting, design, conduct, analysis, and interpretation of randomized controlled trials to inform decision making in health care (Sena et al, 2007; STAIR, 1999). This initiative led to substantial improvements in the reporting and conduct of clinical trials (van der Worp et al, 2007).

On the basis of the available evidence it would now seem reasonable to suggest that preclinical testing in animal models of stroke, and indeed other models of disease, should adopt similar standards to ensure that decision making is based on high quality unbiased data (Dirnagl, 2006; Weaver et al, 2004). Adoption of such standards would have the added benefit of reducing wasteful usage of financial and animal resources.

In general, studies should only be considered for publication if their ‘Methods’ section includes a description of how they have addressed the standards below, or if authors make a cogent argument for why these standards are not relevant to their work. For these components of a paper, citation of methods described in previous publications is not considered sufficient. These requirements should not preclude publication of important observational, pilot or hypothesis-generating data, but the conclusions of such studies should reflect their preliminary nature.

(1) Animals

The precise species, strain, substrain, and source of animals used should be stated. Where applicable (for instance in studies with genetically modified animals) the generation should also be given, as well as the details of the wild-type control group (for instance littermate, back cross, etc.).

(2) Sample size calculation

The manuscript should describe how the size of the experiment was planned. If a sample size calculation was performed this should be reported in detail, including the expected difference between groups, the expected variance, the planned analysis method, the desired statistical power, and the sample size thus calculated. For parametric data, variance should be reported as 95% confidence limits or standard deviations rather than as the standard error of the mean.

(3) Inclusion and exclusion criteria

Where the severity of ischemia has to reach a certain threshold for inclusion (for instance a prespecified decrease in perfusion detected with laser–Doppler flowmetry, or the development of neurologic impairment of a given severity) this should be stated clearly. Usually, these criteria should be applied before the allocation to experimental groups. If a prespecified lesion size is required for inclusion this should be detailed, as well as the corresponding exclusion criteria.

(4) Randomization

The manuscript should describe the method by which animals were allocated to experimental groups. If this allocation was by randomization, the method of randomization (coin toss, computer-generated randomization schedules) should be stated. Picking animals ‘at random’ from a cage is unlikely to provide adequate randomization. For comparisons between groups of genetically modified animals (transgenic, knockout), the method of allocation to for instance sham operation or focal ischemia should be described.

(5) Allocation concealment

The method of allocation concealment should be described. Allocation is concealed if the investigator responsible for the induction, maintenance, and reversal of ischemia and for decisions regarding the care of (including the early killing of) experimental animals, has no knowledge of the experimental group to which an animal belongs. Allocation concealment might be achieved by having the experimental intervention administered by an independent investigator, or by having an independent investigator prepare drug individually and label it for each animal according to the randomization schedule as out-lined above. These considerations also apply to comparisons between groups of genetically modified animals, and if phenotypic differences (e.g., coat coloring) prevent allocation concealment this should be stated.

(6) Reporting of animals excluded from analysis

All randomized animals (both overall and by treatment group) should be accounted for in the data presented. Some animals may, for very good reasons, be excluded from analysis, but the circumstances under which this exclusion will occur should be determined in advance, and any exclusion should occur without knowledge of the experimental group to which the animal belongs. The criteria for exclusion and the number of animals excluded should be reported.

(7) Masked assessment of outcome

The assessment of outcome is masked if the investigator responsible for measuring infarct volume, for scoring neurobehavioral outcome or for determining any other outcome measures has no knowledge of the experimental group to which an animal belongs. The method of blinding the assessment of outcome should be described. Where phenotypic differences prevent the masked assessment of for instance neurobehavioral outcome, this should be stated.

(8) Reporting potential conflicts of interest and study funding

Any relationship which could be perceived to introduce a potential conflict of interest, or the absence of such a relationship, should be disclosed in an acknowledgements section, along with information on study funding and for instance supply of drugs or of equipment.

We consider that these measures are of central importance to Good Laboratory Practice in the modeling of cerebral ischemia. Many groups already perform experiments to these high standards, and we hope that they will now report this in full, and that others follow their lead. Finally, we do not consider these requirements to represent a final or complete list of appropriate measures necessary to avoid bias. Future additions may be required as further evidence emerges and the experience of authors and reviewers evolves.