When replication teaches more than the original experiment — the saga of the unknown unknown

Abstract

We recently published a description of the molecular mechanism involved in “progressive hemorrhagic necrosis”, a pathological process that evolves during several hours after spinal cord injury, that is attributable to progressive capillary fragmentation, and that is due to upregulation and activation of SUR1-regulated channels in microvascular endothelium. In this commentary, we reflect on the independent replication of our original experiment by Dr. Phillip Popovich and colleagues, and how their initial attempt at replication led to the unexpected finding that anisotropy of spinal cord tissues strongly influences the patterns of both primary and secondary hemorrhage that are observed after impact injury to the spinal cord.

Replication is a core principal of the scientific method. To establish validity, the results of an experiment performed by one group of scientists must be evaluated by an independent group of scientists. The second group attempts to repeat the experiment of the first group, based on the original description. If the same experiment is repeated in detail, and if the outcomes are similar, replication has been achieved and the first experiment is validated.

In this issue, Popovich et al. (2011-this issue) report on their replication of our experiment (Simard et al., 2007). Replication based on the original description requires careful preparation. Popovich et al. wisely understood that the published description of our original experiment might well omit some of the detailed minutia required for accurate replication. Therefore, before any experiment was conducted, numerous efforts and conversations were devoted to clarifying and delineating experimental parameters, all in a careful attempt to faithfully replicate our original work.

After this careful preparation, the Popovich team performed the first experiment. To everyone's surprise and disquiet, their experiment failed to reproduce our original finding. After further inquiry, we realized that the specific mechanism of injury used by the Popovich team might be different from what we had used originally. To evaluate this potential difficulty, exchange visits were carried out for demonstrations and training so that the original injury could be reproduced as closely as possible. Fortunately, as Popovich et al. report, this solved the problem. We have since provided a detailed description of our method (Simard et al., 2010).

As Mr. Donald Rumsfeld famously said, “there are known knowns, known unknowns and unknown unknowns”. Replication based on the original description requires that the original experiment be understood. We realized, of course, that our injury technique led to unilateral hemorrhage, which slowly spreads to the contralateral side over the course of several hours (Fig. 1). This, after all, was the foundation for our discovery that progressive hemorrhagic necrosis arises from progressive capillary fragmentation (Fig. 2), which occurs when SUR1-regulated channels are upregulated and activated in microvascular endothelium. However, we simply assumed that any unilateral injury would produce this result. We didn't realize that some unilateral injuries could lead to immediate bilateral hemorrhage that would mask contralateral spread of hemorrhage. To our knowledge, the literature was silent on this – an unknown unknown. Together the 2 teams worked to resolve the issue, discovering in the process important consequences of anisotropy of spinal cord tissues. Our teams have gone on to do additional experiments to expand upon these findings, which we anticipate publishing soon.

Fig. 1.

Progressive secondary hemorrhage. Cross sections of rat spinal cords harvested 30 min and 24 h after a cervical hemi-cord injury, with the impact site marked by an asterisk; the rats were perfused to remove intravascular blood. These images show: (i) grossly intact spinal cords with intact pia and no mechanical disruption of tissues; (ii) primary hemorrhage at 30 min that is confined mostly to ipsilateral grey matter; (iii) primary plus secondary hemorrhage at 24 h, after the hemorrhage has expanded into contralateral grey and white matter; from Simard et al., 2010.

Fig. 2.

Capillary fragmentation linked to Sur1, the target of glibenclamide. A,B: images of the penumbra immunolabeled for laminin showing fragmentation of capillaries in rats administered scrambled oligodeoxynucleotide (SCR) (A), vs. elongated, intact capillaries in rats administered antisense oligodeoxynucleotide directed against Abcc8 (AS), the gene that encodes Sur1 (B). Bar graph: length of capillaries measured with laminin in 3 groups, as indicated; remote sections were from 7 mm proximal to the injury; *, P<0.05; adapted from Simard et al., 2010.

Does the requirement for an identical injury in the replication experiment imply that other injuries would not be benefited by treatment with glibenclamide? Not necessarily — it depends on the question being asked, and the outcome being measured. In the replication experiment, the principal outcome was spread of hemorrhage from one side to the other. If that is the outcome of importance, then the primary injury has to cause only unilateral hemorrhage, not bilateral hemorrhage. However, we have since found that even injuries that initially produce bilateral hemorrhage can be treated propitiously with glibenclamide, but to see such benefits, one needs to measure not hemorrhage acutely but neurological function over the course of several weeks.

In the end, the study performed by Dr. Popovich et al. represents the essence of the ideal replication experiment, teaching us more than the original experiment.