Facilitating transparency in spinal cord injury studies using data standards and ontologies

Extract

Progress in developing robust therapies for spinal cord injury (SCI), traumatic brain injury (TBI) and peripheral nerve injury has been slow. A great deal has been learned over the past 30 years regarding both the intrinsic factors and the environmental factors that regulate axon growth, but this large body of information has not yet resulted in clinically available therapeutics. This therapeutic bottleneck has many root causes, but a consensus is emerging that one contributing factor is a lack of standards for experimental design and reporting. The absence of reporting standards, and even of commonly accepted definitions of key words, also make data mining and bioinformatics analysis of neural plasticity and regeneration difficult, if not impossible. This short review will consider relevant background and potential solutions to this problem in the axon regeneration domain.

Lack of reproducibility

In 2003, the U.S. National Institute of Neurological Disorders and Stroke (NINDS) initiated a groundbreaking project to replicate important studies in the SCI field. The “Facilities of Research Excellence—Spinal Cord Injury” (FORE—SCI) project was designed to explore two issues; the failure of basic science breakthroughs to lead to successful therapeutics and the perceived lack of robustness of findings reported in many high impact papers (Steward et al., 2012). Three SCI centers undertook replication studies to validate interesting results from the original literature. Publications from the University of California at Irvine (Sharp et al., 2012; Sharp et al., 2013; Steward et al., 2006; Steward et al., 2012; Steward et al., 2008; Nielson et al., 2011), the University of Miami (Marcillo et al., 2012; Pinzon et al., 2008; Pinzon et al., 2008), and the Ohio State University (Popovich et al., 2012a; Popovich et al., 2012b) revealed a surprisingly high failure rate in confirming the original studies. In twelve replication experiments, six failed to replicate, four gave partial replication, one was inconclusive and one succeeded but only after three attempts. In a summary of the project the team leaders identified a number of potential explanations for the failures (Steward et al., 2012). If we ignore explanations such as the original results being statistical outliers or scientific misconduct, many of the potential explanations concerned variation in animals and animal care (sources, strains, housing, drugs, handling), injuries (different injury devices, different surgical methods), and differences in reagents (different drug lots, different sources for cell therapies). A major conclusion was that method sections in papers in most journals currently are incomplete, making effective replication studies impossible without direct communication with the original authors.

While the FORE-SCI project was remarkable for its vision and comprehensive investigation of issues related to lack of reproducibility in SCI studies, it is not the only project of this type. The drug discovery industry depends on basic science research, especially that done in academic laboratories, for the identification of many of its therapeutic targets. Scientists at Bayer Healthcare routinely validate studies from the original literature prior to launching a drug discovery campaign. They found in 61 internal projects that only 21% of original studies could be replicated (Prinz et al., 2011). Similarly, scientists at Abbot could reproduce only 11% of landmark studies (Begley et al., 2012). Both groups called for better scientific practices in basic research labs.

Investigator bias

While it is usually difficult to prove in individual cases, investigator bias, including inadvertent and unrecognized bias (Ransohoff et al., 2010), is widely believed to contribute to the reproducibility problem. To overcome investigator bias, leaders in neuroscience recently argued for more rigorous standards in study design and reporting (Landis et al., 2012). Key recommendations included randomization of animals to different study groups and in data collection; blinding investigators, especially those doing assessment or data collection and analysis, to treatments; sample-size estimation; and improved data handling. Moreover, these aspects of a study should be addressed in grant proposals and manuscripts submitted for publication.

Reporting standards

Curiously, while translational and clinical researchers have long had specific recommendations and standards for conducting and reporting research with humans, such standards do not exist in most basic science fields. For example NINDS has extensive recommendations/requirements about common data elements (CDEs) used in clinical research for stroke and TBI (http://www.commondataelements.ninds.nih.gov). Worldwide leaders in the SCI field have proposed design and reporting standards for SCI clinical trials (Anderson et al., 2005; Kwon et al., 2011). Neural Regeneration Research has adopted this idea and has well developed guidelines for authors publishing clinical trial data. However, similar reporting guidelines for animal SCI studies do not yet exist.

Basic science reporting standards are often developed by professional organizations. A well-established example is the data file standard for fluorescent activated cell sorting developed by the International Society for Advancement of Cytometry. In other cases, ad hoc working groups develop such standards. A framework to promote this approach was put in place by the Minimal Information for Biological and Biomedical Investigations (MIBBI) project (Taylor et al., 2008). Perhaps the most widely used MIBBI guideline is the Minimum Information About a Microarray Experiment (MIAME), which established reporting standards for microarray experiments. MIAME established recommendations for describing raw data, reporting how the data were normalized, how samples were prepared, and details of the experimental design (Brazma et al., 2001).

An SCI reporting standard

In 2011 a group of scientists in the U.S. and Japan began planning a project to develop reporting standards for animal studies related to SCI. Thirty-five scientists held a 3 day meeting in New Orleans in the Fall of 2012 titled: Growth Cones and Axon Regeneration: Entering The Age of Informatics. There were a number of scientific talks, and breakout sessions were held to develop a proposal for Minimal Information About a Spinal Cord Injury experiment (MIASCI). Because of the very wide-ranging way SCI experiments are done the MIASCI draft includes a relatively large number of metadata elements (>250, examples in Table 1) that could limit acceptance by authors, editors and reviewers. Many of these elements concern things like animal source and strain, housing conditions, and experimental design issues such as whether the investigators were blinded to treatment conditions. The majority covers various kinds of treatments that are unlikely to be used simultaneously in a given study. For example, it is unlikely, at present, that a study involving a cell therapy would also involve screening a siRNA library AND a compound library. Consequently, the number of data elements that need to be annotated for a given SCI study should be in line with other reporting standards. Nonetheless, it is clear that for the MIASCI standard to be widely adopted and used by authors and annotators, a MIASCI reporting tool will need to be developed that provides simple entry using standardized terminologies, provides definitions and eliminates repetitive entry of required information. Output from such a tool should be both human- and machine-readable, and available in a variety of formats.

Table 1.

Representative examples of data elements in the draft MIASCI

Having SCI researchers report the metadata from experiments along with the data will go a long way toward achieving the transparency of research recommended by Landis et al. (2012). As members of the SCI community become accustomed to reporting issues related to study design (blinded assessment of outcome measures, power analysis to determine sample size, exclusion criteria) they will inevitably adopt more rigorous approaches to the research they conduct. An additional major impact on the SCI domain will be to facilitate data mining and bioinformatic analysis. As the amount of SCI-related information published each year becomes more and more overwhelming, this will become increasingly important.

The SCI data flood

PubMed indexes over 20,000 papers on SCI with 1,723 having been published in 2012. Related topics such as neuroprotection, axon regeneration, axon growth and nerve regeneration account for more than 40,000 additional papers. Scientists cannot digest that much information and, at present, computers can’t either.

Computer scientists specializing in artificial intelligence have developed impressive tools for analyzing structured data of the kind found in relational databases and unstructured data such as that found in the corpus of a particular field. Relational databases are extremely efficient at storing and retrieving data but need to be carefully designed at the beginning and are not easily adapted to incorporate new types of data. They also typically require that the data be stored in a single storage system. To overcome these limitations and to enable the use of data spread across the internet the concept of the “semantic web” was developed (Berners-Lee et al., 2001), along with languages such as Extensible Markup Language (XML), Resource Description Framework (RDF), and Web Ontology Language (OWL). For this latter approach to work ontologies need to be developed that identify key concepts in a particular field, provide definitions and relationships among the concepts and give various synonyms for a given concept. With appropriate ontologies in place large text mining projects can be performed to build a knowledge base. If the ontologies are created with expressive descriptive logic to define relationships among concepts, description logic reasoning engines can be used to answer complex queries made against the knowledge base. Finally, semantic web approaches allow queries across dispersed resources such as the Gene Ontology and the Chemical Entities of Biological Interest (ChEBI) provided that the data are annotated with concepts from ontologies (Shah et al., 2009; Callahan et al., 2011).

The development of an ontology for a particular research area, such as SCI, necessarily requires a combination of text mining to identify recurring terms and phrases as well as input from domain experts who can provide definitions for concepts, identify synonyms and establish relationships among these concepts. Having computer scientists involved is also essential as they develop the logical framework and tools to make the ontology and related knowledge base useful. Finally, members of the scientific community need to participate by providing input as to the type of questions they would like to ask (so-called “use cases”) and how they would like output to be visualized.

We (V.P.L., J.L.B. and U.V.) are leading a team of neuroscientists, informaticists and computer scientists at the University of Miami and Stanford University to develop an ontology about axon regeneration, called the RegenBase Ontology. The long-term goal of this project is to connect information about SCI experiments to information about biological processes, molecular networks and high-throughput screening data to speed the identification and testing of novel therapeutics. The metadata and data that can be obtained from well-annotated research articles are critical to developing a useful web-based resource for the SCI community. Obtaining input from SCI researchers worldwide regarding MIASCI and use cases for RegenBase is critical to the success of this project. An example of a use case using a draft RegenBase ontology is shown in Figure 1.

Figure 1.

A query example using the RegenBase Ontology.

The T-Box includes concepts, such as “compound” and “regeneration”, and relationships, such as “promotes” and “develops in vivo”. When data from real world experiments are incorporated into the A-Box of the ontology, it is possible to ask questions such as “What compounds that inhibit cPKC promote regeneration of proprioceptive DRG neurons in vivo?”. Using a Descriptive Logic (DL) reasoning engine, we found that a compound, Gö6976, satisfied this logical query.

We were able to successfully query the knowledge base for protein kinase C inhibitors that promote regeneration of sensory neurons. Another interesting question for stakeholders, for example, is “would a Chinese language version of MIASCI or of RegenBase facilitate use and adoption?”. In addition, collaboration with journal editors and publishers needs to be explored. Cooperation of these constituencies is critical for the development of true transparency in SCI research.