Exploring MEDLINE Space with Random Indexing and Pathfinder Networks

Abstract

The integration of disparate research domains is a prerequisite for the success of the translational science initiative. MEDLINE abstracts contain content from a broad range of disciplines, presenting an opportunity for the development of methods able to integrate the knowledge they contain. Latent Semantic Analysis (LSA) and related methods learn human-like associations between terms from unannotated text. However, their computational and memory demands limits their ability to address a corpus of this size. Furthermore, visualization methods previously used in conjunction with LSA have limited ability to define the local structure of the associative networks LSA learns. This paper explores these issues by (1) processing the entire MEDLINE corpus using Random Indexing, a variant of LSA, and (2) exploring learned associations using Pathfinder Networks. Meaningful associations are inferred from MEDLINE, including a drug-disease association undetected by PUBMED search.

Introduction

The integration of basic, translational and clinical science has been identified as a critical component of the effort to re-engineer the national research enterprise to address contemporary issues of health and disease [1]. The MEDLINE database contains over 16 million references to journal articles covering a broad range of content domains including life sciences, behavioral sciences, chemical sciences, bioengineering, biology, environmental science, biophysics and chemistry [2]. In many cases, full-text abstracts are freely available. Consequently MEDLINE brings together a broad range of literature from a range of disparate domains, spanning the breadth of the translational pathway from basic science to clinical research. A computational method that is able to derive connections spanning the breadth of MEDLINE's content could support the integration of these research enterprises.

Background

Over the past decade, a range of methods have been developed that acquire semantic knowledge from unannotated text in an automated manner. One such method is Latent Semantic Analysis (LSA), a corpus-based statistical method that is able to learn semantic similarity between terms from their distributional statistics across a large number of documents [3]. This method has attracted considerable attention within the cognitive science community on account of its ability to match human performance on a number of tasks requiring the assessment of the semantic similarity between terms. In particular, LSA matched the performance of the average second-language English-speaking college applicant on the Test of English as a Foreign Language (TOEFL) synonym test, which requires selecting the most similar word to a cue word from four possibilities [3]. This performance was achieved using a text corpus designed to approximate the size and content of an average American college freshman's lifetime reading. A strength of LSA is its ability to make indirect inferences: LSA is able to learn associations between terms that do not occur in the same text passage [3]. These findings invite the question: if LSA infers meaningful indirect associations from an approximation of the reading of a college freshman, what associations might it infer from a corpus with the breadth and volume of MEDLINE's abstracts?

Some technical limitations must be overcome in order to answer this question. Despite the complexity of the inferences it is able to draw, the methodology that underlies LSA can be broken down into a few simple steps: (1) A term-document matrix is constructed, in which a count is kept of the number of times every term occurs in each document in the corpus. This matrix has the dimensions t by d where t is the number of unique terms in the corpus, and d is the number of documents. Rather than using raw term frequency, a weighting scheme is often applied at this point to limit the excessive influence of frequently occurring terms and give greater emphasis to terms that occur focally in the corpus. (2) The semantic space, a reduced-dimensional approximation of the original matrix is derived. This is achieved using the Singular Value Decomposition (SVD) an established technique of linear algebra that generates the reduced-dimensional matrix that best preserves the variance of the original matrix. This matrix will have dimensionality t by k < d where t is the number of terms, and k is usually set at between 150–300 dimensions, the values determined empirically to best approximate human performance in the TOEFL test. Each term is now represented as a vector of length k. (3) Term vectors are compared to one another using a distance metric, most commonly the normalized scalar product between vectors, or cosine similarity. For a more detailed discussion of the SVD and its role in LSA the interested reader is referred to Martin and Berry [4]. This brief discussion, however, should be sufficient to reveal a fundamental limitation of LSA as it is conventionally implemented. The last release of MEDLINE contained 9,003,811 abstracts. The number of unique terms that can be derived from MEDLINE varies according to how a term is defined, but even with extremely restrictive definitions that exclude any non-alphabet characters these number around 300,000. This matrix would require approximately 2.7e¹² cells. Assuming the precision required could be obtained using 16-bit floating point values, this would require around 4500 gigabytes of memory. Furthermore, computing the SVD is in itself a CPU- and RAM-intensive process, even when using approximation algorithms on much smaller matrices.

Recently, Random Indexing (RI) [5] has emerged as an alternative to SVD for the dimension-reduction step of LSA. RI avoids generating the term-document matrix by generating reduced-dimensional term vectors directly using the following procedure: (1) Each document is assigned an index vector of zero values of length k (usually > 1000), the pre-assigned dimensionality of the reduced-dimensional matrix to be generated. (2) A small number (+–20) of these zero values are set to either 1 or –1. These non-zero values are randomly distributed across the index vector. (3) Each time a term occurs in a document, that document's index vector is added to the k-dimensional vector that represents the term. The end result is a t by k reduced-dimensional approximation of the full term-document matrix, a semantic space. For example, in the the hypothetical two-document corpus {doc1 = “antimicrobial agents”} and {doc 2 = “inhibitory agents”}, the vector for “agents” would be the vector sum of the random index vectors assigned to doc1 and doc2. RI and other similar methods are motivated by the Johnson-Lindenstrauss Lemma [6] which states that the distance between points in a vector space will be approximately preserved if they are projected into a reduced-dimensional subspace of sufficient dimensionality. The reader is referred to Sahlgren's excellent introduction [7] for further details. This procedure requires a fraction of the RAM and processing power of SVD, as it avoids generating the initial term-document matrix and uses simple addition rather than complex floating-point arithmetic. However, RI-driven LSA matched the TOEFL test performance of SVD-based LSA using the same text corpus [5].

RI allows for the derivation of a semantic space from the entire MEDLINE corpus. However, once generated, the question arises of how best to explore this considerable semantic space. A range of visualization methods have been applied in similar situations. Landauer et al create sophisticated high dimensional visualizations of document collections, concluding that only a limited representation of meaning can be displayed in three dimensions [8]. Burgess and Lund, whose Hyperspace Analogue to Language (HAL) model is related to LSA [9], use multidimensional scaling (MDS) techniques to project small groups of HAL vectors into two-dimensional space showing clustering of similar concepts [9]. Widdows and Cederberg use another round of SVD to scale vector representations of small groups of related words down to two dimensions for visualization purposes [10]. Cohen and his colleagues use force-directed graphs with terms as nodes and associative strength as edges to visualize groups of related terms [11], showing links between terms with a cosine similarity above a particular threshold. Schvaneveldt's Pathfinder algorithm provides a more sophisticated and powerful method of deriving network representations called Pathfinder Networks (PFNETS) from proximity data [12]. Pathfinder removes edges between nodes in a network when a shorter path between these nodes can be found using an alternate route through others. The Pathfinder algorithm takes two parameters: q which corresponds to the number of nodes to be traversed in search of a shorter path, and r which determines which of the Minkowski family of distance metrics is used to determine path length. PFNETS have been shown to give better approximation of local network structure than methods such as MDS which depend on global distributional statistics. In addition, this method has been successful in various applications evaluating human knowledge structures based on pairwise associations between terms [12]. This paper explores associations in MEDLINE space, a semantic space derived from the MEDLINE corpus using RI. The utility of PFNETS as a visualization tool to reveal the local structure of this space is examined using illustrative examples from clinical and basic science.

Methods

The MEDLINE corpus was licensed from the NLM. RI of all abstracts was performed using the open source Semantic Vectors package created by Dominic Widdows [13]. The author is also a developer on this project, and has made some modest contributions to increase the efficiency of the implementation for the purpose of this paper. Semantic Vectors extends the open source search engine Apache Lucene [14] which generates the term-document statistics used as a basis for the random indexing procedure. Once these were generated, RI took 31 minutes on a 64-bit Linux workstation with 4GB of RAM. The following parameters were used: seed-length, which determines the number of non-zero values in the index vectors, was set to 20. Only terms occurring 10 times or more in the corpus were considered. Terms were excluded if they contained any non-alphabet characters, a limitation of this study as this excludes many biologically relevant terms. A stop-list was used to eliminate frequently occurring terms without semantic content. This stop-list is packaged with the General Text Parser package, which is used in many LSA applications [15]. It is probable that this stop-list will need to be customized for certain biomedical applications, as some words on the list (such as “to”) also refer to specific gene names. However, it serves the purpose of the present investigation to maintain consistency with other LSA implementations. With these restrictions, 306,208 unique terms were obtained from the corpus. PFNETS were generated using (1 - the cosine similarity) between terms as a measure of their distance. The Pathfinder algorithm was applied with q = (the number of terms - 1) and r = infinity. The resulting network was visualized using the force-directed graph feature of the Prefuse visualization library [16]. The following PFNETS were created using a 2000-dimensional space and the log entropy weighting function as defined by Martin and Berry [4]: The nearest neighbors of the term “tuberculosis”, the disjunction of the terms “fibrillin” and “tgf”, and the term “thrombophilia”. Qualitative evaluation of the PFNETS was performed by the author and a domain expert in molecular biology. The choice of PFNETS was guided by the expertise available for their interpretation.

Results and discussion

Figure I illustrates the PFNET for the twenty nearest neighbors retrieved when querying MEDLINE space with the term “tuberculosis”(TB). Nineteen neighbors are visible, as the first term retrieved is always the query term itself. All of the terms listed are correctly associated with TB, ranging from clinical variants (miliary, extrapulmonary), related organisms (the causative organism of tuberculosis is mycobacterium tuberculae), diagnostic tests (sputum, smear, purified protein derivative (PPD) and tuberculin which are both used in skin tests) and treatments (isoniazid and ethambutol). In addition, the PFNET reveals an internal structure: (1) The links between terms related to mycobacteriae family have been preserved. (2) M. Bovis, the attenuated strain of Tuberculosis used for vaccination purposes is linked to “bcg” which stands for Bacille Calmette-Guérin, the name of the vaccine in which it is used. (3) Isoniazid and Ethambutol, two drugs used to treat TB, are linked together as are (4) the skin test components Tuberculin and PPD. These remaining connections illustrate the internal structure of the relationships between this group of concepts.

The nature of these relationships is not revealed by these methods. Unlike systems such as SemREP [17] which employ Natural Language Processing and domain knowledge to identify specific relations between defined concepts, this work seeks to explore the extent to which useful information can be derived from the distributional statistics of the MEDLINE corpus alone. RI provides a quantitative estimate of the similarity between terms, identifying relations of interest for further evaluation by domain experts or knowledge-based methods. Incorporating such measures has been shown to improve the accuracy of relationship extraction [18], suggesting a complementary role for these methods.

The PFNET in Figure II was obtained by finding the nearest neighbors of the disjunction of the terms “tgf” and “fibrillin”, using Widdows' and Peters' semantic space equivalent of logical disjunction [19]. This allows for the comparison between individual term vectors and groups of term vectors. This example was inspired by the recently discovered connection between these two molecules in the pathogenesis of Marfan syndrome [20]. Previously it was thought that Marfan syndrome was exclusively caused by dysregulation of the Fibrillin pathway. However, recently it has emerged that another molecule, Tranforming Growth Factor (TGF) is also involved in the pathogenesis of this disorder. Usually TGF activation is suppressed by Fibrillin, which has regions homologous with TGF-beta binding protein (TGF-bp). TGF-bp represses TGF activity. When this repression fails, TGF over-activates a group of proteins called SMADS which initiates a destructive pathway toward cell death. As Fibrillin is an important component in the formation of microfibrils, it is appropriate that Pathfinder retains the links between these concepts. The system has also retrieved “marfan”, the disorder under inquiry. “Autocrine” TGF-Beta is the specific transforming growth factor that has been linked to Marfan syndrome. Exploring the network directly related to tgf reveals “autocrine”, “tgf”, “beta”, the next step in the pathogenetic pathway “smad” and “transforming”. TGF is also directly linked to “fibronectin”, the expression of which it has been shown to stimulate. However perhaps the most interesting aspect of this network is the pathway from “tgf” to “fibrillin” which passes via “egf” through “microfibrils”. EGF stands for Epidermal Growth Factor, a protein which like TGF-bp is homologous with a substantial region of the fibrillin protein that is affected by mutations causing Marfan syndrome.

Figure III shows a PFNET for Thrompophilia, the propensity to develop thrombosis or blood clotting. Some of the associations in this PFNET are difficult to explain. Of the less obvious associations, “leiden” and “fvl” refer to Factor V Leiden, a common inherited thrombophilia. “Lmwh” is Low Molecular Weight Heparin, an anticoagulant that is used to treat thrombophilia. “vte” refers to venous thromobembolism. Canoas is a Brazilian city where much thrombophilia research occurs. However the most fascinating association here is surely “rppgf”, which stands for Arg-Pro-Pro-Gly-Phe. This is the sequence of an inhibitor of platelet aggregation that could be useful in thrombophilia treatment. Of note, a PUBMED search for “rppgf AND thrombophilia” (conducted on 03/14/08) does not return any results. Further examination of the MEDLINE corpus shows that these terms do not directly co-occur in any abstracts. However, despite this lack of direct co-occurrence, the cosine similarity between these two terms in MEDLINE space is 0.093. This finding is pertinent to the literature-based discovery (LBD) paradigm proposed by Swanson [21], in which two disparate literatures are bridged by a third, allowing for the identification of previously undiscovered relationships. The ability of LSA [22] and HAL [23] to support LBD has been shown using smaller corpora. LSA has been shown useful as a tool to identify bridging concepts [22]. and HAL combined with a range of statistical weighting functions has been used to replicate Swanson's seminal discovery of the connection between Raynaud's Syndrome and fish oil [23]. However, the question of which vector composition methods and statistical weighting functions might support literature-based discovery in a much larger corpus remains to be resolved.

Limitations and future work

This paper explores semantic neighborhoods of a small number of terms using PFNETS. However, a more extensive evaluation is required to determine the potential of these methods. Future work will assess the correlation between RI and empirically measured expert associations and evaluate the ability of these methods to replicate historical literature-based discoveries. In addition, an interface to allow researchers to rapidly explore associations in their domain of interest will be developed and evaluated in the context of ongoing biomedical research.

Conclusion

The integration of literature from disparate domains is a fundamental tenet of LBD. RI learns meaningful associations from the entire MEDLINE corpus of abstracts, including a drug-disease association undetected by PUBMED search. PFNETS provide an effective means of visualizing the local structure of the associative networks that RI learns. While the three networks presented have different emphases, they all contain concepts spanning basic and clinical science. The combination of these methods shows promise as a tool to integrate these research areas.

Figure I.

PFNET for Tuberculosis

Figure II.

PFNET for Fibrillin and TGF

Figure III.

PFNET for thrombopholia