Use of social network analysis tools to validate a resources infrastructure for interinstitutional translational research: a case study

Abstract

Question:

How can knowledge management and innovative technology, cornerstones of library practice, be leveraged to validate the progress of Clinical and Translational Science Awards?

Setting:

The Indiana Clinical and Translational Sciences Institute (Indiana CTSI) promotes interdisciplinary research across academic institutions.

Methods:

Using social networking tools and knowledge management skills enabled the department of knowledge informatics and translation to create a visualization of utilization of resources across different Indiana CTSI programs and coauthorship and citation patterns.

Results:

Contacts with different resources per investigator increased; every targeted program was shown to be linked to another. Analysis of publications established a baseline to further analyze the scientific contribution of Indiana CTSI projects.

Conclusion:

Knowledge management and social networking utilities validated the efficacy of the Indiana CTSI resources infrastructure and demonstrated visualization of collaboration. The bibliometric analysis of publications provides a basis for assessing longer-term contributions of support to scientific discovery and transdisciplinary science.

INTRODUCTION

The Clinical and Translational Science Awards (CTSAs) are bringing new challenges and opportunities for the fields of knowledge management and biomedical informatics. They are also enabling library faculty to serve in unique roles to foster the goals of these innovative research opportunities. The Indiana Clinical and Translational Sciences Institute (Indiana CTSI) recognized, during its proposal development, the importance of library faculty to achieving its goals. The Indiana University (IU) Ruth Lilly Medical Library is an integral part of the IU School of Medicine's Department of Knowledge, Informatics and Translation (KIT). Three KIT faculty members are actively involved in the Indiana CTSI. The director of the library serves as both the director of the tracking and evaluation program and codirector of the biomedical informatics program. Another library faculty member is also funded on both programs. A third departmental faculty member serves full-time as the associate director of the tracking and evaluation program.

Traditionally, both library and biomedical informatics research have focused on tools supporting individual components of basic science and clinical care. CTSAs are built on the premise that the next generation of health care research must cross disciplines to facilitate rapid and appropriately contextual improvements in the quality of care [1]. Libraries are in a unique position to play an integral role in both supporting translational research and utilizing technology tools that validate its efficacy.

In May 2008, the IU School of Medicine received a CTSA from the National Institutes of Health (NIH). What differentiated the Indiana CTSI from other CTSAs was that the grant sought to leverage resources from across the state of Indiana, beginning primarily with the state's three premier research institutions, IU including IU Bloomington, IU Purdue University–Indianapolis (IUPUI), and IU School of Medicine (IUSM); Purdue University; and the University of Notre Dame.

Translating bench research into clinical practice is difficult for large homogeneous academic medical centers. Attempting to foster rapid development of collaborative research across heterogeneous organizations offers huge challenges, but also great opportunities. The mission of the Indiana CTSI is to increase translational biomedical research and improve the health of the people of Indiana and beyond. Although the mission is simple, the steps necessary to realize this vision are complex.

The Indiana CTSI has twelve programs designed to support and foster translational research within and among the major research institutions; these correspond to the ten key functions required for a CTSA application (Table 1). In addition, the three institutions have sixty-three service cores, forty-five of which are designated as Indiana CTSI service cores supporting translational research. However, programs and service cores to support researchers are only as effective as the researcher's understanding of how to leverage these resources. The CTSA concept is to have a “front door” to a research home for all translational researchers, but crossing diverse institutional and programmatic barriers is extremely difficult [2]. To mitigate this perceived obstacle, project development teams (PDTs) were organized to assist the individual researcher to navigate through the myriad of resources provided. Seven such teams, crossing the three Indiana CTSI institutions, provide access to multidisciplinary research expertise to help guide the researchers in proposal development, study design, and identification of support components that can help bring projects to fruition.

Table 1.

Key functions of an institutional Clinical and Translational Science Award (CTSA) as implemented in the Indiana Clinical and Translational Sciences Institute (Indiana CTSI)

Table 2.

Number of peer-reviewed articles by investigators who mentioned benefitting from Indiana CTSI project development team services

The overarching goal of the Indiana CTSI is to transform the culture of the participating institutions into one that facilitates the conduct of clinical and translational science research, understanding that traditional academic competition, both within a single institution and among research institutions, can be counterproductive to effective research development and dissemination. To test the hypothesis that the infrastructure put in place would lead to greater interdisciplinary collaboration, KIT faculty involved with Indiana CTSI evaluation considered several social network analyses.

Proof of such team-based research would be found in interdisciplinary collaboration in both grant applications and scientific publications, tracked over time. Bibliometric analysis readily falls into the purview of library research, and analyzing publications promised to demonstrate interesting collaborations. Although the tracking and evaluation program is collecting data on both, because the Indiana CTSI has been in existence for only three years and due to the time it takes to garner major grants and realize citations to relative publications, at the time of this analysis, there was little likelihood that any positive change would accrue. However, without an interdisciplinary research infrastructure, there would also be little likelihood that the Indiana CTSI would be able to achieve its mission. Therefore, an initial step was to validate the construct devised to foster collaboration, if not between institutions, at least between disparate content areas.

One way to demonstrate collaboration is through the use of social network analysis (SNA) tools. These tools can provide insight into the connections between and among investigators and departments through visualization of the connections and analysis of the strength of the connections among individuals and departments [3, 4]. The increasing interest in social network theory and social network analysis has been documented by Schultz-Jones in a content analysis study of articles [5].

Seven PDTs were established to support a variety of disciplines:

• The Preclinical PDT focuses on translation studies in animal and cellular models.

• The Pediatric PDT facilitates research in children with emphasis on bench-to-beside studies.

• The Adult PDT focuses on early translational studies.

• The Behavioral/Population Science PDT focuses on epidemiological and behavioral research.

• The Purdue PDT emphasizes bioengineering, nutritional, and veterinary medicine.

• The Notre Dame PDT provides broad access to biomedical research expertise.

• The Imaging PDT provides expertise in anatomical, functional, and molecular imaging.

The PDTs are designed to serve as a “one-stop shop” for study development by providing investigators access to multidisciplinary research expertise, biostatistics, internal review board (IRB) or regulatory services, nursing support, and pilot funds. While each of the PDTs provides a “front door” to research support, most translational grants would require access to more than one Indiana CTSI program and its targeted expertise. Demonstrating that investigators use multiple programs should be a proxy for the viability of the infrastructure supporting interdisciplinary research.

In addition to the individual PDTs, several of the other major programs support the translational research process, including the Indiana CTSI Bioethics and Subject Advocacy Program (BSAP), Biomedical Informatics Program (BIP), Design and Biostatistics Program (DBP), and Regulatory Knowledge and Support Program (RKSP). The team hypothesized that these programs would not only provide direct support to translational researchers, but would also link to support being given by the PDTs and the other programs. Tracking these interactions using SNA tools would give a clear picture of the nascent success of the Indiana CTSI infrastructure.

METHODS

Each of the Indiana CTSI programs and the individual PDTs (within the novel translational methodologies and pilot studies program) is required to report individual investigators served by the program. Each program records investigators served during the quarter, including investigator institution and area of expertise based on a list provided by the NIH. These data are reported quarterly, using Microsoft Excel spreadsheets, to the tracking and evaluation team, which compiles the reported data. A knowledge management model was constructed to ensure that collected data would provide appropriate validation of the goals of the project. For the period of this study, December 2009 through November 2010 (grant year 3), 786 investigators were identified as benefitting from Indiana CTSI resources. This number formed the population. The population comprised those who used the services of at least one PDT or other program during the reporting year. A total of 786 investigators used the designated PDTs or programs, resulting in 1,156 reported contacts across all of the programs.

To determine the relationship among the programs and specific PDTs, data from each program and PDT were compiled into a spreadsheet and calculated to discover how many investigators used more than one program and what programs appeared to be providing the greatest amount of support. In addition, to discover the relationships between and among the programs and the PDTs and to provide visualization for the findings, the data were ported into the SNA tool, NodeXL [6]. The use of NodeXL, which is based on Excel, simplified the data entry into the tool for analysis, minimizing the occurrence of data errors in transferring data from the source documents to the analysis tool. Therefore, the primary analysis was done using NodeXL.

Bibliometric citation analysis data were obtained for the articles published resulting from investigators using the Indiana CTSI PDTs. Both ISI's Web of Knowledge and Google Scholar were searched in April 2011 to find the number of citations from peer-reviewed journal articles. Book chapters, dissertations, and news write-ups were not included as citations.

RESULTS

The Indiana CTSI does not have baseline data for investigator use of programs prior to the implementation of the Indiana CTSI. In addition, some programs supported by the Indiana CTSI are new to investigators across the three institutions. This study compared available data compiled in the second year of the CTSA grant.

The 786 investigators reported by Indiana CTSI programs accounted for a total of 1,156 contacts. Of those, 288 were nonclinical investigators and 498 were clinical investigators. The mean number of contacts per investigator was 1.47 (range 1–7 programs), compared to a mean of 1.37 for the previous year. Figure 1 displays the pattern of program and investigator interaction for all Indiana CTSI program reporting investigators benefiting from Indiana CTSI resources in grant year 3.

Figure 1.  1 Pattern of program and investigator interaction for all Indiana Clinical and Translational Sciences Institute (Indiana CTSI) reporting investigators benefiting from Indiana CTSI resources in grant year 3.

The DBP reported contact with 401 investigators (34.7% of all investigator contacts), PDTs (combined) reported 95 (8.2%), BIP reported 93 (7.97%), and RKSP reported 63 (5.5%). All other Indiana CTSI programs reported contact with 134 (11.6%).

For the PDTs, the Pediatric PDT reported 25 investigators (26.3% of all investigator contacts with PDTs); the Adult PDT reported 20 (21.0%); the Preclinical PDT reported 14 (14.7%); the Purdue PDT reported 12 (12.6%); the Behavioral PDT reported 10 (10.5%); the Notre Dame PDT reported 7 (7.4%); and the Imaging PDT reported 7 (7.4%).

Analysis of inter-program support for investigators showed wide variation in investigator use of more than one program. For the PDTs, the analysis used data from the seven PDTs in place for the entire reporting period (Adult, Behavioral, Imaging, Notre Dame, Pediatric, Preclinical, and Purdue PDTs). The data showed that, of the ninety-five reported investigator contacts, two were investigators accessing more than one PDT. No investigator accessed more than two PDTs, which was interpreted as indicating that the structure of the PDTs was sufficient to meet the needs of investigators through access to a single PDT, obviating the need to access multiple PDTs. The PDT usage is seen in Figure 2.

Figure 2.  2 Project development team (PDT) usage.

Analysis of the “one-stop shop” concept for the PDTs shows significant cross-program interactions for investigators using the PDTs. Figure 3 displays the pattern of investigator use of PDTs and other programs. For some programs such as bioethics and subject advocacy, the interface with the investigator through the PDT process represents the most commonly reported venue for providing consultation regarding ethical issues to investigators.

Figure 3.  3 Pattern of investigator use of PDTs and other programs.

Further analysis of cross-program access, including the PDTs, indicated that 217 investigators were served by more than 1 program. Of those, 120 investigators were served by 2 programs, 59 were served by 3 programs, 26 were served by 4 programs, 7 were served by 5 programs, 4 were served by 6 programs, and 1 was served by 7 programs. Excluding the PDTs, 33 investigators used both the DBP and the BIP; 25 investigators used the DBP and the RKSP; and 6 investigators used the BIP and the RKSP. The majority of the multiple contacts were between the PDTs and the 3 primary support programs and the pilot grants program.

In looking at centrality of contacts, the DBP had the greatest number of individual contacts with investigators and the greatest number of dual contacts with PDTs, with ninety-two investigators using each. Visualization of these contacts is shown in Figures 3 and 4.

Figure 4.  4 Biomedical informatics program and PDT contact data.

While accessed by more investigators overall than the RKSP, the BIP had fewer dual contacts with PDTs. These contacts were five and thirteen, respectively. The BIP and PDT contact data are seen in Figure 4. The majority of the dual contacts for the BIP were with the DBP, with thirty-three reported.

Since the Indiana CTSI operation only began in 2008, few longitudinal data were available at the time of this study. The study included a bibliometric analysis of those peer-reviewed journal publications that directly acknowledged the Indiana CTSI. Twenty authors had published articles in thirty peer-reviewed journal publications that directly acknowledged the Indiana CTSI. Among those twenty researchers, the most-used PDTs were the Behavioral/Population Science PDT and Pediatric PDT (six each), followed by five researchers using the Adult PDT. Two researchers used the Preclinical PDT, and one researcher used the Imaging PDT. Out of the thirty articles, eleven were from researchers using the Adult PDT, nine from researchers using the Pediatric PDT, six from researchers using the Behavioral/Population Science PDT, three from researchers using the preclinical PDT, and one from a researcher using the Imaging PDT.

Two of the articles had multiple PDT authors: One paper had three Indiana CTSI authors, with two authors using the Pediatric PDT and one author using the Behavioral/Population Science PDT, and the other paper had two Indiana CTSI authors who both used the Pediatric PDT.

The total number of citations for these 30 articles as of April 2011 was 225, with 198 (88.0%) of those citations being unique citations (not including any self-citing). The Web of Knowledge was searched first, supplemented by results from Google Scholar. The Google Scholar searches added 44 unique citations, making up 22.2% of the total unique citation count. ISI's impact factor, found in Journal Citation Reports (JCR), is a widely used metric. The impact factor calculates the impact a particular journal has in its field of discipline, although it is hard to compare impact factors across disciplines. For the 10 articles with the highest number of citations, impact factor ranged from 1.000 (Journal of Empirical Research on Human Research Ethics) to 14.816 (Circulation). Two of the articles were published in journals, Conflicts in Health and Journal of Autism and Developmental Disorders, that were not listed in the JCR. The article with the most number of total citations, 28, was from the journal Contemporary Clinical Trials, which had an impact factor of only 1.506.

DISCUSSION

The results of the analysis of the investigators using PDTs indicated that the Indiana CTSI's goal in designing PDTs to be a single point of entry and support was achieved. As noted above, no investigator accessed more than two PDTs, which was interpreted as an indication that the structure of the PDTs was sufficient to meet the needs of investigators through access to a single PDT, obviating the need to access multiple PDTs. The PDT process provides access to a rich source of program resources as shown in Figure 3. The organization of PDTs based on distinct phases of research, populations, or location may have contributed to the limited number of investigators who accessed more than one PDT. While the DBP had the greatest number of individual investigator contacts, most of these were not generated through an initial PDT contact. Biostatistics, a department in the IU School of Medicine, has provided statistical consultation at no charge for grant application preparation. Most of the Indiana CTSI investigators are aware of this, and the numbers of program contact were probably based on past contact.

However, the components of RKSP were scattered across the academic medical center until the formation of the Indiana CTSI. The program provides direct support for navigating federal, state, and local regulations and provides assistance in streamlining protocol submission and review, including IRB requirements. The numbers of cross-contacts suggest that the PDTs recognize the value of this resource. While this resource is located in the IU School of Medicine, it is interesting to note that one Purdue investigator used this program.

There were few recorded dual contacts between the PDTs and the BIP; however, the BIP provides a substantial infrastructure support for virtually all aspects of the Indiana CTSI. Of greater interest is the relationship between the BIP and the DBP, because both programs involve database creation and management, and coordination between the two on the surface would appear to be essential to data integration. However, there were only thirty-three instances of investigators using both programs.

The visualization of the program-investigator interaction reveals patterns of utilization for further analysis. One such analysis is the evaluation of the contribution of Indiana CTSI service utilization to the creation of new scientific literature. The Indiana CTSI has begun to compile and analyze data on publications benefiting from Indiana CTSI resources. The analysis will attempt to assess the impact of utilization of one and multiple Indiana CTSI resources to determine if more concentrated use of those services contributes to generating scientific manuscripts that are cited more frequently over a longer period of time as one proxy measure for contributing to advancing clinical and translational science.

While ISI's Web of Knowledge is often used for determining citation counts of journals, searching Web of Knowledge and Google Scholar in tandem returned more citations than each individually. The Google Scholar search added 22.2% of the total unique citation count, indicating that using these tools together provides the most accurate citation count.

CONCLUSION

The use of SNA tools to assist in tracking the interaction of Indiana CTSI programs and investigators has provided the basis for developing a clear picture of the success of the Indiana CTSI infrastructure after the first three years of full operation. The results of this analysis also provided for iterative reengineering of the infrastructure and enabled the Indiana CTSI leadership to more appropriately allocate funds based on programmatic needs. The status of current Indiana CTSI efforts to transform the culture of the participating institutions into one that facilitates the conduct of clinical and translational science research is shown in the interaction between the PDTs and the three other Indiana CTSI programs studied. The analysis and graphic presentation of that analysis provide powerful tools for documenting and assessing changes in collaborative and translational research and for evaluating the impact of resource allocation to different Indiana CTSI programs to support investigators. The use of bibliometric analysis demonstrates the importance of library skills and knowledge management contributions to the new field of translational sciences, both through the introduction of research techniques that traditionally fall into the knowledge management domain, such as bibliometric analysis, and through the use of technology that builds on categorization skills, such as SNA tools.

The technology tools used to provide a visualization of the Indiana CTSI collaborations and the knowledge management models used to collect, organize, and create new knowledge that can be disseminated across the translation spectrum underscore the expanding and critical roles that knowledge faculty can play in emerging research agendas.

Perhaps of equal significance is the use of knowledge management and evaluation faculty in creating a model that is central to validating the efficacy of the CTSA in Indiana, and potentially across the nation.