Breaking Down Silos: Mapping Growth of Cross‐Disciplinary Collaboration in a Translational Science Initiative

Abstract

The importance of transdisciplinary collaboration is growing, though not much is known about how to measure collaboration patterns. The purpose of this paper is to present multiple ways of mapping and evaluating the growth of cross‐disciplinary partnerships over time. Social network analysis was used to examine the impact of a Clinical and Translational Science Award (CTSA) on collaboration patterns. Grant submissions from 2007 through 2010 and publications from 2007 through 2011 of Institute of Clinical and Translational Sciences (ICTS) members were examined. A Cohort Model examining the first‐year ICTS members demonstrated an overall increase in collaborations on grants and publications, as well as an increase in cross‐discipline collaboration as compared to within‐discipline. A Growth Model that included additional members over time demonstrated the same pattern for grant submissions, but a decrease in cross‐discipline collaboration as compared to within‐discipline collaboration for publications. ICTS members generally became more cross‐disciplinary in their collaborations during the CTSA. The exception of publications for the Growth Model may be due to the time lag between funding and publication, as well as pressure for younger scientists to publish in their own fields. Network analysis serves as a valuable tool for evaluating changes in scientific collaboration.

Introduction

Across all branches of science, teams increasingly dominate solo scientists in the production of knowledge.1 One potential explanation for the growth in team science is that modern scientific challenges often require cross‐disciplinary theoretical and methodological approaches.2 For example, obesity, smoking, and Alzheimer's disease are complex bio‐social‐environmental problems that require “cells to society” thinking that crosses traditional disciplinary boundaries.3, 4 Researchers engaged in team science may have a higher likelihood of integrating perspectives that lead to deeper analyses and solutions.2, 4 Our own observation as we have participated in the CTSA consortium is that institutions are starting to devote more resources to build infrastructure for cross‐disciplinary teams. If that continues to be the case, then there will also be an increased need to evaluate team science process and outcomes.

The Institute of Clinical and Translational Sciences (ICTS) at Washington University is an example of a large‐scale scientific enterprise that is designed partly to support cross‐disciplinary scientific collaboration. Funded by the National Institutes of Health's Clinical and Translational Science Awards (CTSA) program in 2007, the ICTS has a major goal of promoting and facilitating collaborative research. It has established interinstitutional partnerships with six large academic institutions in the St. Louis area. ICTS supports cross‐disciplinary collaboration through its 24 core units (e.g., Center for Community Engaged Research, Dissemination & Implementation Research Core, Genome Technology Access Center) that engage researchers from diverse backgrounds who work together on clinical research and translate findings for societal benefit. ICTS also supports a number of internal funding programs that enhance clinical and translational science and education, including a Just‐in‐Time Core Usage Program and a Community/University Health Research Partnerships program. See http://icts.wustl.edu for more information.

With the rise of such team science initiatives, an emerging field of study has been the evaluation of these ventures—known as the science of team science. This field recognizes that the factors associated with successful scientific collaboration are multilevel in nature, and thus has utilized a variety of micro, meso, and macrolevel analytic strategies.2 This science of team science framework was used to inform the evaluation design for ICTS. In particular, the ICTS evaluation focused on tracking collaboration through three phases of scientific activity: (1) study planning and grant development; (2) study implementation; and (3) results dissemination and publication.5 The assumption underlying this model is that scientific collaborations must occur at each of these stages, but that the collaboration characteristics may differ (e.g., persons collaborating at each stage). In this paper, the focus is on the development and evolution of collaborations in grants and publications (phases 1 and 3) over 5 years.

As its name suggests, the science of team science has as a primary focus the study of the mechanisms of scientific collaboration. Social network analysis is uniquely suited to study such collaborative relationships,6 and is a more appropriate methodology to use for mapping collaborations than traditional approaches such as surveys.7 One network evaluation study was conducted with the University of Pennsylvania's Institute of Translational Medicine and Therapeutics, another CTSA recipient. They found that collaborative papers and grants doubled within 5 years among investigators within but not outside the institute.8 Another study found that the creation of a new interdisciplinary life sciences institute increased inter‐departmental coauthorship with no change in publication numbers.9, 10 Other studies have focused on cross‐institution collaboration patterns among federally funded centers in tobacco control, oncology, and Alzheimer's disease.11, 12, 13 Finally, in subject‐specific studies such as open source innovation, researchers show that interdisciplinary patterns may be temporary and academic output re‐aligns with disciplinary borders over time.14

These early studies show the utility of a network approach to assess team initiatives in a variety of settings. However, little is known about how cross‐disciplinary collaborations change over time, whether they differ across different phases of scientific activity, and whether funding designed to promote greater cross‐disciplinary collaborations actually changes scientific behavior. This paper uses network analysis in two ways to start answering these questions. First, network maps that can be used to assess the growth of scientific collaboration since the inception of the ICTS are produced. Second, 5 years of grant and publication network data are analyzed to describe the extent to which cross‐disciplinary collaborations have arisen since ICTS funding began.

Methods

Participants

Researchers become members of ICTS to gain access to core facilities (i.e., biomedical imaging), consulting services including research design and biostatistics, enhanced career development and clinical research training, funding opportunities, and collaboration opportunities. Scientists formally apply to become members by completing an online self‐registration form. ICTS funding began in September, 2007, and the first members registered that December. Collaborations occurring in 2007 can thus be considered “pre” ICTS. ICTS had 482 members by the end of 2008 and 1,272 members by the end of 2011. To facilitate tracking and evaluation, all ICTS members were assigned a year cohort according to registration date. In order to fairly credit any change in rate of grant submissions or publications per year to ICTS membership, a June 30 cutoff was used, as there is likely lag time between membership initiation and a change in collaboration patterns (e.g., an investigator registering on July 1, 2009 would be classified as part of the 2010 cohort). The number of submitted grants and publications for years 2007–2011 is shown in Table 1.

Table 1.

Number of grant submissions and publications by ICTS members

	Grant Submissions		Publications
	2007	2010	2007	2011
Cohort Modela	440	557	1101	1218
Growth Modelb	440	986	1101	2679

^aIncludes 2008 members only.

^bIncludes 2008 members only for 2007 grants and publications, all current members for 2010 grant submissions and 2011 publications.

IRB approval was not required for secondary analysis of grant submission and publicly available publication information.

Measures

Discipline. ICTS members selected a primary disciplinary specialty from a formal list based on the National Institute of Health Field of Training list when registering for membership.15 For ease of analysis and interpretation, disciplines were collapsed from 205 detailed categories into 18 major categories shown in Table 2. About half of ICTS members fall within the Clinical Disciplines category, which is to be expected given the clinical research focus of the CTSA.

Table 2.

Number of ICTS members submitting grants and publishing by year

	Grants		Publications
Discipline	2007	2010	2007	2011
Clinical disciplines	99	258	120	447
Pediatric disciplines	10	27	12	44
Genetics	6	21	8	40
Neuroscience	8	22	8	39
Allied health	10	19	10	36
Public health	9	22	14	34
Immunology	5	18	4	27
Cell and developmental biology	6	18	5	23
Specialty other	4	13	7	21
Psychology, nonclinical	4	12	4	20
Statistics/research methods/informatics	8	14	9	18
Bioengineering	2	11	4	17
Microbiology and infectious diseases	6	10	5	17
Physiology	3	7	3	12
Biochemistry	3	6	3	11
Chemistry	2	9	5	11
Social sciences	1	5	3	9
Nursing	0	1	0	7
Total	186	493	224	833

Grant Development Data. The university maintains databases of all submitted grants, contracts, programs, and subagreements. Records were retained for analysis for all key personnel who were ICTS members by the end of 2010. All new submissions from 2007 and 2010 are reported here, including federal, state, local, and foundation grants, contracts, programs, and subagreements, excluding renewals, resubmissions, etc. Submissions from 2011 were not available due to the biennial structure of the grants data collection. The number of grants submitted by ICTS members in 2007 and 2010 is shown in Table 1.

Publication Data. Bibliometric data on publications by scientists who were ICTS members by the end of 2011 were obtained from Elsevier Scopus as part of the ICTS evaluation. Publications (including articles, conference papers, reviews, and short surveys) from 2007 and 2011 are reported here. The number of publications by ICTS members in 2007 and 2011 is shown in Table 1.

Statistical analysis

The basic unit of analysis in this project was a collaboration tie. For grant collaboration ties, ICTS members were linked if they were listed as key personnel on the same contract, grant, program, or subagreement submission in a given year. For publications, ICTS members were linked if they were coauthors on a published article in a given year. In‐press articles, books, editorials, erratum, letters, and notes were not included.

In addition to basic network descriptive statistics such as network size, density, and average degree,16 three specific network statistics were used to assess cross‐disciplinarity of the ICTS networks: cross‐disciplinary density ratio, E‐I index, and modularity. First, a cross‐disciplinary density ratio was calculated by dividing the density of cross‐disciplinary ties (e.g., a grant collaboration between a neuroscientist and a geneticist) by the density of within discipline ties (e.g., a collaboration between two clinical scientists). A cross‐disciplinary ratio greater than 1.00 thus indicates a network that has a higher cross‐disciplinary density than within‐discipline density. The second metric was the E‐I index,17 which subtracts the number of within‐group ties in a network from the number of between‐group ties in a network, and then divides by the total number of ties. E‐I ranges from –1 to +1, with –1 indicating that all ties are within groups, and +1 indicating that all ties are between groups. The third metric was network modularity, a statistic that measures the strength of community structures in networks by taking the percentage of links in an existing network that connect nodes of the same community and subtracting the percentage that could be expected if links were randomly distributed.18 Modularity ranges from –1/2 to 1, with numbers closer to 1 indicating strong community structure where most ties are observed within the same community. In the context of this research, disciplines serve as communities. Given the interest in the facilitation of cross‐disciplinary collaborations, cross‐disciplinary ratio is expected to increase over time, E‐I index to increase over time, and modularity to decrease over time; that is, the percentage of collaborations between ICTS members of the same discipline should go down relative to cross‐discipline collaborations. These three measures are used because they are constructed in different ways and therefore reflect different nuances of the data. For instance, modularity is chance‐corrected while the cross‐disciplinary density ratio is not. If the different measures provide roughly the same results, there is greater confidence in the true nature of the development of these networks.

In order to demonstrate that the changes in collaboration patterns over time as described by density, cross‐disciplinary ratio, E‐I index, and modularity are larger than would be expected by chance, Monte Carlo simulations were run with the observed 2007 grants and publications networks. If the values in the observed 2010 grants and 2011 publications networks for these measures are outside of the 95% confidence intervals of the simulated networks, the changes over time are not likely to be random.

Descriptive network statistics and modularity were calculated using Pajek version 3.14, cross‐disciplinary ratio and Monte Carlo simulations were calculated with Statnet version 3.1–0, E‐I was calculated with UCINet version 6.507, descriptive statistics were calculated using SPSS version 21.1, and network visualizations were created using Gephi version 0.8.2. Data analyses were conducted in 2013 and 2014.

Results

The primary purpose of this study is to explore how cross‐disciplinary collaboration has grown during ICTS funding. Given the nature of ICTS membership which grows over time, there are two valid ways to examine this pattern of growth. One is to examine only the first cohort of 262 ICTS members and examine how grant and publication collaborations change over time. This Cohort Model allows us to assess how ICTS membership influences collaboration for a group of scientists, all of whom have been a member of ICTS for at least 4 years. The other approach is to examine the entire ICTS membership as it grows year after year. This Growth Model allows us to assess how collaboration is changing in the entire ICTS membership, even if some of the scientists have been an ICTS member for shorter periods of time.

Grant submissions

Descriptive network statistics for grant submissions are displayed in Table 3. When considering the Cohort Model, the number of ICTS members who were key personnel on one or more grants increased from 186 to 193 from 2007 to 2010, and the average number of members they collaborated with increased from 1.65 to 4.41. The growth of collaboration among ICTS members is evident in Figure 1 . The cross‐disciplinary density ratio increased from 1.21 to 1.36, the E‐I index increased from 0.176 to 0.319, and modularity decreased from 0.140 to 0.054, indicating that a greater proportion of collaborations were cross‐disciplinary in 2010 than in 2007.

Table 3.

Descriptive network statistics: networks demonstrate growth and increasing cross‐disciplinary collaboration over time

Year	ICTS Members	No. of Disciplines	Density	Mean Degree	Cross‐Disciplinary Ratio	E‐I Index	Modularity	∆ Modularity
Grant Submissions
Cohort Model
2007	186	17	0.009	1.65	1.21	0.176	0.140
2010	193	17	0.023	4.41	1.36	0.319	0.054	–61%
Growth Model
2007	186	17	0.009	1.65	1.21	0.176	0.140
2010	493	18	0.011	5.51	1.48	0.291	0.071	–49%
Publications
Cohort Model
2007	224	17	0.007	1.61	1.18	0.156	0.093
2011	234	17	0.009	2.14	1.25	0.208	0.071	–23%
Growth Model
2007	224	17	0.007	1.61	1.18	0.156	0.093
2011	833	18	0.004	3.57	0.76	–0.067	0.125	35%

Figure 1.

Grant submission networks, 2008 cohort.

When considering the Growth Model, the number of key personnel increased from 186 to 493, and the average number of other members they collaborated with increased from 1.65 to 5.51. The cross‐disciplinary density ratio increased from 1.21 to 1.48, the E‐I index increased from 0.176 to 0.291, and modularity decreased from 0.140 to 0.071. The growth in cross‐disciplinary collaboration was more evident with the Cohort Model, which demonstrated a 61% decrease in modularity, while the Growth Model demonstrated a 49% decrease in modularity. This may be partially due to the fact that the Growth Model includes ICTS members as they join over time, and newer members tended to be younger scientists, which given funding pressures may lead to lower rates of increases in cross‐discipline collaborations.

Publications

Descriptive network statistics for publication coauthorships are also displayed in Table 3. When considering the Cohort Model, the number of ICTS members who were authors on one or more published papers increased from 224 to 234 from 2007 to 2011, and the average number of other members they coauthored with increased from 1.61 to 2.14. The increase in the size of the network and rate of collaboration is evident in Figure 2 . The cross‐disciplinary density ratio increased from 1.18 to 1.25, the E‐I index increased from 0.156 to 0.208, and modularity decreased from 0.093 to 0.071, indicating that a greater proportion of collaborations were cross‐disciplinary in 2011 than in 2007.

Figure 2.

Publication coauthorship networks, 2008 cohort.

When considering the Growth Model, the number of authors increased from 224 to 833, and the average number of other members they coauthored with increased from 1.61 to 3.57. Unlike the grant network however, the cross‐disciplinary density ratio decreased from 1.18 to 0.76, the E‐I index decreased from 0.156 to –0.067, and modularity increased from 0.093 to 0.125. Similar to the results for grant collaborations, the growth in cross‐disciplinary collaboration was more evident with the Cohort Model, which demonstrated a 23% decrease in modularity, while the Growth Model demonstrated a 35% increase in modularity.

Closer examination of interdisciplinary growth

Although this is primarily a descriptive study, we used Monte Carlo network simulation19 to examine more closely whether the changes observed in the grants and publications networks were large enough to suggest that collaboration patterns were really changing over time, and not due to simple random changes. Traditional inferential tests are not possible to perform on networks for a number of reasons, but one important reason is that network ties violate the assumption of independence. That being said, stochastic models of network ties are possible to create,20 and simulations on those models are useful for exploring hypotheses about network structures and dynamics.19

We built a simple exponential random‐graph model (ERGM) for each of the 2007 grants and publications networks, conditioned only on the number of ties and the pattern of interdisciplinary collaboration ties. Then, we used Monte Carlo simulation to create 500 random networks based on the simple ERGM models. Table 4 presents the means and 95% empirical confidence intervals of four network statistics (i.e., density, cross‐disciplinary ratio, E‐I index, and modularity) across the 500 random networks. If only random changes are occurring over time, then we would expect that the observed network statistics in 2010 (for grants) and 2011 (for publications) would fall well within the empirical confidence‐interval bounds. For both grants and publications, the network densities (which correspond to number of network ties) fall outside the CI bounds, suggesting that more collaboration is occurring in later years within the same cohort network. For the grants network, the 2010 modularity score is smaller than we would expect due to chance, and the E‐I index is on the edge of the 95% CI. This also suggest that at least for grants, in 2010 the collaborations have become more interdisciplinary.

Table 4.

Monte Carlo simulation results testing the hypothesis that the frequency and characteristics of collaboration ties remain stable from 2007 to 2010 and 2011 (n = 500 simulations)

	Grants Cohort Network				Publications Cohort Network
Network Statistic	Obs. 2007	MC Simulation Mean	MC Simulation 95% CI	Obs. 2010a	Obs. 2007	MC Simulation Mean	MC Simulation 95% CI	Obs. 2011b
Density	0.009	0.009	.007, .010	0.023	0.007	.007	.006, .008	0.009
Cross‐disciplinary ratio	1.21	1.22	0.87, 1.66	1.36	1.18	1.20	.874, 1.56	1.25
E‐I index	0.176	0.175	.014, .327	0.319	0.156	0.158	.008, .291	0.208
Modularity	0.140	0.138	.082, .196	0.054	0.093	0.102	.052, .155	0.071

^aDensity and modularity of the observed 2010 grant network fall outside of the 95% CI generated by the simulated 2007 network.

^bDensity of the observed 2011 publication network falls outside of the 95% CI generated by the simulated 2007 network.

To better understand how these collaboration changes might be occurring, we can also visually examine the collaboration networks in more detail. The evolution of research teams that began as relatively isolated in 2007 is demonstrated in Figure 3 . The main network depicted in the figure is the 2007 grant submissions. Two highly cohesive subgroups that represent a small number of disciplines are highlighted. The first is primarily a clinical group (blue nodes), and the second is primarily a statistics group (tan nodes). The growth of their collaborations is shown in the inserts, which display the original investigators plus those they were directly linked to in the 2010 grant network shown in Figure 1 (their network neighborhood). The group of clinical investigators expanded to include several neuroscientists, while the allied health investigator later collaborated with a pediatrician. The group of statisticians and geneticists expanded to include investigators from a number of disciplines, including clinical, chemistry, and cell and developmental biology. More specifically, the 2007 statistician subgroup included three different disciplines, but their collaboration neighborhood in 2010 included 13 different disciplines, strongly suggesting that the grant submission collaborations were more interdisciplinary after 3 years of CTSA funding.

Figure 3.

Evolution in grant cosubmissions from 2007 to 2010. Main graph represents 2007 grant network with two subgroups highlighted. Evolution in subgroup composition demonstrated in 2010 + Neighborhood insert with original 2007 nodes represented as squares.

Discussion

This paper demonstrated growth over time in scientific collaboration among members of the Washington University ICTS. Over a 3–4 year period, ICTS members have become involved in a greater number of scientific planning collaborations (as measured by new grant submissions) and scientific dissemination collaborations (as measured by journal article coauthorships). In addition to this general increase in the amount of collaboration, collaborations were demonstrated to have become more cross‐disciplinary over time. In particular, between 2007 and 2010 grant collaboration modularity (as used here a measure of disciplinary isolation) decreased between 49% (for the Growth Model) and 61% (for the Cohort Model). The simulation analyses suggest that this drop in modularity may reflect a real change in the pattern of interdisciplinary collaboration on grant submissions.

Evidence for growth of cross‐disciplinary coauthorship collaborations was less strong. This may be due to the inherent time lag between planning research, getting research funding, conducting the studies, and then finally publishing the results. The differences in these two models (Growth vs. Cohort) also suggest another explanation. The Cohort Model followed the first members of ICTS, who tended to be more senior scientists, whereas the Growth Model looked at the entire ICTS membership base. It is possible that as the ICTS grew, younger scientists were being added to the membership, and these young scientists may have different coauthorship patterns. In particular, because of pressures to establish research funding and to quickly publish, it may take longer for junior scientists to establish interdisciplinary partnerships. This is supported by ICTS evaluation data collected in 2011,21 where ICTS members reported strong support for the value of interdisciplinary science, but the most common major barrier that they listed for that type of work was the increased amount of time it takes to do such work.

Although the network methods used here suggest that cross‐disciplinary science has increased during CTSA funding, there are a number of caveats when interpreting this pattern. First, this study observes the growth of ICTS over time, but it does not have a comparison group of non‐ICTS members. Although that approach would theoretically allow us to make stronger causal claims, in reality there is no valid comparison group. ICTS members at Washington University comprise the vast majority of scientists involved in clinical and translational research (and over half of the entire medical school faculty), so there are no clinical scientists “left over” who could be legitimately compared to ICTS members. Second, these changes are observed during a specific time period within one academic institution, and obviously caution must be applied when generalizing these results. Third, when interpreting the results it is important to keep in mind how scientific discipline was operationalized; namely by using the NIH Field of Training list. This list is used by all CTSA funded institutions, so this suggests good generalizability. However, ICTS members are asked to list their discipline once when they first become an ICTS member. It is not clear how this selection maps onto the true disciplinary role that an individual scientist plays in a collaborative team. It is possible that disciplinary identity changes over time, or that a scientist has multiple disciplinary identities or roles. Future interdisciplinary science work will thus benefit from research that establishes better definitions and ways to measure disciplinarity. Finally, the results here are primarily descriptive. If these changes in patterns of interdisciplinarity are real, we still know very little about what is driving these changes. Future planned work will use additional waves of collaboration data to build more sophisticated dynamic network models of collaboration. These models will incorporate a richer set of scientist‐level and organizational‐level predictors (e.g., training background, use of internal ICTS resources, participation in internal ICTS funding, general funding environment) to start identifying the causal mechanisms underlying scientific collaboration.

For a number of years, science of team science scholars and large‐scale science funders have stated that it is important for us to establish how interdisciplinary science leads to a stronger scientific evidence‐base that can be better translated into effective practices and policies, and in general represents a greater return on society's investment in the scientific enterprise.22 That is, team science has the potential to break down the single discipline “silos” that are so often representative of the scientific enterprise.23 Before this value proposition can be established, development of appropriate methods for studying interdisciplinary scientific collaboration is required.4 Large‐scale scientific initiatives such as CTSA programs are complex systems, and the heart of these systems are the scientific teams that plan, propose, conduct, and disseminate their research programs. As suggested in this paper, network analysis is an appropriate method for studying changes in interdisciplinary collaborations among scientific teams. Network analytic methods, along with other complex systems approaches such as system dynamics modeling, agent‐based modeling, and other types of computational simulation24 will likely continue to be useful tools as we explore how best to understand and shape the science of team science.

Conflict of Interest

All of the authors with the exception of Amar Dhand were partially funded by the Washington University Institute of Clinical and Translational Sciences grant UL1 TR000448 from the National Center for Advancing Translational Sciences (NCATS) of the National Institutes of Health (NIH). NIH had no role in study design, writing, or decisions to submit this report for publication.

Financial Disclosure

No financial disclosures were reported by the authors of this paper.