Table 1.
Measures of research collaboration quality
| Author | Instrument name | Construct being measured | Sample/ population |
Number of items | Response options | Reliability | Validity |
|---|---|---|---|---|---|---|---|
| Bietz et al. [36] | Collaboration Success Wizard | Past, present, or future perceptions of five factors: nature of work; common ground; collaboration readiness; management, planning and decision making; and technology readiness | 177 university faculty and staff from 12 projects | 44 | Varied | NR | NR |
| Greene et al. [37] | CRN Participant Survey | Five domains: extent of collaboration and quality of communication; performance of projects and infrastructure; data quality; scientific productivity; and impact on member organizations | Investigators and project staff from the HMO Cancer Research Network over a 5-year period | All items not provided; questions modified annually based on feedback and new project needs | Several 5-point Likert scales ranging from “strongly agree” to “strongly disagree” or from “very effective” to “very ineffective, including “can’t evaluate”; also, open-ended items to collect qualitative input | ||
| Hall et al. [12] | Research Orientation Scale | Unidisciplinary, multidisciplinary, or inter/transdisciplinary proclivity of values and attitudes toward research | 56 investigators and staff from four NCI TREC centers | 10 | 5-point Likert scale ranging from “strongly agree” to “unsure” to “strongly disagree” | Cronbach’s alpha = 0.74 | Construct validity: Exploratory and confirmatory factor analyses supporting three factors. Convergent validity: Higher unidisciplinary orientation inversely correlated with cross-disciplinary collaborative activities and multidisciplinary and inter/transdisciplinary research orientation. Higher multidisciplinary orientation correlated with more collaborators, more cross-disciplinary collaborative activities. Similar findings for inter/transdisciplinary research orientation |
| Completing Deliverables Scale | Investigators’ expectations for their projects’ meeting projected year-1 deliverables | One item for each project | 5-point Likert scale ranging from “highly unlikely” to “highly likely”; each project was rated separately | NA | Convergent validity: Inverse correlation between duration of involvement in transdisciplinary projects at their center and researchers’ confidence in meeting year 1 deliverables | ||
| Hall et al. [12]a | History of Collaboration (other individual investigators) | Number of individuals collaborated with and length of time for each individual | 56 investigators and staff from four NCI TREC centers | 2 | Count of collaborators; number of years of collaboration for each collaborator | Convergent validity: Number of collaborators correlated with more center-related collaborative activities, number of years working in inter/transdisciplinary centers, number of years working with inter/transdisciplinary projects, higher multidisciplinary, and inter/transdisciplinary research orientation | |
| History of Collaboration (other centers or projects) | Number of years in inter/transdisciplinary center and projects | 4 | Counts of number of centers and of years involved with center; counts of number of projects and number of years involved with projects | Convergent validity: More years involved in inter/transdisciplinary centers correlated with number of collaborators and inversely correlated with confidence in completion of 1-year deliverables. More years involved in inter/transdisciplinary projects correlated with number of collaborators and inversely correlated with confidence in completion of 1-year deliverables | |||
| Satisfaction with Collaboration | Satisfaction with each individual collaborator | 1 | 5-point Likert scale ranging from “not at all satisfied” to “neutral” to “completely satisfied” | Convergent validity: More satisfaction with individual collaborations correlated with more perceived institutional resources supporting collaboration, more positive impressions, higher ratings of interpersonal collaborations, higher perception of collaborative productivity | |||
| Cross-Disciplinary Collaboration-Activities Scale | Frequency of engagement in collaborative activities outside of his/her primary field | 6 | 7-point Likert scale ranging from “never” to “weekly” | Cronbach’s alpha = 0.81 | Convergent validity: higher frequency of cross-disciplinary collaborative activities correlated with stronger multidisciplinary and interdisciplinary/transdisciplinary research orientation | ||
| TREC-related Collaborative Activities Scale | Frequency of engagement with center-specific activities | 3 | Cronbach’s alpha = 0.74 | ||||
| Institutional Resources Scale | Availability and quality of institutional resources for conducting collaborative research project | 8 | 5-point Likert scale ranging from “very poor” to “excellent” | Cronbach’s alpha = 0.87 | Convergent validity: The better the perceived resources, the more positive researchers’ perception of center, more satisfied with previous collaborators, more positively rated collaborative productivity and interpersonal collaboration | ||
| Semantic-Differential/ Impressions Scale | Investigators’ impressions of their research center | 21 adjective pairs | 7-point continuum on which respondents rate their impressions on adjective pairs (e.g., conflict-harmonious, not supportive-supportive, fragmented-integrated) | Cronbach’s alpha = 0.98 | Convergent validity: Better impressions were correlated with more collaboration satisfaction, more confidence in completion of deliverables, more institutional resources for collaboration, better interpersonal collaboration | ||
| Interpersonal-Collaboration Scale | Interpersonal collaborative process at their center | 8 | 5-point Likert scale with response options ranging from either “very poor” to “excellent” or “strongly agree” to “strongly disagree” with central “neither agree nor disagree” | Cronbach’s alpha = 0.92 | Convergent validity: The better the interpersonal collaboration, the more collaboration satisfaction, the more confidence in completion of deliverables, and the more perceived institutional resources for collaboration | ||
| Hall et al. [12]b | Written Products Protocol | The integrative (transdisciplinary) aspects of written research protocols, disciplines represented, levels of analysis, type of cross-disciplinary integration | 21 center developmental project proposals from four NCI TREC centers | 37 item protocol used to evaluate proposals | Items describing proposal with various response formats; one item – rate whether “unidisciplinary,” “multidisciplinary,” “interdisciplinary,” or “transdisciplinary” proposal, two items regarding transdisciplinary integration and scope of proposal using 10-point Likert scale ranging from “none” to “substantial” | Inter-rater reliabilities based on Pearson’s correlations from 0.24 to 0.69; highest reliability for rating experimental types (0.69), number of analytic levels (0.59), disciplines (0.59) and scope (0.52). Lower reliability in attempts to name the cross-disciplinary integration in the proposal | Convergent validity: Higher number of disciplines in proposal, the broader its integrative score, larger its number of analytic levels. The higher the type of disciplinarity, the broader its overall scope |
| Huang [34] | Trust | Team trust | 290 members of 60 technology research and development teams from the Industrial Technology Research Institute in Taiwan | 7 | 5-point Likert scale ranging from “strongly disagree” to “strongly agree” | Cronbach’s alpha = 0.87; ICC = 0.46 (p < 0.001) | Face and content validity. Construct validity: Common method variance analysis and confirmatory factor analysis using partial least squares latent structural modeling. Convergent validity with other constructs with composite reliability > 0.6 and average variance extracted at least 0.5. Discriminant validity with square root of average variance for construct greater than levels of correlations involving construct |
| Transactive Memory System | Team transactive memory system | 7 | Cronbach’s alpha = 0.84; ICC = 0.42 (p < 0.001) | ||||
| Knowledge Sharing | Team knowledge sharing | 4 | Cronbach’s alpha = 0.79; ICC = 0.55 (p < 0.001) | ||||
| Group cohesiveness | Group cohesiveness (team network ties and collective mind) | 5 | Cronbach’s alpha = 0.78; ICC = 0.42 (p < 0.001) | ||||
| Team Performance | Team performance | 4 | Cronbach’s alpha = 0.82; ICC = 0.53 (p < 0.001) | ||||
| Lee and Bozeman [38] | Collaboration Strategies | Collaboration strategies or motives for collaboration | 443 science faculty affiliated with NSF or DOE research centers at US universities | 13 | 4-point Likert scale ranging from “very important” to “not important” | Cronbach’s alpha for subscales: Taskmaster (2 items) = 0.60; Nationalist (2 items) = 0.57; Mentor (2 items) = 0.57; Follower (3 items) = 0.42; Buddy (3 items) = 0.32; Tactition only one item. No overall Cronbach’s alpha reported | Construct validity: Exploratory factor analysis with varimax rotation supporting six factors: Taskmaster, Nationalist, Mentor, Follower, Buddy, and Tactition. Convergent validity: Nationalist and Mentor collaboration strategies/motives significantly associated with number of collaborators during past 12 months. Only Tactition strategy/motive significantly associated with journal publication productivity as measured by a normal count and a fractional count of publications during the 3 years post survey |
| Collaboration | Number of collaborators | 1 | Count of number of persons, by category, with whom they engaged in research collaborations within the past 12 months. Categories were male university faculty, male graduate students, male researchers who are not university faculty or students, female university faculty, female graduate students, and female researchers who are not university faculty or students | NA | Convergent validity: Zero order correlations between number of collaborators and journal publication productivity, both normal count and fractional count. Two-stage least squares regression results, including other moderating variables, demonstrated a continued significant relationship between number of collaborators and normal publication count | ||
| Mallinson et al. [39] | MATRICx | Motivators and threats to collaboration readiness | 125 faculty, students, researchers | 48 (31 threat and 17 motivator items) | 4-point Likert scale: 4 = describes me/my experience exactly; 3 = describes me/my experience quite well; 2 = somewhat describes me/my experience; 1 = does not describe me/my experience at all | Rasch analysis: Person separation reliability for threat items = 0.92; motivator items (experienced participants) = 0.94; motivator items (inexperienced participants) = 0.85; all items = 0.67 | Construct validity: Rasch analysis and principal components analysis |
| Mâsse et al. [19] c | Transdisciplinary Integration Scale | Attitudes about transdisciplinary research | 216 research faculty, staff, trainees from NCI TTURC | 15 | 5-point Likert scale ranging from “strongly agree” to “unsure” to “strongly disagree” | Cronbach’s alpha = 0.89 | Construct validity: Confirmatory factor analysis. Convergent validity: Correlations with center outcomes |
| Satisfaction with Collaboration | Satisfaction with collaboration within a center | 8 | 5-point Likert scale ranging from “inadequate” to “excellent” | Cronbach’s alpha = 0.91 | Construct validity: Confirmatory factor analysis. Convergent validity: Satisfaction with collaborations within a center correlated with center outcomes related to methods, science and models, and improved interventions | ||
| Impact of Collaboration | Impact of collaboration within a center | 5 | 3 items (meeting, products, overall productivity)-5-point Likert scale ranging from “inadequate” to “excellent”; remaining two items (research productivity, quality research)-5-point Likert scale ranging from “strongly disagree” to “not sure” to “strongly agree” | Cronbach’s alpha = 0.87 | Construct validity: Confirmatory factor analysis. Convergent validity: Impact of collaborations within a center correlated with center outcomes related to methods, science and models, and improved interventions | ||
| Trust and Respect | Trust and respect with collaborations | 4 | 5-point Likert scale ranging from “strongly disagree” to “not sure” to “strongly agree” | Cronbach’s alpha = 0.75 | Construct validity: Confirmatory factor analysis. Convergent validity: Trust and respect with collaborations within a center correlated with center outcomes related to methods, science and models, and improved interventions | ||
| Mazumdar et al. [40] | No formal name | Team scientists’ activities related to grant design, grant implementation, grant analysis, manuscript reporting, teaching, and service | Proposed for academic faculty | 6 | 3 possible ratings for each item: major, moderate, or minor; Supported by qualitative comment by reviewer | NR | |
| Misra et al. [28] | Transdisciplinary Orientation Scale | Values, attitudes, beliefs, conceptual skills, knowledge and behavioral repertoires that predispose an individual to collaborating effectively in cross-disciplinary scientific teams | 150 researchers and academics from the liberal arts, social sciences, natural sciences, and engineering | 12 | 5-point Likert scale ranging from “strongly agree” to “strongly disagree”; middle three response options not anchored | Overall Cronbach’s alpha = 0.93 (values, attitudes, beliefs 6-item subscale alpha = 0.87; conceptual skills and behaviors 6-item subscale alpha = 0.88); second sample Cronbach’s alpha = 0.92 | Construct validity: Confirmatory factor analysis. Convergent validity: Critical ratio of regression weights statistically significant. Discriminant validity: Covariance between two factors and correlation high but suggests discriminant validity. In multiple regression analyses, higher transdisciplinary orientation associated with production of more interdisciplinary scientific papers, more experience in participating in cross-disciplinary team science, and independent ratings of the potential society impact of the research reported in the scholar’s article |
| Oetzel et al. [23]d | Bridging Social Capital | Academic and community partners have the skills and cultural knowledge to interact effectively Domain: Structural/individual dynamics |
138 PIs/PDs and 312 academic or community partners from 294 CBPR projects with US federal funding in 2009 | 3 | 5-point Likert scale: 1=not at all; 2=very little; 3=somewhat; 4=mostly; 5=to a great extent |
Cronbach’s alpha = 0.69 | Construct validity: Confirmatory factor analysis. Convergent validity: Positive and moderate correlations with other structural/individual dynamics scales; and correlations with multiple outcomes |
| Alignment with CBPR Principles: Partner Focus | Develops individual partner capacity and equitable partnerships in all phases of the research Domain: Structural/individual dynamics |
4 | Cronbach’s alpha = 0.82 | ||||
| Alignment with CBPR Principles: Community Focus | Builds on resources and strengths of community for the well-being of community Domain: Structural/individual dynamics |
4 | Cronbach’s alpha = 0.85 | ||||
| Partner values | Shared understanding of project mission, priorities, strategies Domain: Structural/individual dynamics |
4 | 5-point Likert scale: 1 = strongly disagree; 2 = disagree; 3 = neither agree nor disagree; 4 = agree; 5 = strongly agree |
Cronbach’s alpha = 0.89 | |||
| Research Tasks and Communication: Background Research | Community partners’ level of involvement in the background research Domain: Relational dynamics |
5 | 1 = community partners DID NOT/DO NOT participate in this activity; 2 = community partners were/are CONSULTED on this activity; 3 = community partners were/are ACTIVELY ENGAGED in this activity; 4 = not at this stage of research; 5 = does not apply | Cronbach’s alpha = 0.81 | Construct validity: Confirmatory factor analysis. Convergent validity: Positive and moderate correlations with other relational dynamics scales; and correlations with multiple outcomes | ||
| Research Tasks and Communication: Data Collection | Community partners’ level of involvement in the data collection Domain: Relational dynamics |
4 | Cronbach’s alpha = 0.69 | ||||
| Research Tasks and Communication: Analysis and Dissemination | Community partners’ level of involvement in data analysis and dissemination of findings Domain: Relational dynamics |
3 | Cronbach’s alpha = 0.82 | ||||
| Dialogue and Mutual Learning: Participation | Degree to which all partners participate in the process Domain: Relational dynamics |
3 | 5-point Likert scale: 1 = strongly disagree; 2 = disagree; 3 = neither agree nor disagree; 4 = agree; 5 = strongly agree |
Cronbach’s alpha = 0.78 | |||
| Dialogue and Mutual Learning: Cooperation | Degree to which partners cooperate to resolve disagreements Domain: Relational dynamics |
3 | Cronbach’s alpha = 0.83 | ||||
| Dialogue and Mutual Learning: Respect | Degree to which partners convey respect to each other Domain: Relational dynamics |
3 | Cronbach’s alpha = 0.83 | Construct validity: Confirmatory factor analysis. Convergent and discriminant validity: Mixed results | |||
| Trust | Degree of current trust within partnership Domain: Relational dynamics |
4 | Cronbach’s alpha = 0.86 | Construct validity: Confirmatory factor analysis. Convergent validity: Positive and moderate correlations with other relational dynamics scales; and correlations with multiple outcomes | |||
| Influence and Power Dynamics | Degree of voice and influence in the decision-making Domain: Relational dynamics |
3 | Cronbach’s alpha = 0.58 | ||||
| Participatory Decision-Making | Degree to which decisions are made in a participatory manner Domain: Relational dynamics |
4 | 5-point Likert scale: 1 = never; 2 = rarely; 3 = sometimes; 4 = often; 5 = always |
Cronbach’s alpha = 0.83 | |||
| Leadership | Overall effectiveness of project’s leadership Domain: Relational dynamics |
10 | 5-point Likert scale: 1 = very ineffective; 2 = ineffective; 3 = somewhat effective; 4 = effective; 5 = very effective |
Cronbach’s alpha = 0.94 | |||
| Resource Management | Effective use of financial and in-kind resources Domain: Relational dynamics |
3 | 5-point Likert scale: 1 = makes poor use; 2 = makes fair use; 3 = makes average use; 4 = makes good use; 5 = makes excellent use |
Cronbach’s alpha = 0.86 | |||
| Okamoto et al. [41] | Conflict | Task conflict | 167 center directors, research core PIs, individual project PIs, key research personnel from 10 US National Institutes of Health Centers for Population Health and Health Disparities | 6 | 5-point Likert scale from “not at all” to “to a very large extent” | Cronbach’s alpha = 0.82 | Discriminant validity: Task conflict not associated with any of the three network measures |
| Wooten et al. [27] | Team Evaluation Model Matrix | Functioning of multidisciplinary translational teams within a two-by-two matrix based on assessment of two dimensions: Team maturation/development and; research and scientific progress | 11 Multidisciplinary translational research teams within one US National Institutes of Health, National Center for Advancing Translational Science, Clinical and Translational Science Award center | 2 | Expert panel members rated each team on each two dimensions (Maturation/development; and Research/scientific) on a scale with 0 = not present; 1 = low; 2 = medium; 3 = high. On each dimension could have a final score of 0–12 (0–3 scores for each of four criteria under each dimension). After initial scoring by individual panel members, total expert panel discuss to reach final consensus on each team’s scores on each of the two dimensions. Initial ratings based on expert panel members’ review of team logic model, measurement plan, and all assessment data (including survey data) | NR | NR |
NR, not reported; CRN, Cancer Research Network; NCI, US National Cancer Institute; TREC, Transdisciplinary Research on Energetics and Cancer; ICC, intraclass correlation coefficients; NSF, US National Science Foundation; DOE, US Department of Energy; MATRICx, Motivation Assessment for Team Readiness, Integration and Collaboration; NA, not applicable; TTURC, Transdisciplinary Tobacco Use Research Center; PI, principal investigator; PD, project director; CBPR, community-based participatory research.
a
Details obtained by cross-referencing article (TREC Baseline survey) from https://www.teamsciencetoolkit.cancer.gov/Public/TSResourceMeasure.aspx?tid=2%26rid=36 [42].
b
Detail obtained by cross-referencing article (NCI TREC Written Products Protocol 2006-09-27) from https://www.teamsciencetoolkit.cancer.gov/Public/TSResourceMeasure.aspx?tid=2%26rid=646 [43].
c
Details obtained by cross-referencing article (TTURC Researcher Survey 2002) from https://cctst.uc.edu/sites/default/files/cis/survey-TTURC_research.pdf [44].
d
Original instrument shown at http://cpr.unm.edu/research-projects/cbpr-project/index.html--scroll to 2. Quantitative Measures – “Key Informant” and “Community Engagement” survey instruments. Developmental work on measures from Oetzel et al. (2015) continues in an NIH NINR R01 (Wallerstein [PI] 2015-2020 “Engage for Equity” Study; see http://cpr.unm.edu/research-projects/cbpr-project/cbpr-e2.html).