A quality assessment framework for transdisciplinary research design, monitoring, and evaluation: Guidance for application

Abstract

Appropriate definitions and measures of quality are needed to guide research design and evaluation. Traditional disciplinary research approaches have well-established evaluation criteria and processes in which research quality is often narrowly defined. In contrast, emerging change-oriented transdisciplinary research (TDR) approaches integrate disciplines and include societal actors in the research process in multiple ways and contexts. Standard research assessment criteria are simply inadequate for TDR, and inappropriate use of standard criteria may disadvantage TDR proposals and impede the development of TDR. This paper presents a Quality Assessment Framework (QAF) designed for TDR, along with guidance for its application. The background outlines the origin of the framework in a systematic review of literature on the definition and assessment of transdisciplinary research quality, and the subsequent application, testing and refinement of the framework, including discussion of key revisions. It also compares the QAF with two other similar evaluation frameworks. The QAF is designed for a range of users, including: research funders and research managers assessing proposals; researchers designing, planning, and monitoring a research project; and research evaluators assessing projects ex-post. The framework is organized as:

• •Four principles of TDR quality
• •Specific criteria aligned with each principle
• •Standardized four-point scoring

Graphical abstract

Specifications table

Subject area	Environmental Science
More specific subject area	Research Evaluation
Name of your method	Transdisciplinary Research Quality Assessment Framework
Name and reference of original method	Original reference: Belcher, B. M., Rasmussen, K. E., Kemshaw, M. R., & Zornes, D. A. (2016). Defining and assessing research quality in a transdisciplinary context. Research Evaluation, 25(1), 1–17.
Resource availability	Transdisciplinary Research Quality Assessment Framework •Prezi:https://prezi.com/p/edit/52ap5yagsezm/ Templates to support application (supplementary files) •Scoring template (Appendix 1) •Spidergraph template (Appendix 2) •Sample comparative and project level spidergraphs (Appendix 3)

Background

Transdisciplinary research (TDR) is a collective term for a range of integrative research approaches that aim to address urgent and complex societal and environmental challenges. Such research, also referred to as use-inspired research, problem-oriented research, sustainability science, and research-for-development, transcends conventional disciplinary and professional boundaries, actively involving non-academic stakeholders in the research process [1,2]. The key features are: problem-orientation; integration across disciplines; stakeholder collaboration; co-creation of knowledge; system perspective, and; practical application [[3], [4], [5]].

TDR poses unique evaluation challenges [6]. Traditional disciplinary research approaches have well-established evaluation criteria and processes in which research quality is narrowly defined, emphasizing scientific excellence and relevance. TDR evaluation needs to also consider broader concepts of practical relevance and societal effects [7,8]. Standard research assessment criteria are inadequate for TDR, and inappropriate use of disciplinary criteria may disadvantage TDR proposals and impede the development of TDR [9]. TDR requires context-responsive ways to guide and evaluate its design and implementation.

Belcher et al. [10] addressed the need for appropriate concepts and measures of TDR quality with a systematic literature review. The review identified valuable insights and perspectives, but no coherent framework for TDR evaluation. Therefore, the authors used the key themes from the literature to develop comprehensive assessment criteria organized under four principles, with precise definitions and a scoring system, presented as the TDR Quality Assessment Framework (QAF) [10].

The QAF has since been tested in several applications (e.g., [[11], [12], [13], [21]]). Two other related frameworks [[14], [15], [16], [17]] have also been developed and tested in the meantime, providing useful additional insights on evaluation framework design and application (discussed in the Method Validation section below).

This experience has been used to refine and improve the QAF. Specifically, we added missing criteria, removed redundant criteria, and refined criteria definitions to ensure that they refer to only one concept, to reduce ambiguity in interpretation and scoring. When applying the QAF, each criterion should be interpreted as an element of the principle that it represents. We also renamed the principle formerly called Effectiveness to Positioning for use, following the term used in the RQ+ framework [17]. Finally, we revised the scoring system from a 3-point scale to a 4-point scale, but still using a standard assessment scale. That is, the same decision rules are applied to each criterion to determine a score, which helps promote consistency and comparability in scoring.

Here we present the updated QAF along with the main considerations for application.

Method details

Considerations for application

The QAF is intended to be used to guide and evaluate research design and implementation at any stage in a project. Below we present the specific steps to be used by a merit review committee to evaluate and rank TDR project (or program) proposals. With slight modifications (mainly changing verb-tense for some criteria), the framework can be used for ex-post evaluation of completed projects. The principles and criteria also provide a useful checklist and guidance for research teams designing TDR projects and/or for assessing progress in an on-going project, and for peer reviews of TDR articles. Table 1 indicates potential applications by user group.

Table 1.

Potential QAF applications by user group.

User Group	Formative	Summative	Comparative
Researchers/Research Teams	Guide Research design & implementation	Individual & team self-assessment	Transparency (i.e. re competitive grants)
Research Managers	Support research design & implementation; delegation	Individual & team assessment	Resource allocation
Assessors/Peer Reviewers/Evaluators	Structure, criteria for feedback	Thorough, consistent, encourage innovation & effectiveness	Consistency, transparency, fairness
Funders	Structure, criteria for feedback	Thorough, consistent, encourage innovation & effectiveness	Consistency, transparency, fairness
Journal Reviewers & Editors	Structure, criteria for feedback	Thorough, consistent, support inter- and transdisciplinarity	Consistency, transparency, fairness

TDR is highly context-specific, potentially involving multiple partners, methodological and epistemological approaches, and a range of objectives. Therefore, each project must be assessed against its own purpose. Many of the criteria include subjective terms like appropriate, suitable, and adequate; these are to be interpreted with reference to the stated aims and intent of the project.

The QAF principles are adapted from Cash et al. [18], building on the insight that research outputs are more likely to be accepted and used if the intended users consider the research to be Relevant, Credible, and Legitimate from their perspective. Cash et al. [18] also discussed the importance of boundary work, represented in the revised QAF principle of Positioning for Use. Each of the assessment criteria should be interpreted according to the principle that it represents.

A defining characteristic of TDR is that the research problem is situated in, and derived from, a broader societal problem context. The QAF therefore clearly distinguishes the two concepts. The problem context is defined as: The social and environmental setting(s) that gives rise to the research problem, including aspects of: location; culture; scale in time and space; social, political, economic, and ecological/environmental conditions; resources and societal capacity available; uncertainty, complexity, and novelty associated with the societal problem; and the system actors and processes [19]. A research problem is defined as: a specific gap in understanding or knowledge that needs to be filled in order to help solve a societal problem. Specific definitions for all key QAF terms and concepts are provided with the criteria (Table 2). Guidance is also provided to assist with applying the criteria and what to look for when making an assessment.

Table 2.

Transdisciplinary research quality assessment framework (adapted from [10]).

Relevance: The importance, significance, and usefulness of the research problem(s), objectives, processes, and findings to the problem context.
Criteria	Definition	Observations
Clearly defined problem context1	The context is well defined, described, and analyzed sufficiently to identify a research problem and corresponding entry points.	•The research documentation demonstrates holistic understanding of the problem context in which the research is situated (description of the system, including actors situated in the context) •Connection is made between the problem context and the research problem •Research entry points are determined by the problem context
Socially relevant research problem2	The research problem is well defined and described, and considers application to the problem context and current academic discourse.	•The research problem is a timely issue in society or aligns with current actions (e.g., international commitments, governmental mandate, policy development, social movements, etc.) •There is a demand from system actors3 for the research problem to be addressed
Engagement with problem context	Researchers demonstrate appropriate4 breadth and depth of understanding of and sufficient interaction with the problem context.	•Understanding drawn from the literature •System actor perspectives are understood •Where possible, researchers incorporate insights from prior research or professional experiences relevant to the problem context
Explicit theory of change	The research explicitly identifies its main intended outcomes5, how they are expected to be realized, and how they are expected to contribute to longer term outcomes and impacts.	•The logic of the research contributions to a process of change is well described and sound •Key actors, processes, and assumptions are identified •End-of-project outcomes are reasonable to expect with the resources available
Relevant research objectives and design	The research objectives are appropriate to the research problem, and the research design is aligned with the objectives.	•Objectives identify what the research project aims to do or produce •Objectives can be justified in how they address the research problem (e.g., fill a knowledge gap) •The research design logically plans how the project will meet the objectives (i.e., identify what methods, activities, and engagement are needed)
Relevant communication6	Communication during and after the research process7 is appropriate to the context and accessible to stakeholders, users, and other intended audiences.	•Communications with system actors help focus the research, source information, and co-generate and share learning •Communications are timely and responsive to other system processes •Communications are tailored to the target audience
Credibility: The research findings are robust and the sources of knowledge are dependable. This includes clear demonstration of the adequacy of the data and the methods used to procure the results, including clearly presented and logical interpretation of findings.
Criteria	Definition	Observations
Broad preparation	The research is based on a strong integrated theoretical and empirical foundation.	•Breadth and depth of literature and theory from relevant disciplines are reviewed and integrated •Empirical demonstration of gaps is based on previous research or interventions, or identified by system actors (e.g., joint problem formulation)
Clear research problem definition	The research problem is clearly stated and defined, researchable, and grounded in the academic literature and problem context.	•A research/knowledge gap is identified •The importance of and need for the research are demonstrated •The research problem can be answered empirically
Clear research question	The research question(s) is clearly stated and defined, researchable, and justified as an appropriate way to address the research problem.	•The research question(s) is logically derived from the research problem •The research question(s) can be answered empirically (i.e., is researchable) •Justification is given on how answering the research question will address the research problem
Comprehensive objectives	Research objectives8 are clearly stated and sufficient to answer the research question(s).	•Objectives are clear, coherent, and feasible •Objectives indicate what knowledge is needed, and how that knowledge will be acquired •Collectively, satisfying all objectives will answer the research question(s)
Feasible research project	The research design and resources are appropriate and sufficient to meet the objectives as stated, and adequately resilient to adapt to unexpected opportunities and challenges throughout the research process.	•Research design is logically derived from the objectives •The project can be completed with the resources available (i.e., budget, time, hardware, software, human capital, and social capital) •Research design is flexible to accommodate unexpected changes
Adequate competencies	The skills and competencies of the researcher(s), team, or collaboration (including academic and societal actors) are sufficient and in appropriate balance (without unnecessary complexity) to succeed.	•The knowledge, skills, and expertise needed to carry out the research are identified •The necessary knowledge, skills, and expertise are represented in the research team
Appropriate research framework	Disciplines, perspectives, epistemologies, approaches, and theories are combined and/or integrated to meet stated objectives and answer the research question(s).	•Explanation of the theoretical/analytical framework is given •Explanation is provided for why and how disciplines, epistemologies, and theories are used •The process of integration of disciplines, epistemologies, and theories is explained, including how paradoxes and conflicts between integrated components are addressed •Justification is given for the framework selected in relation to the problem context
Appropriate methods	Methods are fit to purpose and well suited to achieve the objectives and answer the research question(s).	•Clear descriptions of methods and how they were applied are given •Methods are justified and logically connected to the objectives •Novel (unproven) methods or adaptations are explained and justified, including why they were used and how they maintain rigour
Sound argument	The logic from analysis through interpretation to conclusions is clearly described. Sufficient evidence is provided to clearly demonstrate the relationship between evidence and conclusions.	•The argument is logical and defensible •Analyses and interpretations are adequately explained and supported by evidence •If applicable, alternative explanations of results are explored
Transferability and/or generalizability of research findings	The degree to which the research findings are applicable in other contexts is assessed and discussed. In cases that are too context-specific to be generalizable, aspects of the research process or findings that may be transferable to other contexts and/or used as learning cases are discussed.	•The potential to transfer results and/or methods to other contexts is discussed •Justification of transferability/generalizability of results is logical
Limitations stated	An explanation is given regarding how the characteristics of the research design or method may have influence on the results or conclusions.	•The influence of internal (e.g., sampling) and/or external factors (e.g., responsiveness of interviewees) on the results is acknowledged and discussed •Researcher(s) assess the extent to which the limitations influence the results
Ongoing monitoring and reflexivity9	Researchers engage in ongoing reflection and adaptation of the research process, making changes as new obstacles, opportunities, circumstances, and/or knowledge surface.	•There is an indication that the researcher(s) considers the need to reflect on and adapt during the research process •Efforts to monitor progress and identify, consider, and respond to changes in context or understanding are discussed •Processes of reflection (whether formal or informal), and the resulting action(s) taken, are explained
Legitimacy: The research process is perceived as fair and ethical. This encompasses the ethical and fair representation of all involved and the appropriate and genuine inclusion and consideration of diverse participants, values, interests, and perspectives.
Criteria	Definition	Observations
Disclosure of perspective	Actual, perceived, and potential bias is clearly stated and accounted for.	•Potential for actual or perceived bias (e.g., positionality, sources of funding, partnerships, mandate, etc.) is identified and acknowledged •Implications of potential bias on the conclusions are discussed
Effective collaboration10	Individuals11 involved in the research process pool their knowledge, experience, and skills together in a constructive atmosphere and in appropriate measure to produce new knowledge and/or social processes that contribute to a common goal.	•A shared understanding of goals and expectations is established •Roles and responsibilities are clear and explicitly agreed upon •Decision-making structures are transparent and fair •A synergistic process capitalizes on the strengths of collaborators (across disciplinary, professional, organizational, and cultural boundaries)
Genuine and explicit inclusion12	The research offers authentic opportunities to involve relevant actors to share their perspectives, knowledge, and values, and/or participate in the research process.	•Participants’ roles and contributions, perspectives, and cultural backgrounds are described •Steps taken to ensure the respectful inclusion of diverse actors and views are explained
Research is ethical	The research adheres to standards of ethical conduct.	•Ethical practice is followed: research does no harm; participants have informed consent; anonymity and confidentiality are maintained •Procedural ethics (e.g., ethical review process) are pursued and documented
Positioning for Use: The research process is designed and managed to enhance sharing, uptake, and use of research outputs and stimulates actions that address the problem and contribute to solutions.
Criteria	Definition	Observations
Strategic engagement	The research process stimulates and/or engages with change opportunities.	•Engagements are timely and responsive to other system processes •Researcher(s) is well positioned to have influence within the problem context •Opportunities to influence change processes are identified and/or generated, and acted upon •Resources are mobilized to influence/act on change processes
New knowledge contribution	The research generates new knowledge and understanding in academic and social realms in a timely, relevant, and significant way.	•An academic knowledge gap is filled •System actors’ knowledge gaps are filled •System actors gain a better understanding of the problem context
Influencing attitudes	The research process and/or findings stimulates and supports system actors to reflect on and/or change their attitudes or perspectives on the problem and solutions to address it.	•Awareness-building of the research problem, the research findings, or a solution/innovation is a first step in changing attitudes •System actors gain a different perspective on the targeted problem as a result of the research process and/or findings
Capabilities	System actors develop skills relevant to the problem context and/or skills to solve the societal problem through the research process and/or findings.	•Research capacities of the researcher(s) and/or partners are developed (e.g., gain research experience, training, testing of new methods/approaches) •Participants and partners gain new or build on existing skills as a result of the research process and/or findings •Skills developed are transferable to other aspects of system actors’ professional or personal lives
Relationship-building	The research process supports new or fortifies existing relationships, networks, and ways of working for solution-building in the problem context.	•Trust between system actors is fostered by the research process •Mutual interests between system actors are recognized •A forum, platform, or network is created or strengthened as a result of the research process •System actors work together in new ways as a result of the research process •The research contributes to shifting the power dynamics toward solution-building •Open communication, equality and equity, co-identification/co-development across the research process, feedback processes, and conflict management and resolution are important components of effective relationships
Practical application	The findings, process, and/or products of research have high potential for use by system actors.	•The potential utility of the research outputs for system actors are discussed •System actors convey intentions to use or apply the research •System actors pilot, adopt, or adapt a method, tool, approach, or innovation from the research •System actors use or refer to the research findings to inform their work
Significant results	The research contributes to the solution of the targeted problem or provides unexpected solutions to other problems.	•The research process and/or findings contribute to behaviour change in the problem context •Expected changes are realized or have potential to be realized in the future

¹Problem context refers to the social and environmental setting(s) that gives rise to the research problem, including aspects of: location; culture; scale in time and space; social, political, economic, and ecological/environmental conditions; resources and societal capacity available; uncertainty, complexity, and novelty associated with the societal problem; and the system actors and processes [19].

²A research problem is a specific gap in understanding or knowledge that needs to be filled in order to help solve a societal problem.

³System actors include other researchers, practitioners, policy actors, and intended beneficiaries.

⁴ Words such as ‘appropriate’, ‘suitable’, and ‘adequate’ are used deliberately to allow for quality criteria to be flexible and specific enough to the needs of individual research projects [20].

⁵ Outcomes are defined as “changes in knowledge, attitudes, skills, and relationships manifested as changes in behaviour” ([13], p.9).

⁶Communication refers to both written communication (e.g., proposal, documents, presentation of findings, etc.) as well as engagement communications (e.g., scoping, data collection activities, meetings, workshops, etc.).

⁷Research process refers to the series of decisions made and actions taken throughout the entire duration of the research project and encompasses all aspects of the research project.

⁸Objectives explain what the research will do (i.e., generate specific knowledge, create or facilitate specific processes) and what steps will be undertaken in order to answer the research question(s).

⁹Reflexivity refers to an iterative process of formative, critical reflection on the important interactions and relationships between a research project's process, context, and product(s).

¹⁰ Collaboration encompasses both internal dynamics within the core research team and external processes with participants, collaborators, partners, and allies. Collaboration comes in many forms in research, ranging from general advice-giving to co-generated knowledge production.

¹¹ Within and external to the core research team.

¹² Some system actors may not want to participate in the research process, but still want their views to be represented in the findings. It is the task of the researcher(s) to ensure that their perspectives are accurately represented.

The framework comprises a total of 29 criteria, their definitions, and application guidance. The criteria are organized around the principles of relevance, credibility, legitimacy, and positioning for use (Table 2).

Applying the framework in merit review of project proposals

As discussed above, the QAF can be used in many ways. Here we present guidelines for the most formal application of the framework: a competitive project proposal peer review and ranking. Some aspects of the process described for proposal adjudication will be unnecessary or could be relaxed in other applications.

Before the QAF can be applied, project information needs to be compiled for review. For ex-ante appraisal, the research proposal will be the main document under review, though other supplementary documents that are commonly part of a proposal application (e.g., letters of agreement, curriculum vitae of the research team, profiles of research partners, etc.) will provide a fuller picture of the proposed research project.

• Step 1: Orientation and test scoring

Reviewers are provided with orientation and training in the interpretation, application, and scoring of criteria. This should be reinforced with an exercise where each reviewer independently scores two or three proposals, using the scoring sheet (Supplementary file A1).

• Step 2: Calibration meeting

All reviewers’ scores are compared at a calibration meeting to identify, understand, and manage inconsistencies.

• Step 3. Individual reviewer scoring

In the next step, reviewers are assigned project proposals for appraisal. The reviewers are to assess each proposal against each criterion using the standard four-point scale:

3 = The criterion is fully addressed, with clear evidence and strong intent.

2 = The criterion is addressed well, but with some lack of evidence or clarity.

1 = The criterion is addressed, but incompletely or superficially.

0 = The criterion not addressed or addressed in a misguided way.

• Step 4: Preparation of scoring table to compare reviewer scores

The scoring table (Supplementary file A2) is used to compile scores assigned by each reviewer for each proposal, with average scores and discrepancies (difference between highest and lowest score) calculated for each criterion.

• Step 5: Consensus meeting to discuss, deliberate, and determine final scores

All reviewers meet to reconcile differences and reach consensus on scoring and ranking proposals. A chair may be nominated to guide the discussion. The focus should be on criteria with large discrepancies in scores. Each reviewer can share their perspectives and justifications, which are recorded for transparency. The discussion of justifications may lead reviewers to revise their individual scores in light of new or overlooked evidence. Averages can be used for criteria with minor scoring discrepancies with no need for detailed discussion.

Scores can be presented in a radar chart (spider graph) either individually or as a set (e.g., Fig. 1, Fig. 2, Fig. 3, Fig. 4, Fig. 5). Visualizing the scores this way illuminates individual and collective strengths and weaknesses. Proponents can easily see how their proposals were ranked by principle/criterion and relative to others in the applicant pool. Funders may observe areas in which many proposals score poorly, indicating a need for improved clarity in application guidelines and criteria.

Fig. 1.

Example radar graph comparing average relevance criteria scores for two sample projects.

Fig. 2.

Example radar graph comparing average credibility criteria scores for two sample projects.

Fig. 3.

Example radar graph comparing average legitimacy criteria scores for two sample projects.

Fig. 4.

Example radar graph comparing average positioning for use criteria scores for two sample projects.

Fig. 5.

Example radar graph comparing average QAF principle scores for two sample projects.

Other applications

As noted above, the proposal merit review process, especially for competitive grants, requires the most formal application of the framework. Other applications can use the QAF in less prescribed ways.

Researchers and research managers can use the framework as a checklist to guide research planning and design, and also for monitoring project progress, to identify areas that may need strengthening or adaptation to changing or unanticipated circumstances.

For ex-post project evaluation, there will be more documents available, including the research proposal, meeting minutes, workshop reports, project reports, final reports, publications, and other outputs. These data sources may be supplemented with surveys, interviews, bibliometric analyses, and other secondary data (e.g., policy documents for evidence of policy influence).

Method validation

We have tested the QAF as part of the ex-post evaluation of several large international research-for-development projects [12,21] and in evaluations of completed master- and doctoral-level research projects [[11], [12], [13]]. These applications followed a similar process as described above for proposal review, with orientation and calibration followed by independent scoring by three (and sometimes four) reviewers. Those independent scores were then compared and discussed in detail to identify and understand any inconsistent interpretations of criteria.

We have also learned from the experience with two other related frameworks. The RQ+ framework [17] uses similar principles and considerations as the QAF, but with fewer and broader assessment criteria and a less-precise scoring system. The CGIAR QoR4D Framework [[14], [15], [16]] is a special-purpose framework developed specifically for assessing CGIAR proposals. It is based on the QAF, using the original four principles, but with fewer and overlapping criteria. Table 3 compares the three frameworks.

Table 3.

Comparison to TDR assessment frameworks.

Framework	Framework elements & terms	No. of criteria	Scoring	Mapping criteria to principles	Testing
TDR QAF ([10;22])	Four Principles (relevance, credibility, legitimacy, positioning for use) with corresponding criteria	29	4-point Likert scale: standardized assessment rubric	Single-factor criteria correspond to one principle	Evaluations of completed research projects [11,12]
RQ+ ([23;17])	Contextual factors plus four Dimensions (scientific rigour, research legitimacy, research importance, positioning for use) with corresponding sub-dimensions of research quality	15	4-level rubric with 8-point scoring scale (2 possible scores per performance level): unique rubric statements for each criterion	Internal and external contextual factors; Multi-factor sub-dimensions (i.e. criteria) correspond to one dimension	Meta-analysis of use-oriented research [24] Evaluation of co-production by 18 subject matter experts [25]
QoR4D [[14], [15], [16]]	Four Elements (relevance, credibility, legitimacy, effectiveness) with corresponding criteria	17 (2021) 12 (2024)	4-point Likert scale; standardized assessment rubric	Multi-factor criteria correspond to 2 or 3 elements	Analysis of external review of research program proposals [14,15]

The tests of the QAF revealed some overlapping and some missing criteria. We removed, added, and refined criteria as necessary. For example, the criterion Clear research question was added under the Credibility principle to fill a clear gap. A criterion called Appropriate project implementation under Relevance was found to be insufficiently clear and could be interpreted in a way that was redundant with other criteria. It was therefore deleted, but the intended essence of was the criterion was captured in the revised principle and criteria for Positioning for use. Some inconsistencies in the interpretation of criteria could be traced back to ambiguous definitions. Substantial effort was invested to refine and improve definitions for clarity and precision and to ensure that each criterion refers to one and only one concept and corresponds to only one principle. This consideration was reinforced by experience with the other frameworks. In both the QoR4D and RQ+ frameworks, a single criterion may refer to more than one concept. In the QoR4D framework, each criterion refers to several concepts and corresponds to two and sometimes three principles. This makes interpretation ambiguous and scoring more difficult than in the QAF.

It also became apparent that the criteria definitions and the rubric statements in the 2016 version were largely redundant. We therefore eliminated the rubric statements and instead provided a set of cues for consideration in applying the criteria (i.e. the observations column).

In the first applications, we used a 3-point Likert scale where a 0 was assigned if a criterion was not satisfied, a 1 if a criterion was partially satisfied, and a 2 if a criterion was fully satisfied. In practice, the 3-point scale proved to be too coarse, with too many criteria assigned an intermediate score of 1. We therefore recommend using a four-point scale as detailed above. We have not yet applied the QAF using the four-point scale, but it was adopted and used successfully in two rounds of CGIAR proposal evaluations using the QoR4D framework [14,15].

We also renamed the principle formerly called Effectiveness to Positioning for Use, following the term used in the RQ+ framework [17]. This was done because the term Effectiveness tended to influence assessors to interpret the related criteria in terms of whether changes had already been realized rather than, as intended, whether the research process was well designed and managed to enhance sharing, uptake, and use.

Limitations

The TDR QAF is designed for the evaluation of use-inspired transdiciplinary research. That is, research that integrates disciplines and epistemologies and engages other system actors to contribute, directly or indirectly, to positive social, economic and/or environmental impacts. The scope of transdiciplinary research is, however, quite broad. This assessment framework will not necessarily apply equally well to all research that is called transdiscplinary.

The first version of the framework has been tested and refined in evaluations of completed projects, but not yet in proposal review. An ideal test of the method would be based on evaluation by independent reviewers of a suitable sample of TDR proposals prepared by proponents with reference to the assessment criteria.

Finally, the method as described gives equal weight to each principle and each criterion. Users may choose to assign different weights to principles and/or criteria.

Ethics statements

N/A

Credit author statement

Brian Belcher: Conceptualization, Validity tests, Comparative analysis, Writing-Original draft preparation.

Rachel Claus: Validity tests, Data management, Visualization, Writing-Reviewing and Editing.

The work has been done as part of a Tier 1 Canada Research Chair and a subsequent Ashoka Research Chair in Sustainability Research Effectiveness.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix. Supplementary materials

Supplementary material and/or additional information

QAF Scoring template (Appendix 1)

Blank Spidergraph template (Appendix 2)

Sample project level and comparative spidergraphs (Appendix 3).

Data availability

Data will be made available on request.