Fostering reflective impact orientation in transdisciplinary research–A multi-method workshop format

Abstract

The primary objective of transdisciplinary research (TDR) is to contribute to the solution of complex ‘real-world’ problems by integrating heterogeneous knowledge and achieving societal effects. However, establishing a continuous impact orientation during TDR processes remains a challenge, as the necessary tools are not yet sufficiently available.

We developed and tested a half-day workshop format for strengthening the impact-oriented project management and research activities of seven TDR projects. Our findings indicate that the reflective impact workshops supported participants in pursuing societal effects systematically. Applying the methodological approach also fosters TDR process qualities such as knowledge integration. Conducted at different project stages, the results can serve as a basis for monitoring and adapting the project design.

The reflective approach

• •includes scientific and non-scientific TDR project team members,
• •draws on Theory of Change as a conceptual framework and motivates participants to reflect on plausible impact pathways and make implicit assumptions about interlinkages between different forms of societal effects explicit, and
• •provides results which enable project partners to adjust their project design for greater societal effectiveness.

Graphical abstract

Specifications table

Subject area:	Earth and Planetary Sciences
More specific subject area:	Process evaluation for impact-oriented project management of transdisciplinary research
Name of your method:	Impact Reflection Workshop
Name and reference of original method:	Theory of Change Dhillon, L., & Vaca, S. (2018). Refining theories of change. Evaluation, 14(30), 64–87. Impact Heuristic Schäfer, M., M. Bergmann, L. Theiler. (2021). Systematizing societal effects of transdisciplinary research. Research Evaluation 30(4), 484–499. https://doi.org/10.1093/reseval/rvab019 Sustainability Analysis of Intervention Benefits: A Theory of Change Approach Mayne, J. (2020). Sustainability analysis of intervention benefits: A theory of change approach. Canadian Journal of Program Evaluation, 35(2), 204–221.
	Framework for Designing Formative Evaluation of Transdisciplinary Research Wiefek, J., Nagy, E., & Schäfer, M. (2024). Formative evaluation of transdisciplinary research for systematic impact orientation in real-world laboratories. GAIA-Ecological Perspectives for Science and Society, 33(1), 94–101.
Resource availability:	Supplementary Materials A) A Step-by-Step Guide with Conceptual and Methodological References B) Evaluation Form C) Guidelines for interviews with project partners from science and practice D) Short description of the project example

Method details

Background

Transdisciplinary research (TDR) stands out by focusing on integrating knowledge from a diverse range of perspectives to achieve not only scientific, but also societal effects [1,2]. A widespread consensus is that a thoroughly designed TDR process involving relevant scientific and non-scientific actors yields results that increase the potential for societal effects [[3], [4], [5], [6]]. Hence, for this ambitious endeavour, TDR requires tools and methods that enable reflexivity and social learning within the whole project team, followed by adaptive management [7,8]. It is strongly recommended, therefore, that the team engages early on and continuously with both intended and non-intended effects ([9]; Verwoerd et al., 2020; [[10], [11], [12]]). This creates “a preliminary basis for developing strategies regarding participation, integration or transfer as major pillars of TDR and for the reflection of processes, results and products during the project” ([5], p. 188f.). The importance of impact-orientation in TDR seems to be widely accepted. A key question, however, is how systematic reflection on, and early anticipation of, societal effects can be methodically integrated and supported in the practice of TDR projects.

In recent years, several approaches and frameworks have been developed to conceptualize the monitoring and assessment of societal effects of participatory, problem-oriented and transdisciplinary research [[13], [14], [15]]. Some of the recently introduced frameworks also include the anticipation of societal effects [16,17,12]. Most of these approaches, however, focus on an encompassing impact evaluation and hence build on relatively complex designs, demand profound expertise in evaluation, and extensive time and personnel resources. For TDR teams on the ground, however, a feasible approach to initiating impact reflection is considered essential. Moreover, TDR projects need a formative approach that enables them to regularly review and adapt the process.

This article presents an integrative workshop format for reflecting on societal effects in TDR projects in order to enable impact-oriented project management. The workshop format encompasses several benefits for the TDR project teams:

• -Systematic reflection on the intended or achieved effects of a TDR project (facilitated by visualization),
• -A shared understanding and prioritization of intended effects,
• -The disclosure of (heterogeneous) perspectives and assumptions about how the project activities bring about societal effects,
• -The utilization of insights into gaps and risks for adaptive management in TDR processes,
• -A basis for monitoring project success in terms of outputs and societal effects.

The workshop formats development has been informed by the following approaches and concepts discussed in literature concerned with reflection on and evaluation of the impacts of TDR.

Participation of scientific and non-scientific actors: Several authors emphasize the inclusion of both scientific and non-scientific actors in the discussion of intended and non-intended effects [4,5,15,17,18]. A common understanding of targeted impact pathways in the TDR team is considered essential in achieving the intended effects [19,20]. In order to foster a rigorous impact orientation, we have made collaboration between scientific and non-scientific actors the fundamental premise of our approach.

Using a Theory of Change approach: We use Theory of Change (ToC) as a framework that has recently gained importance in dealing with the complex process of impact generation in TDR [21,22]. ToCs “are defined as the mental representations and theoretical assumptions that explain how and why activities of an initiative (e.g., projects, programmes, organizations) generate particular changes” ([8], p. 106). Theory of Change is both a process and a product: the process of working out the theory (mainly in group sessions involving practitioners), and the product of that process (a visualization of the change model and a verbal explication of underlying assumptions) [23]. A ToC is usually elicited by extrapolating impact pathways back from a common vision [20,24,25]. Starting a ToC process by establishing a shared vision for the project in focus allows us to acknowledge the normativity of deliberately contributing to change and working towards a common societal goal in the future [26]. Based on the common vision, the TDR team can formulate the effects that are intended to be achieved through the project activities and results. Thus, the impact pathways in a ToC link the project activities and outputs to the joint vision via the intended effects. A ToC therefore also enables forward impact-oriented planning and reflection. Discussing possible mechanisms behind the pathways reveals both uncertainties and risks and can lead to strategic adjustments in the research design [16,27,28]. ToC thinking serves as a mode of joint knowledge production and helps to make potentially different understandings among involved actors explicit, especially when employed in workshop settings [29,8,15]. This is why we draw on ToC thinking in our methodical approach.

Analysis of actors: For effective TDR, several authors recommend reviewing actor composition (Retolaza Eugen, 2011; [5,30]). Actors who play a supportive role are just as important as those who are expected to be reluctant or even block the processes. It is also important to identify and address suitable stakeholders and prepare them to take on responsibility for the continuation and transfer of the results [31]. It is therefore strongly recommended that, during the research processes, the team considers the participating actors’ relevance to the societal problem at stake [5]. Thus, our approach includes a brief analysis of actors.

Analysis of activities, interactions and outputs: Societal effects may arise from both the activities carried out in TDR processes and the relevant outputs resulting from TDR. Outputs are defined as the direct, tangible results of the project, e.g. products such as publications, policy papers, models or prototypes [32,33]. The achievement of societal effects is considered to be dependent on the composition and quality of the activities and outputs [3,5,34]. As regards the TD research process, we build on the concept of productive interactions by Spaapen & van Drooge [35] and also Binder et al. [18], which assumes that productive interactions with stakeholders may lead, for example, to changes in their behaviour, such as initiating new actions or taking on responsibility for the continuation of project results. Spaapen & van Drooge [35] consider interactions such as direct or personal engagement, indirect interactions though text or artefacts, and financial interactions as productive. Based on our experience with formative evaluation of TDR, the direct interactions between relevant actors have the most impact potential. Our approach therefore includes an analysis and discussion of activities, direct interactions and outputs as a basis for creating impact potential.

Using an impact heuristic for different forms of societal effects: Societal effects are characterized as changes taking place at the level of individuals, groups and organizations, or as broader societal transformations (inspired by ICPHR [[10], [36]]) which can manifest in different forms. Several empirical and conceptual studies have advanced categorizations of the societal effects of TDR [26,29,37,38]. Based on the existing literature and their own empirical findings, Schäfer et al. [11] proposed a heuristic encompassing a set of categories and sub-categories for societal effects that may occur within three orders:

• •First-order effects: changes that occur within the duration and/or spatial scope of the project. Possible forms: network formation, learning processes, capacity building, changes in individual and organizational practices.
• •Second-order effects: changes that are observable or anticipated after the conclusion of a project. They occur within the close temporal and/or spatial context of the project (approx. 1–3 years). Possible forms: continuation of activities in the project context or transfer to other spatial contexts.
• •Third-order effects: changes that occur beyond the temporal and/or spatial context of the project in the respective field of action. These types of change are based on the contributions of multiple actors and can seldom be attributed to single research projects. Possible forms: contribution to changes in public discourse, changes in regulation, changes in infrastructure.

First-order effects are also referred to in the societal transformation literature as “direct” effects (directly triggered by project measures), which can lead to various “indirect” or mediated effects in the second order [39]. In our impact reflection workshop, the heuristic by Schäfer et al. [11] serves as an underlying analytical framework for systematizing effects.

Iterative impact reflection as a basis for adaptive project management: Impact reflection and evaluation are widely regarded as a continuous endeavour throughout, and, ideally, even after the conclusion of a TDR project [8]. Several impact evaluation approaches suggest addressing impact reflection at the beginning (ex-ante), during (mid-term) and/or after the conclusion (ex-post) of a research project [16,26,40]. Belcher et al. [16] emphasize that if ToC is “developed ex ante it is important to revisit and iteratively revise the ToC throughout the project lifespan to ensure that activities and partnerships are sufficient and appropriate to achieve the intended outcomes” (ibid., p. 10). A jointly developed ToC should therefore be treated as a “living and dynamic product” to enable continuous learning and impact-relevant adjustment ([21], p. 37; [17]). The suggested workshop format is suitable for periodic implementation throughout the duration of the research project and beyond.

Method validation

We applied the half-day structured workshop format to seven TDR projects, the aim being to foster systematic impact reflection among the core scientific and non-scientific members of the TDR teams.

Selection and characteristics of the supported projects: To acquire suitable TDR projects, we sent out a call to German-speaking TDR communities and research institutes in autumn 2020. We transparently communicated the purpose (exploratory study to test the workshop format) and the time and resources required for participation. We received 22 applications from Germany, Austria and Switzerland. Projects were selected based on the criteria of the stage of the project at the time of the study, the project's duration, and the topic focus within the broad field of sustainability research. All projects in the sample shared a geographical focus on the regional level, which is predominant in TDR. We selected two emerging, three ongoing and two finished TDR projects in the fields of agricultural innovation, energy transition, resource efficiency, climate adaptation, urban mobility, rural development and urban economy. The initial workshops were conducted with all seven project teams in 2021 (first workshop round), and follow-up workshops with four ongoing projects 9–12 months later in 2022 (second workshop round). Each workshop was visually documented, and the discussions were recorded and transcribed for a follow-up analysis.

Evaluation of the workshop format: At the end of each workshop, participants were asked for verbal feedback and to complete an anonymous online survey. The questions concerned the quality of the workshop facilitation (preparation, moderation, atmosphere), the participants’ workshop experience (quality of knowledge integration for a common understanding of project goals, intended effects and respective activities) and the workshop format (applicability for their own purposes, suggestions for methodological improvement). The survey included rating and free-text questions (s. Supplement B). During the workshops, we also took notes on how well the workshop elements worked individually and in combination. The results of the evaluation are described in the section Results of the evaluation of the workshop format.

Workshop format for reflecting on the effectiveness of TDR projects

Preparation1

When conducting impact workshops, facilitators must have an understanding of the project and the actors and perspectives involved. As external workshop facilitators, we found it fruitful to combine analysis of available project documents (e.g. project proposals) with interviews with the scientific project leader and/or coordinator and at least one non-scientific project partner (s. Table 1). This kind of preparation enables facilitators to gain insight into the context and goals of the project. They get a first impression about the societal problem and how it affects different stakeholder groups, the former endeavours to address the problem, the relationships between the project partners and to other actor groups, and the intended effects and possible negative societal effects from different points of view. Based on these insights, it is possible to develop a comprehensive overview of planned activities, interactions and outputs, and to prepare a tentative list of societal effects the project intends to achieve as a basis for ‘ToC thinking’.

Table 1.

Preparatory steps to gaining an understanding of the supported project.

Analysis of project documents (e.g. proposal, reports, website), approx. 2 days	Codes for the analysis: -actors who are part of the TDR project team -actors who have an influence on the project success in the same context -project activities -interactions within the project (between scientists and non-scientists, between non-scientists) -outputs/milestones -intended societal effects in the project context -intended societal effects beyond the project context -higher-level transformations the project aims to contribute to -assumptions about how project activities, interactions and outputs are linked to effects -assumptions about preconditions for the occurrence of societal effects
Interview with project coordinator (approx. 1–1.5 h)	See interview guide supplement C
Interviews with scientist(s) from a different discipline and at least one non-scientific actor (approx. 30–45 min/interview)	See interview guide supplement C
Analysis of the interviews (approx. 2 days)	-Using the same codes as in the preparatory step ‘Analysis of project documents‘ -Comparison of stated outputs and intended effects -Comparison of assumptions

The literature emphasizes the importance of also acknowledging the limitations of methodological formats in terms of generating realistic expectations. For instance, Mitchell et al. [26] suggest aiming for ‘good enough’ rather than striving for an all-encompassing set of impact pathways. Barnett and Gregorowski [41] recommend communicating that ToC workshops are not aiming for final results. They emphasize that “it will be imperfect” (ibid., p. 8.); the essence of ToC thinking lies in the joint discussions and in reaching a common understanding that can be iteratively revised. Additionally, confidentiality is emphasized as a prerequisite for creating an open, trusting atmosphere [42]. Facilitators introduce these principles in a short presentation at the beginning of the workshop.

I. At the end of the path

Step No. 1: Agreement on a joint vision

By beginning the ‘building exercise’ at the end with a joint formulation of a vision, the project gains a common understanding of the long-term objective it aims to contribute to. This joint vision will be the starting point for the joint ToC-thinking. Firstly, facilitators ask every participant to note down their vision individually:

• •From your perspective, what is the main goal of the project?
• •What overarching change do you intend to contribute to?

The exemplary statement: ‘achieve the goal XYZ to contribute to the overarching change ABC’ is suggested. Secondly, they ask participants to share and discuss the different statements with a focus on common denominators, the aim being to resolve contradictions and formulate a preliminary joint project vision that can be used for the duration of the workshop and further elaborated in the next workshop(s). Fig. 1a provides a worksheet and 1b an exemplary result of the implementation of step No. 1 (s. project description in the supplement D).

Fig. 1a.

Worksheet Step No. 1:Agreement on a joint vision.

Fig. 1b.

Project example for Step No. 1.

II. Collecting building materials

The ‘building materials’ for the impact pathways encompass a brief actor analysis and an activities-interactions-outputs table. The workshop format is designed for projects that have already been set up based on a project proposal which outlines the project design. However, since often not all project partners were involved in conceptualizing the project design and project teams are often confronted with a longer time span between formulation and acceptance of proposals, reflection on the project activities, interactions, outputs and relevant actors provides a common basis for the discussion of intended impacts.

Step No. 2: Identifying relevant actors and their roles in relation to the project goals

As a first step, the TDR team conducts a brief actor analysis focusing on the impact-relevant roles of problem-related actors, i.e. whether they are expected to act as movers, floaters or blockers with respect to the project goals (this step is based on [43]).

The participants are asked to reflect on the role of the actors:

• •Movers: Which actors (can) actively contribute to the desired change?
• •Floaters: Which actors neither actively support the desired change nor block it (but may change their position and be convinced)?
• •Blockers: Which actors are against the desired change (due to their own interests being negatively affected) and (might) block the process?

This concept is introduced by means of a matrix differentiating movers, floaters and blockers on the x-axis and scientific and non-scientific actors on the y-axis (see Fig. 2a, Fig. 2b). Fig. 2a provides a worksheet and 2b and exemplary result of the implementation of step No. 2.

Fig. 2a.

Worksheet Step No. 2:Identification of relevant actors and their roles in relation to the project goals.

Fig. 2b.

Project example for Step No. 2.

To illustrate the exercise, facilitators can suggest relevant actors and their position in the matrix as a starting point for discussion, based e.g. on information from the project documents. Participants then start naming and positioning further actors and actor groups individually. In a next step, the group jointly allocates further actors they consider relevant and discusses their position in the matrix.

Step No. 3: Reflection on planned activities, interactions and outputs

This step encompasses reflection on planned activities, (direct) interactions and outputs. First, the facilitator introduces a preliminary table filled with information derived from the analysis of the project proposal and interviews. Confronted with this overview, participants are asked to adapt the table according to their understanding, make connections between the different elements in a discursive manner, while also paying attention to both formal and informal direct interactions [44]. Fig. 3a provides a worksheet and 3b an exemplary result of the implementation of step No. 3.

Fig. 3a.

Worksheet Step No. 3:Reflection on planned activities, interactions and outputs.

Fig. 3b.

Project example for Step No. 3.

Guiding questions for reflection on how the collected elements contribute to effectiveness are:

• •What are the key TDR activities in the project?
• •What activity-related interactions take place? (formats, frequencies, participants)
• •What are the key (tangible) outputs of the project activities?

The facilitator may expose possible contradictions or gaps by asking: How will your activities and interactions lead to the planned outputs? At a later stage in the running of the workshop, elements can be marked according to their state of implementation (e.g. not met, initiated, on time, completed and expanded; based on [40]).

III. Building impact pathways

The phase of impact reflection is the core of the workshop and lays the foundation for impact-oriented project planning, because it is in this phase that short-, medium- and possible long-term effects are discussed and defined in relation to the previously elaborated elements. The impact heuristic by Schäfer et al. [11] serves as the underlying analytical framework and is visualized in three concentric spheres (similar to [16]).

This part of the workshop format consists of five steps: identifying intended effects using the impact heuristic, identifying possible unintended and negative effects, connecting the elements to plausible impact pathways, assessing the project's influence on the generation of effects, and agreeing on the effects the project team is prioritizing. On the worksheet, three nested spheres representing the orders are inserted between the activities-interactions-outputs table and the joint vision, indicating relevant connections. The actors’ matrix should remain visible to the participants so it can be included in the deliberations. Fig. 4a provides a worksheet and 4b an exemplary result of the implementation of steps No. 4–8.

Fig. 4a.

Worksheet for the Steps No. 4–8. in III:Building impact pathways.

Fig. 4b.

Project example for Steps No. 4–8.

Step No. 4: Identifying intended effects using the impact heuristic

First, the facilitator introduces the analytical framework and suggests initial intended effects derived from the document analysis and/or interviews. Subsequently, and guided by the following questions, the participants suggest, discuss and consolidate the effects they are targeting in the project (based on [16]):

• •What effects do you intend to achieve within the duration and/or spatial scope of the project?
• •What effects do you reasonably expect to see within the close temporal and/or spatial context of the project (approx. 1–3 years)?
• •What contributions to greater societal changes do you hope to see beyond the temporal and/or spatial context of the project?

The participants formulate effects and place them on the worksheet in the respective order. Although the impact heuristic by Schäfer et al. [11] is not explicitly introduced and used, it can serve as a background framework for the facilitator to ask the group about possible forms of effects that have not been mentioned.

Examples for questions regarding effects in the 1st order:

• •What new relations do you expect to be developed through the project activities? (network formation)
• •What learning effects and capacity building do you expect in the different actor groups?
• •Who does what differently now due to the project activities and outputs? (changes in individual and organizational practices)

Examples for questions regarding effects in the 2nd order:

• •Which part of the project do you expect to be sustained (continuation)?
• •Which actor groups in other contexts do you expect to pick up project results due to transfer and communication?

Examples for questions regarding effects in the 3rd order:

• •In what ways do you expect the project will contribute to changes in public discourses, laws and regulations?
• •What kind of contribution are you aiming to make with this project e.g. effecting changes in infrastructure, business models?
• •In what ways are you aiming to contribute to higher-level transformation (e.g. reducing climate-relevant emissions, fostering sustainable consumption, preventing the loss of biodiversity, etc.)?

The facilitators encourage the participants to agree on the positioning of the effects. Tentative connections between activities, interactions, outputs and effects can already be drawn to indicate preliminary impact pathways.

Step No. 5: Identifying and reflecting on possible unintended, negative effects

This step sheds light on possible unintended, negative effects guided by the following questions (similar to [45]; Schäfer et al., 2021):

• •What (unintended) negative/undesired effects may result from the project?
• •What effects could be counterproductive in terms of contributing to solving the problem?
• •Are there any negative second-order effects that can be expected from choosing certain solutions to the respective problem?

Participants suggest and discuss possible negative effects, write them on different-coloured cards and place these cards on the board next to the project activity or intended effect they are associated with. Taking over other perspectives, for example those of ‘blockers’, may be helpful for identifying possible negative effects. Also, taking the role of an advocatus diaboli, the facilitators can put suggestions up for discussion.

Step No. 6: Connecting the elements to form plausible impact pathways

In this step, the focus is on drawing connections between the separate effects in order to visualize possible impact pathways and make interrelations visible. The guiding questions are:

• •What are the possible links between the effects and the project activities, interactions and outputs?
• •What connections between different effects can be drawn that result in plausible impact pathways?
• •Do some effects depend on the occurrence of other effects?

The facilitator encourages the discussion by suggesting possible connections, drawing the lines and arrows to indicate the direction of the jointly defined relation. In this phase, cards denoting effects can be moved around to build plausible impact pathways. Change processes need not be linear [23], thus a mere concentration on linear connections should be avoided. In some cases, the occurrence of effects depends on the occurrence of later effects. The participants should also be called upon to identify possible feedback loops [20].

Step No. 7: Estimating the project's influence on effects

The next step in this phase estimates the relative influence of the project on achieving the intended (and unintended) effects as well as on avoiding negative effects. The rationality behind this step is the recognition that the relative influence of any project usually declines with increasing distance from the project boundary (based on [16,46]). While the influence on the occurrence of first-order effects (e.g. project partners learning from one another, or network formation between different actor groups) is mostly quite high, second-order and especially third-order effects usually depend on the activities of other actors and favourable context conditions. The projects must therefore estimate their influence on each effect individually. Here, the facilitator asks the following guiding questions (based on the concept of different spheres of control by [16]):

• •Which effects does the project have a high level of control over in terms of their occurrence? (Sphere of control)
• •Which effects can the project exert influence on, even though their occurrence cannot be fully controlled, since other actors or framework conditions play an important role? (Sphere of influence)
• •Which greater societal changes does the project aim to contribute to, even though their occurrence is outside the sphere of the project's direct influence? (Sphere of interest)

The participants are asked to mark each effect with a green (control), yellow (influence) or red (interest) frame in order to make the expected project influence visible.

Step No. 8: Prioritizing impact intentions

The comprehensive reflection on the impact pathways ends with a discussion on the priorities of the project partners with regard to generating societal impacts. First, each partner is asked to indicate a maximum of three prioritized effects in the figure (positioning a little red dot). Based on these indications, the facilitators encourage the project partners to discuss which effects, in addition to the tangible outputs, they consider to be central to the project's success. The guiding question is:

• •Which would be the most important effects to achieve (without which you would consider the project as having failed)?

The facilitator marks the agreed-on most desired effects with a red circle. This round of sharing ‘most significant changes’ [47] can either strengthen the joint understanding of the project when there is a high level of consistency among the participants or emphasize the need for balancing priorities when different effects are selected. If an evaluation concept is based on the results of the workshop, this prioritization also helps to limit the number of indicators.

IV. Testing the pathways for their stability

Step No. 9: Disclosing underlying assumptions

Step No.9 focuses on how the intended change is expected to occur [27]. The discussion on the participants’ different assumptions also sheds light on external factors and makes potential internal strategies visible ([28]; based on [22]). For this phase, a new worksheet is used which allows the relevant impact pathways to be transferred from the previous sheet and leaves room beneath each pathway to add the corresponding assumptions. Fig. 5a provides a worksheet and 5b an exemplary result of the implementation of step No. 9.

Fig. 5a.

Worksheet Step No. 9.Disclosing underlying assumptions.

Fig. 5b.

Project example for Step No. 9.

The facilitators introduce the phase and ask the following questions:

• •What assumptions are ‘hidden’ behind the suggested connections between individual effects?
• •Which other actors may need to be involved to get from effect A to B?
• •What conditions must exist for the subsequent effect to be achieved?
• •Which existing conditions can be influenced by the project?
• •What supportive conditions can be created?
• •What gaps and risks does the disclosure of the assumptions reveal?

Assumptions can be formulated regarding internal project strategies and external context factors [33,48]. The assumptions suggested by the participants are written on cards that are linked to the connecting lines between an effect and the subsequent effect on the impact pathway, symbolizing that the respective assumption must be fulfilled for the subsequent effect to be achieved. Since this discussion may reveal implausible formulations of effects or connections, adjustments can be made to all previously developed elements. At this point in the reflection, it is useful to look at the actor analysis (step No. 2) and to discuss which actor or actor group could be influenced to contribute to and support higher-order effects of the project. These considerations may lead to planning new TDR activities, such as addressing stakeholders who act as intermediaries, or contacting decision makers regarding inhibiting context conditions. The facilitators document the discussion on the necessary strategic adjustments to the project design so that these can be picked up by the project team later.

Step No. 10: Forming a plausible impact story

To conclude the workshop, the facilitators verbally present a short summary narrative based on the impact pathways and assumptions, delineating the project's vision and the anticipated ‘pathways’ leading to its realization. Then the participants answer the following questions:

• •Does the overall impact pathway picture represent the project goals from your point of view? If not, which aspects would you like to change?

The facilitators document the notes on the impact story, which are later included in the written formulation.

Feedback round

In our study, we closed our workshop with a feedback round and a discussion about the next steps. We asked the participants the following questions:

• •What will you take with you from this workshop?
• •What next steps are necessary from your point of view?
• •What improvements do you suggest for our workshop format?

In addition, the participants also had the opportunity to give anonymous feedback on the quality of the workshop and what they had learnt (see feedback questions in Supplementary Material B). We recommend integrating a feedback round with questions tailored to the needs of the project.

Follow-up

As a follow-up, the facilitators document the different worksheets in detail and assess them for readability and clarity. Then, the project team receives the documentation for future use in their impact reflection, monitoring and evaluation.

Using the visualized results of the workshop, the facilitators can create different forms of presentation of the ‘impact story’ that has been developed. They may formulate a concise narrative in the form of a short document, which, depending on its further use, could be a few sentences to two pages long. They might also translate the ‘impact story’ into a detailed diagram model or a simplified overview graphic of the impact pathways created [33]. The impact story is subsequently reviewed by the project partners in a co-writing process after the workshop. The following key questions ([[28], [49]]; based on [9]) can help to support reflection on the impact narrative:

• •Is this the story you wish to tell in relation to the project?
• •How credible and plausible is the story from your point of view?
• •What are the weaknesses or gaps in the story?
• •What perspectives may be missing?

If the impact narrative is agreed on by the project partners, it can be used to communicate the project's goals to the public or further actors, e.g. by publishing it on the project website.

Iteration workshops

Depending on the duration of the transdisciplinary project in focus, we recommend repeating the workshop with the project team every twelve to fifteen months. In our experience, significant project developments and changes in the project environment become apparent within this time frame, which can be discussed as part of a renewed impact reflection. The repeated workshops help the project team assess whether they are on the right path towards reaching the planned outputs and achieving the intended impacts, or whether it is necessary to adapt the project design.

It is important to repeat all the steps and reflect on the results of the previous workshops. The central question regarding the results of the respective steps is whether and how the documented elements have changed and whether new insights need to be included. If further or new project partners from science and society become relevant to the project. By recalling the assumptions behind the intended effects, any shortcomings in the approach to achieving specific effects are likely to become more apparent to the participants.

In our experience, TDR projects usually reassess the achievability of some of the effects in the iterated workshops. In some cases, the priorities regarding project success shift, while in other cases unexpected obstacles push the expected occurrence of effects further away in time. The discussion may lead to the formulation of additional necessary impact-related activities, such as improving external communication or strengthening collaboration with relevant stakeholders.

The facilitators can support the project team in recognizing shortcomings by e.g. asking questions such as:

• -You expect the project results to be taken up by actors from other contexts and that this might lead to further societal effects. – Are the planned communication activities sufficient to reach different target groups?
• -You expect that approved implementations continue after the scientific partners withdraw. – Can you assure that other actors will take over the responsibility for carrying on with the project activities?

The requirements for additional activities are noted down on a worksheet and it is discussed whether they can be realized with the given project resources and who will take over responsibility for them. It may also be that planned activities no longer seem necessary and are cancelled. In the follow-up workshop, this table is revisited, asking which additional tasks could be, or will be, completed – or what the reasons are for not taking up the planned activities. Table 4 shows examples for additional impact-oriented activities based on workshops with several projects.

Table 4.

Examples for impact-oriented activities initiated based on the reflections in the workshops.

Continuation and upscaling of project activities in the original project context	Transfer to other contexts
Communicate the added value of new products or (public) services in order to foster a willingness to make behavioural changes.	Frame as a lighthouse project and make visible why the project is considered best practice for a common problem.
Address actors who could potentially take over responsibility for the continuation of project activities and upscaling.	Identify multipliers and communicate via their communication channels (e.g. newsletters).
Offer further training activities to strengthen the skills required to consolidate the project results.	Disseminate project results via presentations at practitioners’ events and publication in practice-oriented media.
Address politicians with regard to hindering regulation and framework conditions which prevent the upscaling of solutions developed by the project in the local/regional context.	Address politicians with regard to hindering regulation and framework conditions which prevent the upscaling of solutions developed by the project in the national context.

Results of the evaluation of the workshop format

In the first workshop round, we received responses to the anonymous feedback survey from 31 out of 40 participants from seven projects (77.5 %), and from 18 out of 23 participants in the second workshop round with four ongoing projects (78.3 %). We compared this data with the verbal feedback and our notes.

Generally, the workshop concept, methods, preparation and facilitation were evaluated positively by the majority of respondents (95 %) in the anonymous questionnaire. Most of the participants (74 % in the first and 89 % in the second workshop round) stated that the workshop helped to better visualize and understand their project's contribution to the overarching problem in focus. Regarding knowledge integration, three-quarters of the participants stated that the workshop had made different perspectives on specific points in the project team more transparent. 47 % of the participants in the first workshop round, and 67 % in the second workshop round (strongly) agreed that they now appreciated the relevance of systematic consideration of the intended and achieved effects of projects and were able to (further) differentiate and specify their intended (or, in two cases, achieved) short-, medium- and long-term effects.

Regarding the workshop quality, the preparation and facilitation from a project-external perspective were highlighted as beneficial for reflection by at least one person in every verbal feedback round. Participants praised, in particular, our detailed preparation based on project documents and interviews. Some project teams highlighted our useful compilation of project activities, interactions and outputs, which we provided in a compact overview based on document analysis (mainly project proposals). The comparison between the project plan and the project status was highly valued in every project phase. The trusting and pleasant workshop atmosphere was mentioned by almost all participants (81 % in the first workshop round, and 100 % in the second workshop round).

According to this data, the participants increasingly acknowledged the usefulness of repeating the workshop format and reflecting further on impact. Asked about future applicability, about two-thirds of all participants in the first workshop round and three-quarters in the second felt the methods employed and the results yielded would be of practical use in future projects. The verbal feedback gained in the workshops suggests that, on the whole, the scientific project partners, and especially the project coordinators, saw potential for future applications. Two project managers, one from a civil society organization and another from an intermediary organization, also openly expressed their intention to implement some of the methodical elements for their purposes in future.

The evaluation shows that a methodically guided approach to reflection on impact is appreciated by transdisciplinary project teams and fosters systematic engagement with the relevant goal of achieving societal effects in TDR. To do this, however, concrete occasions must be created and maintained [5]. Hence, resources for the preparation, communication and execution of this workshop format are necessary. External facilitators can help to take on these tasks and provide a different perspective on the topic.

Level is identical to the level of Methon backround

The participants’ feedback showed that the workshop format offers a valuable additional space for reflection and discussion across the whole team. Bringing in external experts to facilitate the preparation, implementation and follow-up documentation had several advantages for the TDR projects.

• •First, time resources were handled very carefully: besides taking part in the workshop, only a few members of the TDR team had to provide a short time slot for a preparatory interview. This is especially important for the participation of non-scientific actors, who often only have scarce – or no – time resources that are financed by the project. The external team also kept the topic of impact reflection on the agenda by regularly addressing the project coordinators and supporting them by guaranteeing the resources and skills to plan, design and conduct the workshop and process the results further.
• •Second, as external facilitators, we provided a different approach in that we structured the project materials as well as the project partners’ views and contributions. From this external perspective, we asked questions that prompted discussion amongst the project partners. Reflecting on perceived gaps between project activities and outputs and intended effects or risks connected with the successful continuation of activities within the project context, the workshop format contributed to knowledge integration for a common understanding of the project goals and intended impacts.
• •Third, since the workshops were externally facilitated, all project partners could take part as equals. It seems especially important that the project coordinators do not take on the role of facilitators but are able to contribute their perspective in the same way as all the other TDR members.

We therefore strongly recommend including resources in the budget for external facilitation or for a dedicated sub-project within the project consortium which is responsible for preparing and implementing this type of workshop with methodological expertise.

The increasingly positive evaluation results after the second workshop round show that the participants’ understanding of the added value grows through the iteration of the workshop. The joint reflection on the state of completion of planned activities and outputs supports the project's coordinators in project controlling. The regular revision of achieved and intended impacts and assumptions about how these can be fostered allows teams to adjust the project design in response to internal and external changes. This is a strong argument for running the workshop several times and ensuring the continuous participation of the whole TDR team.

It has proven to be essential to motivate and enable non-scientific project partners to participate in the impact reflection workshop, since a balanced mixture of participants is crucial in gaining new insights. Besides careful management of the individual project partners’ time resources, it is necessary to reflect on whether the entire project or only sub-projects can be dealt with in a workshop. If a larger consortium encompasses several sub-projects that aim for quite different societal effects, it might be more efficient and motivating to initially run separate workshops. The results of these shorter sub-workshops could then be brought together in a joint workshop with representatives from each sub-project. It remains a challenge to strike a balance between project resources and the realization of the workshops in a way that benefits everyone.

In the workshop we used the impact heuristic by Schäfer et al. [11] as an underlying analytical framework. The forms of effects described in the heuristic were useful for developing questions about potential effects during the workshop. Furthermore, it allowed participants to analyse and systematize the collated effects after the workshops. We applied a clear and simple terminology, e.g. by employing the metaphor of path-building and different orders of effects, that could be understood by all participants regardless of their background. The definitions used seemed appropriate for the workshop format, even if we sometimes had to repeatedly explain certain elements.

Comparing the results of several workshops, we also noted that some of the effects, such as learning and network building, emerge at different impact orders but vary in their occurrence for different actor groups. Further differentiation might be useful in the future. We also learnt that the question of whether e.g. network formation is understood as an output, a first- or even a second-order effect is a project-specific decision that may need to be discussed repeatedly over the course of the project. Fritz et al. [20] also point out that “the same type of effect can occur at different positions in pathways of interlinked societal effects” (p. 72). We therefore recommend employing the categorization of societal effects provided by Schäfer et al. [11] in a flexible way [50].

Regarding reflection on possible negative unintended effects, the workshop format mainly aims to sensitize the TDR team to the fact that, for example, positive effects for some actor groups may be accompanied by negative effects for others. In the workshops, the participants mostly referred to the negative side of intended positive effects as risks (e.g. “insufficient demand by consumers” as opposed to the intended positive effect that the number of consumers interested in a new product or service increases with the communication of additional social and ecological benefits). In the subsequent step of reflecting on assumptions about the occurrence of intended effects, the discussion may turn to whether these risks can be met with certain project activities or whether they are beyond the project's influence. Within the suggested, rather compact, workshop format it is not possible to deal with this issue in more depth.

During the processes of reflection on impact, our study identified an increasing need for ‘meta-reflection’ on several procedural aspects. For example, visualizing pathways bears the risk of falling (back) into linear thinking about the relations between effects of different orders. Similarly, causally linking effects to specific activities and/or outputs is tempting, but misleading in a complex project setting. This is especially true for effects that are expected to occur beyond the project as the consequence of a complex chain of activities by different actors. Introducing feedback loops and asking for the assumptions behind the visualized impact pathways helped the participants consider complexity adequately and think about impact pathways in terms of plausible rather than causal links between the elements.

The workshop format presented here is limited to fostering reflection on impact as an anticipatory practice in order to expatiate and justify strategic decisions within a TDR project. The format can be further expanded to monitor societal effects by developing indicators and collecting respective data. The results of the workshop provide a good basis for formulating process and output qualities that increase impact potential. The project team – or external evaluators – can build upon the collected intended effects and assumptions about their occurrence and formulate indicators that capture both the first-order effects which occur during the project duration and the potential for further second- or even third-order effects [50]. Ideally, there should also be an ex-post reflection – or assessment – on impact not only at the official end of the project but one or two years after finalization.

After testing the workshop format, we see potential for further conceptual development. Regarding the analytical potential of the workshop results, the assumptions and impact stories developed in the workshops could be underpinned with theories of action or social change in order to enhance the explanatory power and understanding of the change process in focus. This theoretical extension would complement the use of the impact heuristic. As also pointed out by Álvarez et al. [51], integrating methodological elements for reflecting on and dealing with power relations both within the transdisciplinary workshop setting and with regard to the emergence of societal effects could also be given more explicit consideration. However, further theoretical and methodological enhancement may lead to greater complexity of the workshop format, exceeding the resources and capacities of individual TDR projects.

Ideally, the workshop is facilitated by two moderators: one leads the discussions during the several methodological steps, while the other documents the participants’ suggestions and observes the discussion process. The workshop format employs visualizations as boundary objects to be addressed and worked on by all participants. This visual approach is adaptable to both online groups and on-site gatherings. Workboards can be prepared and jointly edited on a digital visual collaboration platform (e.g. Miro or Mural) or on large pinboards or tables covered with brown paper, using different-coloured sticky notes and markers. Table 2 provides an overview of the general data and the preparatory steps of the workshop.

Table 2.

Preparation of the workshop.

General data: •Duration: 4 h (incl. breaks) •Location: online, or on site •Number of participants: 4–12 key project partners, balance of scientific and non-scientific participants •Facilitation: at best 2 moderators •Material: online: online whiteboard; on site: pinboards or tables covered with brown paper, different-coloured sticky notes and markers

Preparation (approx. 2 days) 1.Prepare templates of visualizations as worksheets on brown paper/workboards for the subsequent methodical steps and enter initial data based on the document analysis and interviews 2.Prepare outline for the workshop design (timetable and responsibilities of the 2 facilitators) 3.Prepare short presentation for the introduction (aims, workshop design, benefits and principles) 4.Collect interesting aspects and questions that can be integrated at different steps of the workshop based on the analysis of the documents and the interviews

Our half-day workshop format is designed to be integrated into the project design of TDR projects. It combines a set of methods and visual elements. We recommend using the metaphor ‘building pathways’ to communicate the workshop concept and structure. The workshop format has four main structural elements:

• •I. At the end of the path: envisioning the desired future to be built
• •II. Collecting building materials: identifying relevant actors, planned activities, interactions and outputs of the TDR project
• •III. Building the impact pathways: identifying intended effects and their plausible interrelations
• •IV. Testing the pathways for their stability: disclosing assumptions, gaps and risks, thereby reviewing the model that has been developed

The overall results of the workshop enable the project partners to gather ideas for adjusting the project design – if necessary – in order to create further impact potential.

While ToC builds the conceptual framework of the impact workshop format, ten methodical steps structure the impact reflection. To foster a productive atmosphere, introductions to each step and breaks are included. At the end of the workshop, participants share their experiences and lessons learnt. Table 3 provides a step-by-step overview of the workshop structure. More detailed guidance is to be found in Supplement A.

Table 3.

Step-by-step overview of the workshop structure.

Min1 (′)	Main structural elements	Methodical steps	Guiding Questions
10′	Input		Aims and principals of the workshop
20′	I. At the end of the path	Step No. 1.:Agreement on a joint vision (10′)	•From your perspective, what is the main goal of the project? •What overarching change do you intend to contribute to? Exemplary Form: ‘Achieve the goal XYZ to contribute to the overarching change ABC’
40′	II. Collecting building materials	Step No. 2.:Identifying relevant actors and their roles in relation to the project goals (20′)	•Movers: Which actors (can) actively contribute to the desired change? •Floaters: Which actors neither actively support the desired change nor block it (but can change their position and be convinced)? •Blockers: Which actors are against the desired change (due to their own interests being negatively affected) and (might) block the process?
		Step No. 3.:Reflection of activities, interactions, and outputs (20′)	•What are the key activities in the project? •What interactions take place regarding the activities? (formats, frequencies, participants) •What are the key (tangible) outputs of the project resp. its activities? Additional question: •How will your activities and interactions lead to the planned outputs?
10′	Short break
80′	III. Building impact pathways	Step No. 4.:Identifying intended effects using the impact heuristic (30′)	• Which effects do you intend to achieve within the duration and/or spatial scope of the project? ○ What new relations do you expect to be developed through the project activities? (network formation) ○ What learning effects and capacity building do you expect in the different actor groups? ○ Who does what differently now due to the project activities and outputs? (changes in individual and organizational practices) • What effects do you reasonably expect to see within the close temporal and/or spatial context of the project (approx. 1–3 years)? ○ Which part of the project do you expect to be sustained (continuation)? ○ Which actor groups in other contexts do you expect to pick up project results due to transfer and communication? • What contributions to greater societal changes do you hope to see beyond the temporal and/or spatial context of the project? ○ In what ways do you expect the project will contribute to changes in public discourses, laws and regulations? ○ What kind of contribution are you aiming to make with this project e.g. effecting changes in infrastructure, business models? ○ In what ways are you aiming to contribute to higher-level transformation (e.g. reducing climate-relevant emissions, fostering sustainable consumption, preventing the loss of biodiversity, etc.)?
		Step No. 5.:Identifying and reflecting on possible unintended, negative effects (15′)	•What (unintended) negative/undesired effects may result from the project? •What effects could be counterproductive in terms of contributing to solving the problem? •Are there any negative second-order effects that can be expected from choosing certain solutions to the respective problem? Additional questions Facilitators take the role of an advocatus diabolus and make suggestions on possible negative effects (prepared based on document analysis). Taking over other perspectives, for example those of ‘blockers’, may be helpful for identifying possible negative effects.
		Step No. 6.:Connecting the elements to form plausible impact pathways (15′)	•What are the possible links between the effects and the project activities, interactions and outputs? •What connections between different effects can be drawn that result in plausible impact pathways? •Do some effects depend on the occurrence of other effects? Additional recommendation: Change processes need not be linear. In some cases, the occurrence of effects depends on the occurrence of later effects. The participants should also be called upon to identify possible feedback loops.
		Step No. 7.:Estimating the project's influence on effects (10′)	•Which effects does the project have a high level of control over in terms of their occurrence? (Sphere of control)? •Which effects can the project exert influence on, even though their occurrence cannot be fully controlled, since other actors or framework conditions play an important role? (Sphere of influence)? •Which greater societal changes does the project aim to contribute to, even though their occurrence is outside the sphere of the project's direct influence? (Sphere of interest)
		Step No. 8.:Prioritizing impact intentions (10′)	•Which would be the most important effects to achieve (without which you would consider the project as having failed)?
30′	Longer break (in which facilitators may adapt the further course of the workshop)
40′	IV. Testing the paths for their stability	Step No. 9.:Disclosing underlying assumptions (30′)	•What assumptions are ‘hidden’ behind the suggested connections between individual effects? •Which other actors may need to be involved to get from effect A to B? •Which conditions must be given for the subsequent effect to be achieved? •Which given conditions can be influenced by the project? •Which supportive conditions can be created? •What gaps and risks does their disclosure reveal? Additional question: Which strategical adjustments can be made to all previously developed elements to avoid or mitigate these risks?
		Step No. 10:Forming a plausible impact story (10′)	•Does the overall impact pathway picture represent the project goals from your point of view? If not, which aspects would you like to change?
10′	Feedback round	Participants are asked to give verbal feedback to the guiding questions and to answer the anonymous survey (s. Supplementary Material B)	•What will you take with you from this workshop? •What next steps are necessary from your point of view? •What improvements do you suggest for our workshop format?

¹Time specifications should be handled flexibly. For larger groups, more time should possibly be planned.

CRediT authorship contribution statement

Martina Schäfer: Funding acquisition, Methodology, Validation, Writing – review & editing. Emilia Nagy: Validation, Visualization, Writing – review & editing. Josefa Kny: Project administration, Visualization, Writing – original draft.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix. Supplementary materials

Supplementary Material A: A Step-by-Step Guide with Conceptual and Methodological References

Supplementary Material B: Feedback Form Impact Reflection Workshop

Supplementary Material C: Guidelines for interviews with project partners from science and practice

Supplementary Material D: Short description of the project example

Data availability

• The data that has been used is confidential.