Reflections on COVID-19 and internal evaluation in a humanitarian non-profit

Abstract

This article uses reflective practice and social interdependence theory to unpack the ways in which our practice as internal evaluators was impacted by the COVID-19 pandemic, using the development of a monitoring and evaluation framework as a case study. The reflections are separated into six stages: description, feelings, evaluation, analysis, conclusion and action plan. Initial reflections on the impact of COVID-19 drew out the negative effects of the pandemic and associated restrictions, which limited our ability to build rapport and stretched our capacity to balance work and home commitments. Deeper investigation revealed that the disaster pushed us to develop new ways of working that will augment and improve our future efforts. We foresee that these learnings will enable a future with greater ability to offer hybrid online/face-to-face collaborative opportunities that will enhance inclusion and active participation, thereby promoting monitoring and evaluation with greater collective ownership and relevance to a wider audience.

Introduction

In the opening months of 2020, the domestic emergency services team at Australian Red Cross embarked on an overhaul of our largely ad hoc and fragmented internal monitoring and evaluation (M&E) systems. The virus causing COVID-19 entered Australia in late January and, by mid-March 2020, Victoria’s borders were shut and personnel from the Melbourne head office were encouraged to work from home. Meanwhile, staff across the country were otherwise affected by restrictions, including to interstate travel, social gathering limits, physical distancing and the closure of non-essential services. As we are based in Naarm/Melbourne on Bunurong and Wurundjeri Country, which has experienced the world’s longest lockdowns in response to COVID-19, we critically reflect on how the pandemic restrictions affected our ability as internal evaluators to collaboratively and inclusively develop an M&E framework. This article is written by one of the two national M&E leads for the emergency services team; however, as we job share one full-time load, I refer to we, meaning the two of us, throughout the article and acknowledge my colleague’s input in the work, reflections on the work and checking of this article.

This article is written as a personal narrative using Gibbs’ (1988) reflective cycle and the theory of social interdependence to unpack our experience. As such, it is deeply subjective and aims to surface our learning and innovation to support critically reflective practice as a key aspect of good evaluation (Patton, 2011, 2015). Additionally, these learnings have particular relevance for those in internal evaluation roles whose work is embedded and continuous within an organisation. Rather than providing guidance on the development of an M&E framework per se, which is covered in the scholarly and grey literature (e.g., Markiewicz & Patrick, 2016), this article uses the framework development as a case study to present our experience of remote and hybrid collaborative M&E. The mechanisms employed to enhance meaningfulness and utilisation of internal M&E under these challenging conditions may assist others dealing with similar circumstances.

This article builds on a significant base of literature regarding the effectiveness of virtual teams. The literature has a broad reach across organisational psychology, social psychology, management studies, human resources, behavioural studies and sociology. Research conducted years prior to COVID-19 highlights useful insights and frameworks for understanding effective teamwork. These provide guidance for establishing virtual teams and note the importance of careful consideration of team composition, transparent communication, clarified group goals and individual roles, clear boundaries, rewards for effort, respectful group culture, flexible management and adequate resourcing (Axtell et al., 2004; Hertel et al., 2004, 2005; Lurey & Raisinghani, 2001; Maznevski & Chudoba, 2000). While this body of literature provided a valuable foundation for understanding factors of effective virtual teamwork, the rapid, unanticipated movement of our whole team into the remote space signified an unprecedented and disruptive situation that is more accurately captured by the literature on virtual teams that has been published specifically about the pandemic. This literature examines how teams rapidly pivoted to working remotely without the luxury of well-planned change management processes and how this movement impacted wellbeing and group functioning (Al-Habaibeh et al., 2021; Şentürk et al., 2021; Wang et al., 2021; Whillans et al., 2021; Yang et al., 2022).

Background context

The emergency services team is comprised of around 230 staff who focus on the psychosocial aspects of community resilience and recovery from Australian disasters, including pandemic. While the head office is in Naarm/Melbourne, there are key centres in state and territory capitals and other staff located in disaster-affected and high disaster risk regions nationwide. Pre-pandemic, staff would regularly travel between offices while other interstate communications were conducted over telephone, email or Skype.

Before 2020, state and territory teams conducted M&E in their own ways, with little oversight or coordination nationally. This resulted in pockets of good practice but an overall lack of consistency and an inability to draw comparable data nationwide. While an M&E framework had been developed years earlier, it had never been implemented.

In early 2020, during the Black Summer bushfires that devastated much of the country, the national emergency services team committed to build a systematic and streamlined approach to M&E across Australia, starting with development of the framework. The timing was difficult as the bushfires meant that emergency services personnel were already stretched; however, this work was prioritised. We started the M&E refresh by reviewing the extant documentation and processes in place across the team. As the bushfire response eased throughout February, we planned overarching goals for implementation of the new approach. These goals included the development of a new M&E framework, a template for programme and project level M&E plans, a centralised data collection strategy that linked to the M&E framework, guidance around when and how to analyse and evaluate the data, and a plan for utilisation of findings. This article solely focuses on the M&E framework, the elements of which are depicted in Table 1. Shortly after reviewing the situation and planning our approach, COVID-19 cases started to emerge in Melbourne. Victorian staff started working from home and long running pandemic restrictions began.

Table 1.

The elements of the M&E framework.

Principles	A set of guiding principles that underpin our M&E work
Theoretical framework	A theory of linked concepts that offer a framework for analysing and interpreting M&E data
Audience	Explanation of who the M&E framework document is for
Assumptions	A list of untested ideas that influence our belief in ‘what works’
Theory of change	Depiction of how we intend our programmes to contribute to short-, medium- and long-term change
Guidance on logframes	Explanation of how to use the theory of change and indicator bank to develop project level logframes
Indicator bank	A set of 1–5 indicators for each outcome identified in the theory of change. The means of verification is listed for each indicator, along with standardised survey questions where appropriate
Key evaluation questions	A set of five overarching key evaluation questions that can be used consistently to guide our evaluations
Data utilisation	Ideas for knowledge translation and utilisation of M&E findings, far beyond upward accountability
Glossary	Explanation of key terms to encourage consistent usage

Method

To critically reflect on how pandemic restrictions affected our ability to develop the M&E framework, we utilise Gibb’s (1988) reflective cycle to guide our thinking. Reflective practice is commonly used for developing reflexive thought, critical thinking and solution finding, particularly in human-centred disciplines such as healthcare, education and social work (Bassot, 2015). There are many reflective practice frameworks in existence, each offering a unique perspective and questions to help practitioners learn and innovate through discussion (Brookfield, 1995; Fook & Gardner, 2007; Johns, 2009; Kolb, 1984). Gibb’s cycle was chosen as it offers broad categories of analysis that are relevant to a wide range of interventions. Thus, it was rationalised that Gibb’s cycle would have utility for readers of this article. While reflective practice can enhance self-awareness using theory as a lens to think through positionality and actions, here we use it as a form of informal everyday evaluation (Kelly & Rogers, 2022; Wadsworth, 2011) to guide our account of what happened, why and how it could be improved.

Gibb’s (1988) cycle is set out around six stages: description, feelings, evaluation, analysis, conclusion and action plan, as shown in Figure 1. In the description phase, we ask ourselves what happened, when and where, who was there, who did what and what we had hoped would happen. The second stage seeks to surface feelings about the situation. We then make value judgements to evaluate the situation, followed by analysis of those considerations. The analysis utilises the theory of social interdependence, as defined by scholars including Tindale and Anderson (2002), Johnson (2003) and Johnson et al. (2011), which is described later in this article and is framed around five elements of cooperative teamwork, as depicted in Table 2. Utilising theory in the analysis phase can assist the reflection process by offering a lens to organise and understand a situation (Kelly & Rogers, 2022). In the conclusion stage, we ask how negatives could have been mitigated and positives strengthened. The last stage presents an action plan to improve future effectiveness.

Figure 1.

Gibb’s reflective cycle.

Table 2.

Five elements of cooperative teamwork from social interdependence theory.

Positive interdependence	Each team member can see how their contribution feeds in toward realising the group’s goal
Individual accountability	Team members know what they are responsible for and are held accountable for delivering agreed tasks
Promotive interaction	The group celebrates wins and acknowledges efforts so that team members feel appreciated and encouraged
Social skills	Team members know what is happening and why because communication is transparent, relevant, timely, respectful and accessible
Group processing	Time is carved out for group reflection on team dynamics and functioning

As this is a critical reflection on the impact of COVID-19 on our M&E activities, we layer Gibb’s reflective cycle with Patton’s (2015) reflective practice of ‘triangulated inquiry’ (p. 72). In this approach, Patton proposes that ‘reflective screens’ accompany every reflective question (p. 72). As such, when we ask ourselves what happened during the first stage of Gibb’s cycle, we simultaneously layer the reflection through considering the impact of our positionality. Therefore, how did our identity, ethnicity, age, gender, sexuality, values, culture, social status and education impact what happened? Patton extends this, asking us not only to consider this from our own perspective, but also from the perspectives of other actors taking part (or excluded) from the situation. Incorporating Patton’s deeper reflexive questions helps ameliorate some of the criticisms levelled at Gibb’s approach that it does not examine the underlying assumptions of the reflector. Figure 2 demonstrates how Gibb’s reflective cycle is used throughout this article alongside the five elements of cooperative teamwork from social interdependence theory and Patton’s triangulated inquiry.

Figure 2.

The theoretical framework.

There are obvious limitations to this auto-narrative based on an internal evaluator’s subjective reflection. The findings are not generalisable. However, they offer a different perspective via rich description and contextual insights that are beyond the scope of large-scale, generalisable, quantitative studies.

Reflecting around the cycle

Stage 1: Description

Throughout 2020 and 2021, we led the development of the M&E framework from the national office in Melbourne. While we were the facilitators, end user representatives from each region were engaged throughout the process. This participatory approach was vitally important for ensuring the framework was relevant (Fetterman et al., 2018), as end users provided context sensitive knowledge and information regarding the practicalities of proposed ideas. The inclusion and collaboration of these intended users was also essential for building buy-in and ownership of the framework, which invaluably impacted uptake (Patton, 2012).

The development of the M&E framework began with us gathering existing M&E related documentation from across emergency services and mapping the goals, outcomes, indicators and activities listed in each. We began this work using post-it-notes and whiteboards to theme the outcomes and activities, merge duplicates and distil key statements.

Once we started working from home in mid-March 2020, instead of isolating us from our colleagues, Red Cross rapidly invested in online collaboration tools and communication platforms. As occurred across workplaces globally (Al-Habaibeh et al., 2021), we went from fairly minimal use of Skype and telephone to heavy daily use of digital technology including MS Teams for videoconferencing, document sharing and task maintenance as well as embedded use of collaborative blackboards for group mind-mapping and problem solving. Although we were working from home, our interaction with a diversity of colleagues increased.

Because our capacity to work remotely increased, it facilitated greater involvement of staff nationwide in the framework development. Instead of a more tokenistic sense-checking approach that we may have used with remote staff members prior to the pandemic, we adopted principles of participatory monitoring and evaluation to enhance relevancy and utilisation through prioritising shared decision-making and learning, engaging in dialogue to negotiate direction and emphasising joint ownership of the product (Mulwa & Nguluu, 2003; Onyango, 2018). We coupled this with agile project management, which is recognised as providing a flexible, dynamic and collaborative approach suited to the uncertain, complex and rapidly evolving disaster and emergency management space (Beekharry, 2017; Kelly et al., 2022; Upadhyay et al., 2022). We employed the scrum methodology of the agile approach, which included establishing a group of staff (a squad) with nationwide representation. We held 2–3 hour weekly working meetings to delegate tasks and work through problems, 15-minute daily check-ins to assess progress and clear blockages, and monthly solutions-focused retrospectives (Flora & Chande, 2014). Having a dedicated project team meant that we had capacity to improve the rigour of the framework by reviewing and mining indicators from 40 similar frameworks existing in the literature. The group spent time building consensus by workshopping appropriate outcomes and indicators and discussing where, how and by whom the data contributing to each indicator would be collected and utilised.

The group members provided information and amendments to ensure that the framework would be meaningful in practice and trialled the drafted framework in each of their regions. The implementation process began before the official launch in December 2020 and continued throughout the first half of 2021, with all new programmes and projects linked to the framework by August 2021. We supported the implementation by providing advice and facilitating the development of logframes and M&E plans. Noting that evaluation capacity building helps embed and sustain M&E processes (Labin et al., 2012; McCoy et al., 2013; Rogers et al., 2019), we encouraged programme staff who had worked with us on the framework to take up roles as evaluation advocates (Rogers, 2021) and support their colleagues to do the same.

In August 2021 we reviewed the framework and deleted outcomes and indicators that had not been chosen in programme and project level logframes, assuming that this indicated irrelevancy. The framework narrative was heavily edited to provide a succinct document with enhanced utility for busy humanitarian workers. By mid-October 2021, we were still in lockdown when we sent the revised version of the framework to a designer.

Stage 2: Feelings

Writing this section, it was Day 261 of lockdown in Melbourne and the feeling all around was pretty flat, or ‘mediocre to poor’ as one of our colleagues remarked.

In early 2020, when we first started this process in the Melbourne office, we were excited to get started and comfortable with the methods of meeting with colleagues around a table, mapping thoughts out on a whiteboard and discussing ideas face-to-face. It felt like we just getting going when we were sent into the first lockdown and we worried that the work we had begun on the M&E framework would come to a halt as we were all pulled away to support the pandemic emergency response. We were concerned about what the pandemic would mean for our work and families, with many personal and professional unknowns. Research with 459 participants found that being female was a significant predictor of depression, anxiety and stress among people working from home during the pandemic (Şentürk et al., 2021). This finding aligns with our experience. As mothers to young children, we were overwhelmed by the simultaneous responsibilities of managing a demanding professional role whilst educating our children from home through remote learning.

Before the pandemic, we had made minimal use of technology such as Sharepoint or videoconferencing. The technology was a sharp learning curve, which we had to adopt quickly (Al-Habaibeh et al., 2021). The technology facilitated better collaborative opportunities with staff nationwide and it was exciting to see their high levels of maintained engagement, buy-in and cooperation. Their input added invaluable insights that improved the overall meaningfulness of the end product and has greatly enhanced its embeddedness across our emergency services team; hence why participatory methods are promoted throughout the evaluation literature (e.g., Fetterman et al., 2018; Patton, 2012).

Despite some minor frustrations on the rare occasion, a task was not completed on time, the process was positive and we enjoyed the opportunity to work more closely with staff in distant offices than we had previously. We felt that building rapport with staff nationwide, many of whom work in community facing roles, helped break down barriers between us in our national roles. Sometimes it seems that people working outside of the national team perceive the national team as authoritarian and disconnected from the work happening on the ground. Listening to and acting on their input, and implementing other participatory principles, seemed to enhance their engagement with the process and cultivate their interest in monitoring and evaluation (Fetterman et al., 2018; Mulwa & Nguluu, 2003). The collaborative immersion strengthened people’s commitment to the project, which translated into a stronger, more meaningful, and more fully adopted framework; thus, demonstrating the positives of internal evaluation roles outlined in the evaluation literature (Bourgeois et al., 2011; Kelly & Rogers, 2022; Sonnichsen, 2000; Yusa et al., 2016).

While we handled the overlapping burdens of managing our jobs and families during Melbourne’s long lockdowns, we feel tired and worn out. Considering the success of the M&E framework (the document is completed and is being adopted), we feel that COVID-19 did not negatively impact our productivity or output, in fact, it may have improved it as we will unpack in the next section. However, we feel that the additional stressors of lockdown negatively impacted our health, as was the case for women participating in the aforementioned study by Şentürk et al. (2021). Instead of pushing back, taking time off and prioritising self-care, we worked harder and longer to counter time spent supporting our families’ educational and emotional needs and ended up near burnout, despite organisational efforts to support our wellbeing.

Stage 3: Evaluation

To evaluate the impact of COVID-19 on the development of the M&E framework, we asked ourselves four key questions: How did COVID-19 make our work harder? How did it make things easier? What impact did it have on effectiveness and efficiency? And what impact did it have on collaboration across the team?

Beginning with the negatives, the impacts of COVID-19 negatively affected our ability to think clearly and be fully present in our professional lives. Instead, we worked longer hours to counter our perceived lack of productivity and ended up providing significant hours of free labour – noting that this is absolutely not condoned organisationally. We found it difficult to manage our busy schedule of online meetings alongside helping our children with their remote education (see also Al-Habaibeh et al., 2021). It was hard to lead a meeting or maintain a train of thought when either being interrupted, expecting to be interrupted or feeling bad for ineffectively catering to everyone’s needs.

Working from home had positives but was difficult when connections did not work, when family were distracting us during meetings, or when a lack of social cues meant that meanings were lost in translation. It was difficult not to be able to meet people in person, especially as many new staff joined during the pandemic. Pre-COVID-19, staff from across the country had opportunities to meet in person at least twice a year. These professional interactions were solidified into friendships through after-work drinks and lunchtime walks. Videoconferencing is an excellent communication tool, but opportunities for informal conversations and time to get to know people was not factored into professional meeting times (see also Al-Habaibeh et al., 2021; Wang et al., 2021). Additionally, with significant amounts of work to get through and never knowing when the next child was going to disrupt needing attention, work was prioritised over rapport. As well as limiting opportunities for rapport, the lack of informal conversations limited the ability for colleagues to raise little side queries, work through problems or discuss challenges that maybe did not fit clearly within the remit of a given meeting agenda (Wang et al., 2021; Whillans et al., 2021). The often fast-pace or awkward silences in online meetings made gauging where people were at difficult. This was important for us to know as monitoring and evaluation was new to many of our colleagues. It was hard to discern if silence equated to boredom, confusion, or acquiescence. This was especially the case when meeting without cameras, as noted by respondents in a study on remote working by Wang et al. (2021). While cameras were useful for rapport and gauging non-verbal cues, camera fatigue necessitated a balance.

Online collaboration worked well for some people, but there were others who seemed silenced by the technology, who we knew would offer more if approached informally in person. Similarly, we were not available to provide micro-support for people who could have walked past our desk in the office. Many people reached out to us for support during lockdown, including for small queries, but it was mostly people who we knew well and not the more random requests from less known people we would have received in the office setting. This sense of office networking with colleagues from across the various teams was certainly halted by lockdown, as noted in other workplaces (Whillans et al., 2021), where it led to feelings of loneliness and social isolation (Şentürk et al., 2021; Wang et al., 2021). Considering Patton’s (2015) reflective screens, we felt that part of this disconnection was our location in the national office, which as mentioned above can be seen as a place of power.

Despite the negatives, there were many unexpected positive outcomes that would not have occurred without the COVID-19 restrictions. These mostly related to enhanced efficiency, effectiveness, collaboration and inclusiveness. The key facilitators of these improvements were linked to working from home, travel restrictions and technology.

Working from home impacted efficiency by reducing travel times, limiting occasions when we were pulled into other work and quickening the pace of our work with back-to-back virtual meetings unimpeded by the need to move between physical rooms. It also had surprising positive impacts on effectiveness, collaboration and inclusion. Prior to the pandemic, emergency services staff would spend significant time travelling to communities and between offices. As this was largely curbed by restrictions, staff spent more time at their desks and were thus more available to participate in framework development. While this was likely a negative for community and other aspects of programme delivery, it was a positive for the M&E work. While our work is with domestically based staff members, we note the potential of reduced travel as a powerful catalyst for decolonising practice in international development and community-based settings (Mwambari, 2020).

As mentioned previously, the technology made our work easier in many ways. The pandemic forced us to become technologically savvy and increased our use of online platforms exponentially, which resulted in greater inclusion and the ability to purposefully hand-pick appropriate collaborators from across the country rather than simply grabbing whoever was available in the Melbourne office as we would have pre-COVID. This purposeful inclusion not only improved the quality and relevance of the M&E work, but also escalated buy-in as it cultivated nationwide advocates who were able to champion the work and assist with implementation (Rogers, 2021). Purposefully collaborating with diverse end users meant that our experience differed diametrically to the experience reported in research by Yang et al. (2022, p.43) who found that ‘remote work caused the collaboration network of workers to become more static and siloed, with fewer bridges between disparate parts’.

As well as helping us build bridges to collaborate with geographically distant co-workers, necessity increased organisational investment into technological solutions and hastened the phasing out of older, less efficient systems. Meanwhile, the significant spike in use encouraged technological companies to further improve their innovations (Al-Habaibeh et al., 2021). The fact that new technological solutions were rolled out nationwide meant that all staff were expected to learn the same technology simultaneously, which facilitated efficiency and collaboration as previously staff across the country worked in different ways.

Additionally, technology helped us organise our files and work collaboratively to plan and do tasks. It was helpful to have everything in one place, where our colleagues could access the necessary documents and information and work together. We found platforms such as Microsoft Teams useful for facilitating videoconferencing, recording meeting minutes, communicating through the chat functions, sharing live documents for simultaneous editing, storing necessary files and maintaining our group task planner. Online collaborative blackboards such as Miro allowed us to brainstorm, mind-map, plan and create with unlimited space and potential. These examples of technology helped us develop the M&E framework; however, other innovations helped us with its implementation, such as cloud-based data collection tools, which are beyond the scope of this article.

Stage 4: Analysis

When we consider why things went well or not well, it seems that most of the negatives were out of our realm of influence. The interruptions of children and the barriers that exist when attempting to build rapport in the digital space were difficult to ameliorate when options such as childcare and face-to-face meetings were not possible due to state of emergency and strict lockdown rules. We wielded much more control, however, over interpersonal elements. The process of developing the M&E framework was supported by three key facilitators: 1) incorporation and building on existing material, 2) inclusion of multiple, purposively chosen end users and 3) time dedicated to implementing principles of participatory monitoring and evaluation, building rapport, listening to everyone’s views, building in their feedback and acknowledging their contributions.

Analysis of what was happening on an interpersonal level can be assisted by the theory of social interdependence. Social interdependence is a social psychology theory, which has underpinned and framed research in a diverse range of disciplines for well over a century (Johnson, 2003). More recently, the theory has been applied to evaluation (Johnson et al., 2011). Social interdependence theory helps unpack group behaviour and understand how teams work together, define their purpose and achieve their goals (Tindale & Anderson, 2002). As such, it is deeply relevant to evaluation. However, it has particular significance to internal evaluation where enduring organisational relationships can benefit from closer investigation to improve evaluative processes, outputs and outcomes over the long term (Rogers et al., 2021). Examining our actions against this theory helped us analyse why things went well and what we could have done differently. Building on earlier theoretical foundations of social interdependence (Deutsch, 1949), five elements for operationalising effective teamwork were developed and empirically tested by Johnson and Johnson (2003, 2009). Thus, this section focuses on looking at our behaviours through the lens of social interdependence and its five elements of cooperative teamwork: positive interdependence, individual accountability, promotive interaction, social skills and group processing.

The first element of cooperative teamwork, positive interdependence, centres on linking individuals to the success of the group and is an important aspect of participatory and utilisation focused evaluation, which note the fundamental value of primary intended users’ ownership for utility, relevance and sustainability of the product (Fetterman et al., 2018; Patton, 2012). We were able to link individuals to the success of the group by highlighting why they had been purposefully hand-picked for the project and how their unique knowledge was vital for the project’s success. This acknowledgement was vital, particularly as many of the team members were new to evaluation and voiced early on that they wondered if they had anything to contribute. Team members noted that centring their contributions on their special skill and role as end users helped them to relax and know that their input was valued.

In the collaborations around building the M&E framework, a key aspect of our weekly meetings surrounded jointly developing a plan using an online task management and planning tool. This activity linked to the element of individual accountability where particular individuals were allocated specific tasks with short timelines. The small size of each task meant that tasks were nearly always completed on time, giving team members a sense of achievement and contribution. Daily 15-minute check-in meetings helped keep team members on track and engaged. A minority of team members were late with their tasks, which may link to Wang et al.’s (2021) finding that remote working during the COVID-19 pandemic increased procrastination as people struggled with self-discipline.

We attempted to enhance promotive interaction through providing encouragement, celebrating milestones and acknowledging team members’ contributions throughout the process. This included calling out particular contributions in meetings and providing positive feedback to team member’s supervisors. Team members knew that this encouragement was genuine and not tokenistic as they saw their suggested inclusions and changes reflected in the M&E material, which is a vital aspect of participatory monitoring and evaluation (Fetterman et al., 2018; Onyango, 2018).

Social skills were supported by our structured cadence of meetings and daily briefings that enabled regular interaction. Communication transparency and effectiveness was supplemented by summation of meeting minutes logged on the Microsoft Teams chat site and regular maintenance of the task planner, which sent reminders to individuals for timely completion of tasks. This element of cooperative teamwork was probably the one most negatively impacted by COVID-19 as the pandemic thwarted opportunities for face-to-face discussions (Yang et al., 2022). While efforts were made to include time for small talk, virtual meetings tended to be much more fast-paced and professional than would have been expected in face-to-face meetings, which tend to begin with more time for informal conversations (Wang et al., 2021; Whillans et al., 2021).

Group processing was facilitated through fortnightly retrospectives where the team engaged in informal everyday evaluation (Kelly & Rogers, 2022). These were typically 90-minute sessions where we used online blackboards to discuss three questions around what worked well, what could be improved and next steps. The last question was action-based and the agreed changes identified by the group were enacted soon after the meeting. Research on remote working during COVID-19 identified reduced opportunities for brainstorming and dialoguing through problems and ideas (Whillans et al., 2021). As these interactions need to be ‘deliberately orchestrated when working from home’ (Wang et al., 2021, p. 34), the agile cadence of meetings helpfully increased our ability to reflect, discuss and solve (Kelly et al., 2022).

Stage 5: Conclusion

In this penultimate step of the reflection, we note several learnings from our experience of developing an M&E framework during a pandemic. We learned how to pivot all of our tried and tested ways of bringing others along on the journey into the digital space. This was a sharp learning curve, necessitating quick adoption of new platforms, technology and project management approaches. Other learnings became evident in the months after the framework was launched. The M&E framework is being integrated through every aspect of operation, from programme design and practice, to broader strategy, research and operational guidance. The positive uptake is partially due to careful and consistent implementation support, but also due to the collaborative nature of the development process. This reinforced the importance of participatory monitoring and evaluation principles and of taking time to review, incorporate, acknowledge and build on existing documentation and practices. It demonstrated that purposefully cultivating cooperative teamwork not only increases effectiveness and efficiency, but also that the sense of ownership and improved relevance garnered through participatory ways of working has substantial positive ramifications for uptake (Fetterman et al., 2018; Patton, 2012), even in the remote online space. Linking back to other research, which found that remote working causes staff to work in siloes (Yang et al., 2022) and stifles collaborative brainstorming (Whillans et al., 2021), we found that these issues can be easily ameliorated by purposefully handpicking a project team of relevant end users and technical specialists from across the broader team and by establishing a regular cadence of meetings with space and suitable tools for brainstorming. Further, the uptake of the M&E framework highlights the benefits of dedicated internal evaluation staff who are trusted and known, and who are available to continually integrate, support and encourage adoption of evaluative tools and thinking (Kelly & Rogers, 2022; Love, 1991; Rogers et al., 2019; Sonnichsen, 2000; Volkov, 2011; Yusa et al., 2016).

Despite positives, ideally it would have been helpful to have in-person rapport with colleagues and ability to observe practice. Of the two of us who led the formation of the M&E framework, one started at Red Cross a few weeks before COVID-19, preventing scope for in-person rapport and programmatic observation. However, the other’s long tenure meant that she was known and trusted, which gave us a good foundation for collaboration (Kelly, 2021; Fetterman et al., 2018).

Another area for improvement was identified in the finalisation stage of the M&E framework. While remaining silent throughout the process, finally one of our colleagues raised some concerns about their ability to get local staff on board with the new framework and M&E tools. We met with them and their staff individually and resolved the concerns; however, the fact that this person had felt unable to raise their doubts earlier in the process made us question our approach and whether we had created enough space for dissent and inclusion of quiet voices. Using Patton’s (2015) screens for triangulated inquiry helped us consider why she might have waited and helped us recognise that we needed to develop skills to enhance inclusion in online forums and clearly reframe our positionality as facilitators rather than experts. Thinking back to the strategies outlined under the social skills element of cooperative teamwork, we realised that we could have put additional mechanisms for communication in place. For example, instead of expecting people to interject into the discussion, we could have explicitly suggested that people use the chat function or post questions on our online noticeboard. We could have run a session specifically asking colleagues about their concerns. We could also have created a virtual suggestion box or survey link to allow people to provide information anonymously.

Stage 6: Action plan

In this final step of the cycle, we consider how we will use the learnings from this experience to change our practice. This includes consideration of what we will do differently going forward to boost uptake and embedment of the framework. Having a plan for next steps that outlines skills to develop and actions to improve provides the reflective process with a clear pathway for development.

If we had the opportunity to work face-to-face, we would prioritise that; however, we have necessarily learned skills that enhance our ability to work collaboratively online. Rather than simply returning to face-to-face, we would incorporate the new tools and skills to develop a hybrid approach that enables the benefits of greater participation from geographically distant and accessibility challenged end users as well as increased efficiency via adoption of online methods, without the cost to rapport building and nuanced support that can be achieved through face-to-face interactions.

With the online component of the work, we would build-in various ways of ensuring inclusion and active participation. This could be an area for skills development around online group facilitation, which could be augmented through discussion with quieter colleagues about what would help them feel comfortable to contribute and how we could design workshops and meetings to maximise their participation. We would continue to develop project teams that draw diverse members and provide space for collaborative discussion to avoid the siloes and stagnation identified as a consequence of remote working by Yang et al. (2022) and Whillans et al. (2021). We would enhance these meetings, now that our children are back in the classroom and our personal lives are less entwined with our professional ones, by setting aside time for informal conversation to ameliorate known concerns regarding the lack of space for organic dialoguing and rapport building in remote working (Al-Habaibeh et al., 2021; Wang et al., 2021). As well as drawing from the literature on inclusion and active engagement in participatory monitoring and evaluation (e.g., Kumar, 2002), these efforts would contribute to this literature through expanded experience collaborating through hybrid and online-only means. Additionally, rather than simply utilising social interdependence theory to guide our reflections, we would operationalise it using the practical strategies for more effective teamwork outlined by Rogers et al. (2021).

Implementing the M&E framework across such a large team required significant investment of support capacity, which is ongoing. If we were to repeat the process, we would focus earlier and more purposively on identifying and upskilling evaluation advocates, using the field-guide developed by Rogers (2021). Clearly tasking these advocates to help with the rollout and build capacity among their colleagues could speed up the process, improve sustainability and remove single point dependencies, as noted throughout the evaluation capacity building literature (Beere, 2005; King, 2007; Labin et al., 2012; Volkov, 2008).

Implications for practice and final thoughts

Throughout history, pandemics have catalysed social transformations. The COVID-19 pandemic has fundamentally shifted the way we work with more of us working from home more often and greatly increased adoption of online videoconferencing and collaboration software. The importance now is not letting go of the lessons we have learned during remote working. Rather than reverting back to old ways, we should extend and improve our ability to work with people who for whatever reason find accessing face-to-face difficult. We can use the tools and skills born out of pandemic necessity to focus on inclusion and enhancing active participation in our M&E work, weaving the ideas of scholars such as Fetterman et al. (2018) and Mulwa and Nguluu (2003) with innovative online technology. In continuing this work, the frameworks guiding this article can offer others with a means of reflecting on their practice and positionality, particularly in regard to inclusion of marginalised groups.

ORCID iD

Leanne M Kelly https://orcid.org/0000-0002-3360-5212