Getting evidence into use: The experience of the Campbell Collaboration

When I became CEO of the Campbell Collaboration in late 2015, I had two goals: to increase production of Campbell reviews, and to increase their use by decision‐makers. Here I talk about the second of these, since, as I said at our first What Works Global Summit in London in 2016: ‘Research, what is it good for? Absolutely nothing… unless it is used by policy makers and practitioners’.

Before joining Campbell, I set up and led the International Initiative for Impact Evaluation (3ie), a global evidence fund for development interventions. At 3ie we monitored ‘policy influence stories’, from which I knew that the around 200 primary studies we funded generally had far more traction in influencing decisions than did the over 100 systematic reviews we supported. The reason was clear: primary studies have a ‘natural constituency’—the agencies implementing the programme being evaluated—with whom the researchers necessarily engage to conduct their study. Whereas a systematic reviewer can sit in her ivory tower and never meet a practitioner.

My approach to addressing this problem has been to work with user‐commissioners. These commissioners are not traditional research funders who allow researchers to go off and do what they like within some broadly defined parameters. User‐commissioners have a specific set of research questions they want answered, and a plan for using the findings.

There are two approaches to engaging with decision‐makers for reviews, which I call the ‘Nordic model’ and the ‘What works’ model. We have used both approaches.

The name Nordic model comes from the practice of government‐funded research agencies in Denmark, Norway and Sweden who produce systematic reviews to address policy issues agreed in advance with government agencies to address upcoming policy decisions. The policy‐makers are closely involved in setting review questions and interpreting the findings.

This approach has been used by Campbell reviews, such as that on body‐worn cameras commissioned by Arnold Ventures (Lum et al., 2020), the evidence and gap map on violence against children commissioned by UNICEF (Pundir et al., 2020), and the Bill & Melinda Gates Foundation‐commissioned review on deworming, intended to influence WHO guidelines (Welch et al., 2016). Most notable is our programme of reviews on countering violent extremism, brokered by the Co‐chair of our Crime & Justice Group, Peter Neyroud, and funded by the US Department of Homeland Security and Public Safety Canada. This programme has published four reviews to date—for example, Carthy et al. (2020) and Mazerolle et al. (2021)—with another nine ongoing—including Zych and Nasaescu (2021) and Wolfowicz et al. (2021).

But the Nordic model cannot work where decision‐making is decentralised amongst headteachers, social work teams, prison governors, police chiefs and so on, who number in their thousands across the country. In such cases the evidence‐based decision‐making products, such as evidence portals and guidance are a good approach. A prime example is the UK Education Endowment Foundation's Teaching and Learning Toolkit, which is used by two‐thirds of schools across England to inform decisions on how to spend resources and classroom management. This has an enormous real‐life impact for the research underpinning the toolkit. These products are based on systematic reviews. Ensuring that evidence‐based decision‐making products for agencies around the world are based on Campbell systematic reviews is a pathway to influencing decision‐making around the world.

The earliest evidence portal is the US Institute of Education Sciences' What Works Clearinghouse. There are now What Works centres across the USA and UK, covering a broad range of topics. Some of these—like the US National Institute of Justice's crimesolutions.gov, and What Works for Crime Reduction the UK—were already basing some of their content on Campbell systematic reviews. For example, the crimesolutions.gov practice entry ‘Focused deterrence strategies’ is informed by the review of Braga and Weisburd (2012).

In other cases, we have worked directly with What Works centres to produce evidence products for their needs. We have developed a strategic use of evidence and gap maps (White, 2021) in which we first map the available evidence before proceeding to commission reviews, or use existing reviews for the evidence portal.

The first such relationship has been the Centre for Homelessness Impact (CHI) for whom we produced evidence and gap maps of both effectiveness studies and process evaluations. The maps showed that homelessness is an ‘under‐reviewed’ area. Hence, CHI has commissioned a number of reviews which are based on studies in the map—for example, Keenan et al. (2021)—which are then used to inform the content of their evidence portal, which they call the intervention tool.

In addition, the forthcoming National Institute of Health and Care Excellence (NICE) guidance ‘Integrated health and social care for people experiencing homelessness’ used the evidence map to identify studies. The maps are being updated annually to form the basis for additional reviews being commissioned by CHI.

The same process is being used on an even larger scale with the Youth Endowment Fund (YEF), whose mission is to tackle serious violence amongst children aged 10‐14. As in the case of CHI, we began with evidence mapping. This is an area with many systematic reviews, many of which are published by Campbell, such as the review by Gaffney et al. (2021) on anti‐bullying programmes. Scoping notes were prepared and discussed with the research team and YEF staff, who would sometimes consult a broader range of stakeholders, such as the Home Office. We wrote technical reports summarising the evidence for each prioritised approach to preventing children and young people from becoming involved in violence. These technical reports are the basis for the YEF toolkit.

Thus far, 17 approaches have been added to the toolkit, which has topics of interest based on existing reviews. For later releases of the toolkit we are now commissioning systematic reviews from research teams, in some cases ‘topping up’ on‐going reviews, or commissioning updates of existing reviews or entirely new reviews.

We are now engaged in a similar process—map the evidence, then commission the reviews—with the Youth Futures Foundation on youth employment (Apunyo et al., 2022), and working with WHO on elder abuse and social isolation. With funding from the European Commission, we are also building a toolkit for youth employment in sub‐Saharan Africa.

What lessons can be drawn from this experience? The first relates to relationship building and reputation. Quite some time was spent to engage with the UK What Works movement, first at a general level and then with specific centres. In each case, there were conversations for one to two years before any work was commissioned. Each such experience builds reputation so the demand for following the approach also grows.

These relationships serve two purposes: increasing production of commissioned reviews and uptake of review findings. The What Works centres do the work of disseminating study findings. And, importantly, they have convening power we lack. An early stakeholder meeting for the YEF evidence and gap map included representatives from the Home Office, the probation services, the Youth Justice Board, youth offending teams, non‐profit organisations working with young people, and the Metropolitan Police. As Campbell alone we could not have convened such a group.

A second lesson relates to internal capacity. One part of this is that we established grant management capacity within the Campbell secretariat, for which I was fortunate to have 3ie's finance officer, Jatin Juneja, follow me from 3ie to Campbell. The second part is to have in‐house research capacity to respond to requests for studies in a timely manner. For this we first worked with a network of early‐ to mid‐career researchers across the world. But we now mainly rely on a research team based in India, which, under the very able leadership of Ashrita Saran became a free‐standing regional Campbell centre, Campbell South Asia.

A third lesson is preserving the Campbell brand and our reputation for gold standard systematic reviews. At the initiative of our Publications and Communications Manager, Chui Hsia Yong, we established Campbell Systematic Reviews as a journal published by Wiley, with an editorial board made of the Co‐chairs of the Campbell editorial groups. This change helps support our reputation and editorial independence from the secretariat, as did appointing our first full‐time Editor in Chief Vivian Welch.

I am very happy to have had the opportunity to help build the Campbell Collaboration. We now have a stronger publication track record, and many more cases of our reviews directly contributing to our mission of better evidence for a better world. May it continue in the years to come.