Skip to main content
Health Research Policy and Systems logoLink to Health Research Policy and Systems
letter
. 2019 Mar 28;17:33. doi: 10.1186/s12961-019-0432-3

The dark side of coproduction: do the costs outweigh the benefits for health research?

Kathryn Oliver 1,, Anita Kothari 2, Nicholas Mays 3
PMCID: PMC6437844  PMID: 30922339

Abstract

Background

Coproduction, a collaborative model of research that includes stakeholders in the research process, has been widely advocated as a means of facilitating research use and impact. We summarise the arguments in favour of coproduction, the different approaches to establishing coproductive work and their costs, and offer some advice as to when and how to consider coproduction.

Debate

Despite the multiplicity of reasons and incentives to coproduce, there is little consensus about what coproduction is, why we do it, what effects we are trying to achieve, or the best coproduction techniques to achieve policy, practice or population health change. Furthermore, coproduction is not free risk or cost. Tensions can arise throughout coproduced research processes between the different interests involved. We identify five types of costs associated with coproduced research affecting the research itself, the research process, professional risks for researchers and stakeholders, personal risks for researchers and stakeholders, and risks to the wider cause of scholarship. Yet, these costs are rarely referred to in the literature, which generally calls for greater inclusion of stakeholders in research processes, focusing exclusively on potential positives. There are few tools to help researchers avoid or alleviate risks to themselves and their stakeholders.

Conclusions

First, we recommend identifying specific motivations for coproduction and clarifying exactly which outcomes are required for whom for any particular piece of research. Second, we suggest selecting strategies specifically designed to enable these outcomes to be achieved, and properly evaluated. Finally, in the absence of strong evidence about the impact and process of coproduction, we advise a cautious approach to coproduction. This would involve conscious and reflective research practice, evaluation of how coproduced research practices change outcomes, and exploration of the costs and benefits of coproduction. We propose some preliminary advice to help decide when coproduction is likely to be more or less useful.

Keywords: Coproduction, research ethics, stakeholder engagement, evidence use, policy and practice

Background

It is now widely recognised that research evidence requires ‘translation’, beyond the standard academic journal article into some format which can be easily absorbed by policy-makers, practitioners or other ‘users’, if it is to have influence outside the academy [13]. In addition to interventions to increase knowledge translation and uptake [4], there has been a re-evaluation of what ‘counts’ as evidence for practice and policy [5, 6]. Moving away from its medical roots, the evidence-informed policy and practice movement and debate increasingly value experiential and practical knowledge [712]. Many researchers also feel that their research skills should be deployed in the service of those members of society who are less empowered – through, for example, participatory action research [13] – and that science in general should be made more inclusive and democratic [1416].

To achieve these diverse aims, collaborative research practices have been welcomed and promoted by many as the best way to answer these challenges [1725]. There are many forms of collaborative research practices, including coproduction, co-design, co-creation, stakeholder and public engagement, participation/involvement and integrated knowledge translation all sit under this umbrella [25], reflecting a very diverse set of motivations, activities, identities and discourses about how research interacts with the rest of society [26]. Coproduction originally referred to the development of more responsive, personalised public services through “the joint working of people who are not in the same organisation to produce goods or services” [2729]. These principles are increasingly applied to the production of knowledge, and coproduction is now a mainstream term in health research [19]. In addition, researchers are required and supported by universities, funders and publishers to create impactful research, with ‘engagement’ as a routine section on most grant proposals. Similarly, integrated knowledge translation approaches have gained traction, led by the example of Canadian researchers, governments and funders (e.g. Canadian Institutes of Health Research).

Given the confluence of ethical, practical and procedural reasons in support of coproduction, one might think that there is a good level of consensus about what coproduction is, how to do it, what the effects are, and an evidence-based set of techniques to achieve these effects. In fact, there are very few evaluations of how coproduction works in practice or on the impact of coproduction on research, practice, policy or population outcomes [30]. Given the pressures on researchers to coproduce in order to create research impact, we feel it is timely to closely examine the arguments and evidence for this. The purpose of this paper is to present the motivations for coproduction, identify associated tensions and costs, and identify where coproduction appears warranted.

Why coproduce?

There are four main arguments for coproduction in the literature:

  1. Substantive: where engagement is undertaken to improve the quality of the research [31]. This may be through helping researchers and policy-makers develop a more holistic understanding of a context, an issue and/or a solution [32, 33], both epistemologically and ontologically [25, 34]. The mechanisms by which research quality may improve are not always articulated, but many commentators agree that engagement makes research more relevant by focusing on appropriate topics which need elucidating [24, 35], and that by talking with others, we discover ‘unknown unknowns’ [36] and create new knowledge together.

  2. Instrumental: these arguments are based on “a desire to see research findings utilized in effective ways” ([31], p. 230). Many commentators claim that co-designed and coproduced research is likely to be more impactful [23, 24, 35, 37]. The actual process of collaborative research can help identify practice-based research questions and outcomes that are related to the implementation setting [38]. This includes practical mechanisms such as upskilling and creating capacity amongst non-academics [29, 35], and creating a sense of trust and empowerment amongst potential users, thus increasing the likelihood of research utilisation [39] and of evidence sharing [40, 41]. Innovative research designs and feasible ways to collect data can be identified through collaboration [42]. Further, stakeholders can describe how receptive the practice or policy setting is for subsequent implementation of the research findings [43].

  3. Normative: where engagement is simply “of intrinsic value” ([44], p. 153). This set of arguments connects to the discourse of the civic or public university/academic, focusing on accountability to (public) funders, and the conduct of research to serve public interests. Second, authors argue that sharing expertise (in its guise as power) is simply a way to be fairer and more ethical [37, 45]. Flinders characterises this view as the belief that coproduction can be “transformative not solely in research terms but in social terms: the engagement of citizens and social groups nourishes the renewal of democracy” ([26], p. 261). This often sits alongside a view that mutual and continual learning is a virtue of collaborative research practices – a clear shift from the paternalistic ‘science advice’ model still prevalent (see e.g. [46, 47]). Of course, individuals may also view coproduction as a satisfying and enjoyable way to practise – a rarely articulated but, in our view, valid justification for attempting it.

  4. Political: Often less explicitly made, but equally important, is a set of arguments used to justify coproduction on political grounds. By involving non-researchers in the elite, de facto exclusive process of research, coproduction can make users feel empowered and included [48, 49], and increase a sense of ownership (a prerequisite for acting on research findings) [39]. The integrated knowledge translation view specifies partnering with research users who have the authority to implement the findings [50]. Greater collaboration can reduce negative stereotypes held by researchers and policy-makers about each other [51], and improve trust [36, 52], smoothing the way for research to be more impactful. Several commentators also claim that coproduced research is more credible and relevant to intended audiences [39], and that the legitimacy of the knowledge [53], goals and activities [54] are increased. The motivation for either approach – collaborating with the disempowered or the powerful – is political, in the sense that it is about changing the attitudes of different groups to attain a particular end, even to the extent of being explicitly framed as a political and strategic response to diminishing belief in public services [55].

Coproductive approaches are therefore used for a variety of reasons. In the spirit of evidence-based enquiry, we ask whether coproduction is likely to enable researchers to realise these four objectives and at what cost.

How to coproduce?

There is a multiplicity of modes by which researchers may interact with stakeholders [25]. At very low, shallow levels of involvement, stakeholders may be the mere recipients of research findings [56]. More active interaction types may include reaching out through soft interpersonal networks to seek advice about policy agendas, or to influence policy dialogues individually [57]. Researchers may be employed by public policy institutes to do research commissioned by and directly applicable to policy issues. At the most engaged level, what many would regard as ‘truly’ coproductive approaches have been tried in specific, partnered project grant calls, research institutes or long-standing research programmes (such as the English NHS Collaborations for Leadership in Applied Health Care and Research; CLAHRCs) [5861] implying an equality of status between all co-researchers. Often, a financial contribution by the stakeholder is indicative of an authentic coproduction partnership. These coproductive approaches often include elements such as provision of opportunities to co-learn about research and substantive topics, to work iteratively over the lifespan of a project to co-steer the questions and direction of research, or to develop interpretations of data and implications of these together. However, there are very few evaluations of such approaches or projects, and existing evaluations often point to a “lack of hard evidence as to its impact on the quality of research” ([62], p. 3) and a need to “determin[e] best practices for engaging patients” ([62], p. 3). We concur with these findings.

As originally intended, coproduction referred to a significant shift of power from researchers or decision-makers to service users (e.g. patients) [26, 27], implying deeply embedded collaborative practices. However, it has come to refer to activities as diverse as consultation on topics, co-designing research questions, co-interpreting results and recommendations, and embedded or in-house researchers within policy or service delivery agencies [25]. These activities describe very different modes of interaction, and imply different practical resourcing needs, skills and processes, as well as outcomes. However, there is very limited evidence about the impact of each type of strategy [25, 60, 63], no consensus about the best theories to use to inform this work [64], and little knowledge as to which strategies will help achieve which aims. Yet, millions of pounds are invested into deliberative, coproductive, collaborative research processes, and to related initiatives supporting research uptake and research impact, without (1) looking at what works or (2) knowing the pros and cons of different approaches or (3) evaluating and monitoring these processes. We might expect to see changes to policy or practice, to population outcomes, to research practices or to research outputs, yet no evaluation of coproduction attempts to explore all the domains holistically [65, 66]; this matters, because coproduction is not free of risk or cost. As Flinders et al. note [26], coproduction carries significant risks for academics, who are required to adopt practices far from those traditionally taught, adopted, recognised or rewarded by the academy. They argue that coproduction is “time-consuming, ethically complex, emotionally demanding, inherently unstable, vulnerable to external shocks, subject to competing demands and expectation” ([26], p. 266). In the next section, we explore how coproductive approaches may generate tensions through the research process and identify costs to participants.

Challenges and costs to coproduction

Challenges can arise throughout the lifecycle of a research project, and more generally through the research process (see Table 1 for a non-exhaustive list based on our own experiences). From the framing of research questions to the development and dissemination of recommendations, coproductive research can cause conflict, consume resources and lead to misunderstandings. The very purpose of a collaboration may not always be clear to all, or be shared. The direction of a research project may not (or possibly cannot) reflect the values and priorities of all stakeholders, and team members may have different views about what the research findings show, or what to do with them.

Table 1.

Challenges and costs in coproduced research

Challenges which may arise Costs
Developing mixed research teams Stakeholders not homogenous, and can disagree
‘Usual suspects’ can take over, where coproductive discussions are dominated by certain individuals
The research process may take more time compared to a traditional research process
Shared decision-making is threatened when process dominated by certain voices or interests
Framing research questions Stakeholders and researchers may have different priorities and values
Useful research can lack originality
Research can be co-opted by partners, for example, to justify status quo or historical decisions
Damage to interpersonal or organisational relationships
Damage to research careers
Damage to researcher independence and credibility
Collecting data Researchers may pressure stakeholders to allow their organisational resources to be used to facilitate data collection –e.g. using staff time or applying pressure for site access Damage to interpersonal or organisational relationships, particularly with more powerful stakeholders
Analysing and interpreting data Stakeholders may want to know which participant agreed to participate or what they contributed to the dataset
Stakeholders may want to help analyse the data
Violation of research ethics obligations
Researcher needs to train stakeholders and format data in an appropriate way to conform with research ethics obligations
Formulating recommendations May be little agreement about the importance of research
Researchers may be pressed to frame findings in particular ways
Findings are misrepresented
Damage to researcher independence and credibility
Disseminating research Researchers or stakeholders may be prevented from sharing unwanted findings
Stakeholders may want to share findings before researchers are ready
Damage to researcher independence and credibility
Damage to the credibility of the research process
Implementing change Tension between advocating for research, or advocating for policy/practice changes
Researchers show little interest in providing assistance with implementation efforts
Can damage relationship with practice or policy colleagues
Implementation of research findings fail

Beyond the lifecycle of an individual research project, coproductive research can also mean (1) investing time and resources into relationships with no guaranteed concrete output; (2) calling on favours and being able to provide favours in return; and (3) in general, having the capacity to manage sometimes tense relationships. Each of these challenges is also associated with costs for researchers and stakeholders. Below, we list and describe these costs (see Fig. 1 for a summary).

Fig. 1.

Fig. 1

Costs of coproduction

Firstly, there are clear practical costs to doing research in this way. It is expensive, as it requires the presence or time of multiple actors who are often not on site, have other primary responsibilities, or need travel or other reimbursement. The administrative burden of arranging spaces and attendance is significant. In addition, it takes skill to recruit and manage mixed groups who are working together to create something new. Managing group dynamics, not letting the loudest dominate and maintaining rules of conduct require interpersonal skills which not all academics are trained in or endowed with. Thus, a set of professional skills and resources are needed – but this type of work is often added on (to ‘real research’) with little thought for how to properly resource it [63]. A gendered aspect may feature in coproduction work; our own experiences suggest that coproduction conferences are attended by women and coproduced projects are often led by women. Senior men may be keener to manage coproduction work with senior managers, clinicians and policy advisers than with lay people, patients, carers, etc. How, why and the consequences of this possible imbalance require further thought.

Second, there are significant personal costs to researchers. Many of the tensions above play out through interpersonal conflict, difficult conversations and/or outright disagreement. Many people find this very difficult to handle in the workplace, and when research careers, outputs and funding are at stake, this only adds to the pressure. There are reputational risks associated with conflict, which, for many, is reason enough to avoid difficulties; yet, ignoring these challenges and going along with stakeholder views with little negotiation of mutual interests can lead to unnecessary duplicative research and perhaps less imaginative or derivative research, which can also compromise research(er) integrity. It is too easy to dismiss these tensions as being because of interpersonal tensions; rather, we argue that these tensions are how inherent power imbalances and conflicts in coproductive research are expressed. Badging them as being because of interpersonal difficulties dismisses the emotional labour of working collaboratively, and risks ignoring the need to find solutions for real challenges inherent in this type of research practice. Managing the engagement process takes personal qualities that are not hugely abundant amongst the academic population. These interpersonal conflicts can lead to – as can the additional work associated with coproduction – serious stress and burnout [67].

Third, there are professional costs to researchers. Taking time out of seeking research funding, writing publications in impactful journals, doing administration, and teaching is a huge ask for academics, and coproduction requires significant commitment of time and resources, often with little guarantee of success or recognition. Coproduction can lead to research outputs that are regarded as being of lower quality than ‘real’ or ‘pure’ research [26], or simply hard to publish. For engagement to occur successfully, researchers are often required to do professional favours (e.g. do extra presentations), be easily and readily available, and be able to engage with no expectation of a guaranteed impact as measured in academia. This is clearly not the case for all researchers, and may lead to the costs of coproduction being borne by those least able in terms of time or capacity – in our experience, often the most junior, temporary, female and/or of colour.

Additionally, researchers risk being seen as partisan and/or lacking in credibility [68, 69], by both their colleagues and by their stakeholders. Researchers may find they are being used to add legitimacy to political positions [70] and may find that adopting political or policy positions, or being seen to endorse political viewpoints, can hinder their ability to work with a range of stakeholders – even compromising personal safety [71]. It can also lead to the view that the research itself is not a neutral representation of the data, but a cherry-picked narrative to support a political position – producing “policy-based evidence” [72]. Finally, as well as being regarded as partisan and biased, coproductive researchers risk being regarded as an academic “lightweight” [73], producing nothing of substance.

Fourth, there are risks to the research itself. Under business-as-usual rules, researchers spend their time identifying genuine and novel gaps in the knowledge base, which have to be justified at length to colleagues and funders. However, the coproduction process can lead to researchers being asked to answer questions which are dull, not novel (little contribution to the scientific literature), or not generalisable (focused on local issues) – and therefore not easily publishable. As Kothari indicates, coproductive research can “encourage a reassertion of control and power by dominant individuals and groups” who possess the skills to engage with the project, or those whose contributions fit in with any preconceived ideas of the ‘right’ responses will be highlighted over those who “lack the skills to perform as required” ([74]: p. 142). This can lead to research being conducted on unnecessary or repetitious topics, or on topics which serve the interests of some groups over others. Of course, this is often the case, but coproduction offers a vision of equality and fairness which can mask these dynamics.

The research process can be significantly affected, with delays caused by the additional work of recruiting and engaging with stakeholders; differences of opinion about the purpose and role of coproduction or the research itself; tensions around how to frame interpretations and recommendations that may result in research being co-opted or researchers being silenced (at a micro level, during conversations, or procedurally through being prevented from publishing); and in how and when to share the findings. These concerns are not new to implementation science or ethnographic researchers (see, for example [7577]) and the tensions around the research process itself have long been a domain of inquiry for social researchers. Yet, in the current climate, the culture of hit-and-run research (get funding, do research, achieve impact, leave) does not easily allow researchers the time and space to reflect on their practice, and any proffered thoughts would be considered incidental to the main project, irrelevant or possibly unwanted by funders.

Fifth, there are costs to stakeholders. In addition to taking time and resources away from their primary responsibilities, coproduction requires that participants – especially policy-makers – make themselves vulnerable, by sharing uncertainties or sensitive information [36]. This could include admitting the political uncertainty, lack of policy direction, or political mistakes – all of which could incur costs. More broadly, one can imagine scenarios where participants were asked for input, which was then cherry-picked to suit research agendas, or where their contributions were not treated transparently and systematically. Policy-makers, service users and professionals can feel ignored or used, may not understand why their voices are not represented in the way they had expected, or may feel that they were not accorded the usual respect they enjoy in the workplace. Far from empowering people politically, participation in research can lead to personal narratives and experiences being dominated by senior interests, leading to lack of motivation to engage again [54], and the ‘subjugation’ of participants [78, 79].

Finally, there are costs to the entire profession, or the ‘scientific endeavour’, by the reduction of trust in science and scientists. Doing coproduction recklessly, discourteously or without due attention to professional etiquette can cause significant ill-feeling about participating in research. This can lead to mistrust of researchers, and a lack of willingness to engage in other research processes. Furthermore, the issues about independence and credibility can call the value of academic research into question. If researchers are working so closely with parties with clearly vested interests in particular aims, how can anyone be confident that the findings have been represented truly and holistically? This can lead to research being viewed as simply one more lobby group, rather than holding a special status [80].

A cautious approach to coproduction: how to minimise the costs and maximise the benefits of coproduction

We recognise the transformative potential of coproduced research, and also feel that calls to do impactful research are unlikely to go away. If done well, coproduced research processes can – indeed must – manage and alleviate these tensions. Yet, there is a significant dearth of empirical evidence about coproduction processes or outcomes, which makes its widespread advocacy troubling. We simply do not know which strategies are the most promising for collaborative research. For example, which stages of the research process are critical for stakeholder involvement, and which stages are less essential? What are the best ways to keep stakeholders engaged with the process? Under what circumstances should coproduction be used, rather than other (e.g. consultative) approaches? What types of infrastructure need to be in place to support productive coproduction?

What then should be done? We suggest a two-step process when commissioning and/or initiating each research project, as follows:

  • First, considering whether coproduction is likely to be useful in helping the research meet its aims and selecting strategies accordingly, and

  • Second, considering whether other approaches are as or more likely to help achieve those aims.

This process is likely to be the responsibility not just of the researchers but also policy-makers, practitioners, funders and patient/user representatives.

Using the four arguments for coproduction above, researchers and others (that is, funders, policy-makers, managers, clinicians, the public, patients, carers, etc., depending on the topic) should identify specific potential aims for any coproduction process – for example, to identify or refine a research question, or to broker access to a political community – and decide whether coproductive strategies can best help them achieve those aims, and if so which ones. Having identified the purpose of the research and its desired outcomes in relation to the context and the interests potentially affected by the findings, the researchers and others should attempt to identify the best strategies to achieve these, and thus decide when and where coproduction might help, and when and where it might not.

Given the lack of much evidence in this area, we have to make judgements based on the circumstances of each project. Some early thinking along these lines is summarised in Table 2, which particularly relates to coproduction of evaluations with policy-makers and service managers.

Table 2.

Some early thoughts on when to favour coproduction and when not to

Less emphasis on coproduction when … More emphasis when …
The policy or programme is likely to be controversial or the findings are likely to be contested The policy or programme is widely regarded as a ‘good thing’ and the findings are unlikely to be contested
Conflicts of interest between stakeholders are likely to be hard to manage (e.g. policy-makers are directly responsible for the successful delivery of a policy or programme) There are few fundamental conflicts of interest between stakeholders (e.g. policy-makers are not directly responsible for successful delivery of a policy or programme)
There is less concern to use the findings directly and immediately for policy or management decisions The main goal is to ‘use’ the findings for policy and management decisions
Funders and/or commissioners of the research value ‘expert’, dispassionate scientific inquiry above other forms of knowledge There are few concerns about the limitations of ‘policy-based evidence’
The nature and purpose of the policy or programme is relatively well defined and agreed upon The policy or programme still needs considerable definition, refinement, testing of feasibility, acceptability, etc.
The prime purpose of the research is to establish whether the policy or programme ‘works’ and there is strong prior commitment by policy-makers or managers to acting on the findings (i.e. ‘decision space’ is available) The prime purpose of the research is to identify how best to implement the policy or programme rather than whether or not to proceed with it
Undertaking the research is less dependent on cooperation of policy agencies or local programme implementers The research cannot easily be carried out without the active cooperation of policy-makers and/or local programme implementers
There are good, informal, ongoing relationships between the researchers, and policy-makers and service managers There is a need to increase mutual awareness and understanding between researchers, and policy-makers and service managers
One or more of time, resources and expertise are in limited supply to involve the key stakeholders at all appropriate points in the research process Time, resources and expertise are available to involve the key stakeholders at all appropriate points in the research process

We suggest that a cautious approach to coproduction would include examining the costs and benefits to all involved, recognising the significant costs and risks to investing time and resources into good facilitation and management of expectations, establishing ground rules and processes, and deciding on evidence-informed strategies to achieve established and shared aims and outcomes. Below, we summarise some key questions which we believe all those involved in coproduction should ask themselves prior to attempting any engagement.

Individual researchers, funders, commissioners and participants in the research process should ask themselves:

  1. What is everyone bringing to the table? For example, policy-makers and funders bring money, knowledge of the political context, pressure for answers; researchers bring topic and methodological expertise; public and patients bring their experiences.

  2. Under which circumstances are these needed, for what purpose, and at which stage of the research process? For example, when is it better to have patient representatives articulate the user perspective rather than derive understanding from a systematic review of patient experiences? Should policy-makers be involved all the way through, or only at specific stages?

  3. What are the costs? How will the time, administrative, cultural and professional costs be borne and defrayed by those involved?

  4. How are decisions taken, and how will responsibility and accountability be shared? Will group dynamics, market forces, authority or some other decision-making process control the process? How will this be governed and managed?

Research organisations and funders need to consider:

  1. How to create (co-create) and support the infrastructure and leadership for coproduction

  2. How to provide training in coproduction, and help researchers and funders take this seriously as a skill set

  3. How to reward good practice, and to recognise the work coproduction may take even if it does not lead to impact

  4. How to evaluate the potential impact(s) of coproduction

  5. How to ensure that coproduction supports, rather than mitigates against diversity and quality in research and policy

Conclusions

Coproduction is an exciting approach to research that can generate truly novel, unexpected outputs [19]. However, it takes investment, skills, time and courtesy. Engaging stakeholders should be done for the right reasons and in the right way [52], yet there is so little empirical evidence about how coproduction changes research, policy or practice, or how it may compare to alternatives. Coproduction as a way to generate knowledge can be seen as one way to make the resulting research more ‘implementation ready’ on the assumption that, through the coproduction process, the team has already anticipated research users’ needs, capacities and priorities. Yet, there may be alternative ways to achieve this outcome without risking the costs detailed above.

The main motivations of researchers to engage in coproductive work are of course positive, with genuine belief in the importance and potential transformational nature of working with those outside academia. However, the political arguments are often not made explicitly (see, e.g. [31]). To us, this suggests that, either some researchers engage naively (albeit in good faith), or that researchers appreciate the political nature of decision-making and research, but prefer not to openly and honestly explain their strategies to deal with these dynamics. We also note that most normative, substantive and instrumental justifications for coproduction can be achieved through other means. Thus, we argue that political reasons for engaging in coproductive research may be the least-discussed, yet most important rationale made by researchers. Given the costs, we argue that this requires serious attention by the research community, including those who commission, participate in and evaluate research.

Coproduction and participation may have a profound impact on the practice of research and the process of decision-making, and there are many promising and exciting examples of joint inquiry between researchers and stakeholders [30, 59, 8183]. There are certainly many unanswered questions about the potential value of coproduction and collaborative research, and outlining the benefits and the mechanisms generating those benefits is an important task.

However, the tensions and challenges, costs and opportunities must also be described; they are unequally experienced and borne, strongly suggesting that mindful engagement is essential for the ethical practice of research. Many of these points of tension and costs are about power, and we recognise that all the implementation type tools, frameworks and processes cannot design away these tensions. There must be a more reflective, open discussion about when to do coproduction, the ethics of coproduction, and so on; we need more empirical evidence about its processes and outcomes; and we do not know which types are best suited for which problems, institutions or team configurations. As a community, it is imperative that we are more reflective about how coproduction influences the process of research, and the roles and responsibilities of everyone involved in collaborations [84].

Acknowledgements

We thank all attendees of the UK Implementation Science conference 2018 who saw and commented on an early version of this paper.

Funding

This paper was not supported by any current funding.

Availability of data and materials

Not applicable.

Authors’ contributions

KO had the idea for the paper, and led the writing process. AK and NM contributed critical aspects of the argumentation, developed the conceptual basis for the paper, and contributed to several drafts. All authors approved the final version.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Kathryn Oliver, Email: kathryn.oliver@lshtm.ac.uk.

Anita Kothari, Email: akothari@uwo.ca.

Nicholas Mays, Email: nicholas.mays@lshtm.ac.uk.

References

  • 1.Traynor R, DeCorby K, Dobbins M. Knowledge brokering in public health: a tale of two studies. Public Health. 2014;128(6):533–544. doi: 10.1016/j.puhe.2014.01.015. [DOI] [PubMed] [Google Scholar]
  • 2.Connelly S, Vanderhoven D. Translation across borders: exploring the use, relevance and impact of Academic Research in the Policy Process: Policy Press; 2015. p. 181–98.
  • 3.Lane JP, Rogers JD. Engaging national organizations for knowledge translation: comparative case studies in knowledge value mapping. Implement Sci. 2011;6(1):106. doi: 10.1186/1748-5908-6-106. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Boaz A, Baeza J, Fraser A. Effective implementation of research into practice: an overview of systematic reviews of the health literature. BMC Research Notes. 2011;4:212. [DOI] [PMC free article] [PubMed]
  • 5.Cairney P. The Politics of Evidence-Based Policy Making. London: Palgrave Macmillan UK; 2015. [Google Scholar]
  • 6.Oliver K, Lorenc T, Innvær S. New directions in evidence-based policy research: a critical analysis of the literature. Health Res Policy Syst. 2014;12(1):34. doi: 10.1186/1478-4505-12-34. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Fre Z. Knowledge Sovereignty among African Cattle Herders. London: UCL Press; 2018.
  • 8.Elliott H, Popay J. How are policy makers using evidence? Models of research utilisation and local NHS policy making. J Epidemiol Community Health. 2000;54(6):461–468. doi: 10.1136/jech.54.6.461. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Russ LW, Takahashi LM, Ho W, Tseng W, Ponce NA. Bridging academic-legislative divides: models of policy-relevant health research and practice by the university of california. Prog Community Heal Partnerships Res Educ Action. 2012;6(1):95–102. doi: 10.1353/cpr.2012.0009. [DOI] [PubMed] [Google Scholar]
  • 10.Wieringa S, Greenhalgh T. 10 years of mindlines: a systematic review and commentary. Implement Sci. 2015;10:45. doi: 10.1186/s13012-015-0229-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Greenhalgh T. Narrative based medicine: narrative based medicine in an evidence based world. BMJ. 1999;318(7179):323–325. doi: 10.1136/bmj.318.7179.323. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Oliver S, Duncan S. Editorial: The challenges of sharing different ways of knowing. Res All. 2018;2(1):1–5. [Google Scholar]
  • 13.Langley J, Wolstenholme D, Cooke J. ‘Collective making’ as knowledge mobilisation: the contribution of participatory design in the co-creation of knowledge in healthcare. BMC Health Serv Res. 2018;18(1):585. doi: 10.1186/s12913-018-3397-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Sabine M, Peter W, Maasen S, Weingart P. Democratization of Expertise? - Exploring novel forms of scientific advice in political decision-making: Springer; 2005. p. 238.
  • 15.Carrozza C. Democratizing Expertise and Environmental Governance: Different Approaches to the Politics of Science and their Relevance for Policy Analysis. J Environ Policy Plan. 2015;17(1):108–126. [Google Scholar]
  • 16.Stewart R. Terminology and tensions within evidence- informed decision-making in South Africa over a 15-year period. Res All. 2017;1(2):252–264. [Google Scholar]
  • 17.Oliver K, Cairney P. The dos and don’ts of influencing policy: a systematic review of advice to academics. Palgrave Commun. 2019;5(1):21.
  • 18.Durose C, Mangan C, Needham C, Rees J, Hilton M. Transforming local public services through co-production. Swindon: Arts and Humanities Research Council; 2013.
  • 19.Filipe A, Renedo A, Marston C. The co-production of what? Knowledge, values, and social relations in health care. PLoS Biol. 2017;15(5):e2001403. doi: 10.1371/journal.pbio.2001403. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Traynor R, Dobbins M, DeCorby K. Challenges of partnership research: insights from a collaborative partnership in evidence-informed public health decision making. Evid Policy A J Res Debate Pract. 2015;11(1):99–109. [Google Scholar]
  • 21.Shippee ND, Domecq Garces JP, Prutsky Lopez GJ, Wang Z, Elraiyah TA, Nabhan M, et al. Patient and service user engagement in research: A systematic review and synthesized framework. Health Expect. 2015;18(5):1151–1166. doi: 10.1111/hex.12090. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Boivin A, L’Espérance A, Gauvin F-P, Dumez V, Macaulay AC, Lehoux P, et al. Patient and public engagement in research and health system decision making: A systematic review of evaluation tools. Heal Expect. 2018;21(6):1075–1084. doi: 10.1111/hex.12804. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Oliver S, Liabo K, Stewart R, Rees R. Public involvement in research: making sense of the diversity. J Health Serv Res Policy. 2015;20(1):45–51. doi: 10.1177/1355819614551848. [DOI] [PubMed] [Google Scholar]
  • 24.Barber R, Beresford P, Boote J, Cooper C, Faulkner A. Evaluating the impact of service user involvement on research: a prospective case study. Int J Consum Stud. 2011;35(6):609–615. [Google Scholar]
  • 25.Fransman J. Charting a course to an emerging field of ‘research engagement studies’: A conceptual meta-synthesis. Res All. 2018;2(2):1–49. [Google Scholar]
  • 26.Flinders M, Wood M, Cunningham M. The politics of co-production: risks, limits and pollution. Evid Policy A J Res Debate Pract. 2016;12(2):261–279. [Google Scholar]
  • 27.Ostrom E. Crossing the great divide: coproduction, synergy, and development. World Dev. 1996;24(6):1073–1087. [Google Scholar]
  • 28.Durose C, Needham C, Mangan C, Rees J. Generating “good enough” evidence for co-production. Evid Policy. 2017;13:135–151. [Google Scholar]
  • 29.Iedema R, Carroll K. The “clinalyst.”. J Organ Chang Manag. 2011;24(2):175–190. [Google Scholar]
  • 30.Gagliardi A, Kothari A, Graham I, Gagliardi AR, Kothari A, Graham ID. Research agenda for integrated knowledge translation (IKT) in healthcare: What we know and do not yet know. J Epidemiol Community Health. 2017;71(2):105–106. doi: 10.1136/jech-2016-207743. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Duncan S, Oliver S. Editorial: Motivations for engagement. Res All. 2017;1(2):229–233. [Google Scholar]
  • 32.Walter I, Nutley S, Davies H. Research impact: a cross sector literature review. St. Andrews: Research Unit for Research Utilisation, Department of Management University of St. Andrews; 2003. 
  • 33.Oliver K, Rees R, Brady LM, Kavanagh J, Oliver S, Thomas J. Broadening public participation in systematic reviews: a case example involving young people in two configurative reviews. Res Synth Methods. 2015;6(2):206–217. doi: 10.1002/jrsm.1145. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Liabo K, Stewart R. Involvement in research without compromising research quality. J Health Serv Res Policy. 2012;17:248–251. doi: 10.1258/jhsrp.2012.011086. [DOI] [PubMed] [Google Scholar]
  • 35.Goodyear-Smith F, Jackson C, Greenhalgh T. Co-design and implementation research: challenges and solutions for ethics committees. BMC Med Ethics. 2015;16:78. doi: 10.1186/s12910-015-0072-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Maybin J. How proximity and trust are key factors in getting research to feed into policymaking. LSE Impact Blog. 2016; http://blogs.lse.ac.uk/impactofsocialsciences/2016/09/12/how-proximity-and-trust-are-key-factors-in-getting-research-to-feed-into-policymaking/. Accessed 1 Aug 2018.
  • 37.Nesta. Seven Principles for Public Engagement in Research and Innovation Policymaking: Nesta; 2018. https://www.nesta.org.uk/report/seven-principles-public-engagement-research-and-innovation-policymaking/. Accessed 15 Dec 2018.
  • 38.Rycroft-Malone J, Burton C, Wilkinson J, Harvey G, McCormack B, Baker R. Collective action for knowledge mobilisation: a realist evaluation of the Collaborations for Leadership in Applied Health Research and Care. Health Services Delivery Research. Southampton: NIHR Journals Library; 2015. [PubMed] [Google Scholar]
  • 39.Ghate D. Developing theories of change for social programmes: co-producing evidence-supported quality improvement. Palgrave Commun. 2018;4(1):90. [Google Scholar]
  • 40.Dobbins M, Robeson P, Ciliska D, Hanna S, Cameron R, O’Mara L, et al. A description of a knowledge broker role implemented as part of a randomized controlled trial evaluating three knowledge translation strategies. Implement Sci. 2009;4:23. doi: 10.1186/1748-5908-4-23. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Armstrong R, Waters E, Roberts H, Oliver S, Popay J. The role and theoretical evolution of knowledge translation and exchange in public health. J Public Health. 2006;28:384–389. doi: 10.1093/pubmed/fdl072. [DOI] [PubMed] [Google Scholar]
  • 42.Bowen S, Martens P. Demystifying knowledge translation: learning from the community. J Health Serv Res Policy. 2005;10(4):203–211. doi: 10.1258/135581905774414213. [DOI] [PubMed] [Google Scholar]
  • 43.Gagliardi A, Dobrow MJ. Identifying the conditions needed for integrated knowledge translation (IKT) in health care organizations: qualitative interviews with researchers and research users. BMC Health Serv Res. 2016;16:256. doi: 10.1186/s12913-016-1533-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Conklin A, Morris Z, Nolte E. What is the evidence base for public involvement in health-care policy?: Results of a systematic scoping review. Health Expect. 2015;18(2):153–165. doi: 10.1111/hex.12038. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Doubleday RRR, Wynne B. Despotism and Democracy in the United Kingdom. In: Jasanoff S, Caplan A, editors. Reframing Rights: Bioconstitutionalism in the Genetic Age; 2011. p. 239–61.
  • 46.Gluckman P. The art of science advice to the government. Nature. 2014;507:163–165. doi: 10.1038/507163a. [DOI] [PubMed] [Google Scholar]
  • 47.Sutherland WJ, Burgman MA. Policy advice: use experts wisely. Nature. 2015;526:317–318. doi: 10.1038/526317a. [DOI] [PubMed] [Google Scholar]
  • 48.Muir Gray JA. Evidence based policy making. BMJ. 2004;329(7473):988–989. doi: 10.1136/bmj.329.7473.988. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Beresford P. Developing the Theoretical Basis for Service User/Survivor-Led Research. Epidemiol Psychiatr Sci. 2005;14(1):4–9. doi: 10.1017/s1121189x0000186x. [DOI] [PubMed] [Google Scholar]
  • 50.Kothari A, McCutcheon C, Graham I. Defining integrated knowledge translation and moving forward: a response to recent commentaries. Int J Heal Policy Manag. 2017;6(x):1–2. doi: 10.15171/ijhpm.2017.15. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14(1):–2. [DOI] [PMC free article] [PubMed]
  • 52.Boaz A, Hanney S, Borst R, O’Shea A, Kok M. How to engage stakeholders in research: design principles to support improvement. Health Res Policy Syst. 2018;16(1):60. doi: 10.1186/s12961-018-0337-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 53.Cash DW, Clark WC, Alcock F, Dickson NM, Eckley N, Guston DH, et al. Knowledge systems for sustainable development. Proc Natl Acad Sci U S A. 2003;100(14):8086–8091. doi: 10.1073/pnas.1231332100. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54.Di Giulio A, Defila R. Lessons from “Real-world Laboratories” about Transdisciplinary Projects, Transformative Research and Participation. https://i2insights.org/2018/03/22/lessons-from-real-world-laboratories/. Accessed 15 Dec 2018.
  • 55.Iedema R, Merrick E, Piper D, Britton K, Gray J, Verma R, et al. Codesigning as a discursive practice in emergency health services: the architecture of deliberation. J Appl Behav Sci. 2010;46(1):73–91. [Google Scholar]
  • 56.Martin S. Co-production of social research: strategies for engaged scholarship. Public Money Manag. 2010;30(4):211–218. [Google Scholar]
  • 57.Oliver K, Faul MV. Networks and network analysis in evidence, policy and practice. Evid Policy. 2018;14(3):369–379. [Google Scholar]
  • 58.Rycroft-Malone J, Seers K, Chandler J, Hawkes CA, Crichton N, Allen C, et al. The role of evidence, context, and facilitation in an implementation trial: Implications for the development of the PARIHS framework. Implement Sci. 2013;8(1):28. doi: 10.1186/1748-5908-8-28. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 59.Kislov R, Wilson PM, Knowles S, Boaden R. Learning from the emergence of NIHR Collaborations for Leadership in Applied Health Research and Care (CLAHRCs): a systematic review of evaluations. Implement Sci. 2018;13(1):111. doi: 10.1186/s13012-018-0805-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 60.Oborn E, Barrett M, Prince K, Racko G. Balancing exploration and exploitation in transferring research into practice: A comparison of five knowledge translation entity archetypes. Implement Sci. 2013;8(1):104. doi: 10.1186/1748-5908-8-104. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 61.Rushmer R, Shucksmith J. AskFuse origins: system barriers to providing the research that public health practice and policy partners say they need. Evid Policy. 2018;14(1):81–101. [Google Scholar]
  • 62.Canadian Institutes of Health Reseacrh. Evaluation of the Strategy for Patient-Oriented Research. http://www.cihr-irsc.gc.ca/e/49937.html. Accessed 18 Mar 2019
  • 63.Brett J, Staniszewska S, Mockford C, Herron-Marx S, Hughes J, Tysall C, et al. A systematic review of the impact of patient and public involvement on service users, researchers and communities. Patient. 2014;7(4):387–395. doi: 10.1007/s40271-014-0065-0. [DOI] [PubMed] [Google Scholar]
  • 64.Watermeyer R. Challenges for university engagement in the UK: towards a public academe? High Educ Q. 2011;65(4):386–410. [Google Scholar]
  • 65.Camden C, Shikako-Thomas K, Nguyen T, Al E. Engaging stakeholders in rehabilitation research: a scoping review of strategies used in partnerships and evaluation of impacts. Disabil Rehabil. 2015;37(15):1390–1400. doi: 10.3109/09638288.2014.963705. [DOI] [PubMed] [Google Scholar]
  • 66.Gagliardi A, Berta W, Kothari A, Boyko J, Urquhart R. Integrated knowledge translation (IKT) in health care: a scoping review. Implement Sci. 2016;11:38. doi: 10.1186/s13012-016-0399-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 67.Graffy EA. Enhancing policy-relevance without burning up or burning out: A strategy for scientists. In: Science into Policy: Water in the Public Realm: Proceedings, Specialty Conference, June 30-July 2, 1999, Bozeman, Montana. Herndon: American Water Resources Association; 1999. p. 293–8.
  • 68.Behrens TR, Gray DO. Unintended consequences of cooperative research: impact of industry sponsorship on climate for academic freedom and other graduate student outcome. Res Policy. 2001;30(2):179–199. [Google Scholar]
  • 69.Lidskog R, Sundqvist G. From consensus to credibility: new challenges for policy-relevant science. Innovation. 2004;17(3):205–226. [Google Scholar]
  • 70.Himmrich J. How should academics interact with policy makers? Lessons on building a long-term advocacy strategy. LSE Impact Blog. 2016; http://blogs.lse.ac.uk/impactofsocialsciences/2016/06/20/how-should-academics-interact-with-policy-makers-lessons-on-building-a-longterm-advocacy-strategy/. Accessed 10 Jul 2018.
  • 71.Zevallos Z. Protecting Activist Academics Against Public Harassment. Other Sociologist. 2017; https://othersociologist.com/2017/07/06/activist-academics-public-harassment/. Accessed 18 Mar 2019.
  • 72.Davey Smith G, Ebrahim S, Frankel S. How policy informs the evidence. BMJ. 2001;322(7280):184–185. doi: 10.1136/bmj.322.7280.184. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 73.Maynard A. Is public engagement really career limiting? Times High Education. 2015; https://www.timeshighereducation.com/blog/public-engagement-really-career-limiting. Accessed 18 Mar 2019.
  • 74.Kothari Uma. “Power, Knowledge and Social Control in Participatory Development”. In: Bill Cooke and Uma Kothari. Participation: The New Tyranny? London: Zed Books; 2001. pp. 142-143
  • 75.Spano R. Potential sources of observer bias in police observational data. Soc Sci Res. 2005;34(3):591–617. [Google Scholar]
  • 76.Kanuha VK. “Being” native versus “going native”: conducting social work research as an insider. Soc Work. 2000;45(5):439–447. doi: 10.1093/sw/45.5.439. [DOI] [PubMed] [Google Scholar]
  • 77.Bell C, Encel S. Inside the Whale: Ten Personal Accounts of Social Research. Oxford: Pergamon Press; 1978. [Google Scholar]
  • 78.Aarts N, Leeuwis C. Participation and power: reflections on the role of government in land use planning and rural development. J Agric Educ Ext. 2010;16(2):131–145. [Google Scholar]
  • 79.Cooke B, Kothari U. Participation: The New Tyranny? London: Zed Books; 2001.
  • 80.Gluckman P. Policy: the art of science advice to government. Nature. 2014;507(7491):163–165. doi: 10.1038/507163a. [DOI] [PubMed] [Google Scholar]
  • 81.DuMont K. Reframing Evidence-Based Policy to Align with the Evidence: William T. Grant Foundation; 2019. http://wtgrantfoundation.org/digest/reframing-evidence-based-policy-to-align-with-the-evidence. Accessed 28 Jan 2019.
  • 82.Prainsack B. The “we” in the “me”: solidarity and health care in the era of personalized medicine. Sci Technol Hum Values. 2018;43(1):21–44. [Google Scholar]
  • 83.Callard F, Rose D, Wykes T. Close to the bench as well as at the bedside: involving service users in all phases of translational research. Health Expect. 2012;15(4):389–400. doi: 10.1111/j.1369-7625.2011.00681.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 84.Callard F, Fitzgerald D. Rethinking Interdisciplinarity Across the Social Sciences and Neurosciences. Wellcome Trust–Funded Monographs and Book Chapters. Basingstoke: Palgrave Macmillan; 2015. [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

Not applicable.


Articles from Health Research Policy and Systems are provided here courtesy of BMC

RESOURCES