Patient involvement in research programming and implementation

Abstract

Background

The Dialogue Model for research agenda‐setting, involving multiple stakeholders including patients, was developed and validated in the Netherlands. However, there is little insight into whether and how patient involvement is sustained during the programming and implementation of research agendas.

Aim

To understand how the Dialogue Model can be optimised by focusing on programming and implementation, in order to stimulate the inclusion of (the perspectives of) patients in research.

Methods

A responsive evaluation of the programming and implementation phases of nine agenda‐setting projects that had used the Dialogue Model for agenda‐setting was conducted. Fifty‐four semi‐structured interviews were held with different stakeholders (patients, researchers, funding agencies). Three focus groups with patients, funding agencies and researchers (16 participants) were organized to validate the findings.

Results

Patient involvement in programming and implementation of the research agendas was limited. This was partly related to poor programming and implementation, partly to pitfalls in earlier phases of the agenda‐setting. Optimization of the Dialogue Model is possible by attending to the nature of the agenda and its intended use in earlier phases. Attention should also be given to the ambassadors and intended users of agenda topics. Support is needed during programming and implementation to organize patient involvement and adapt organizational structures like review procedures. In all phases the attitude to patient involvement, stakeholder participation, especially of researchers, and formal and informal relationships between parties need to be addressed to build a strong relationship with a shared goal.

Conclusion

Patient involvement in agenda‐setting is not automatically followed by patient involvement in programming and implementation. More attention should be paid, in earlier stages, to the attitude and engagement of researchers and funding agencies.

Introduction

Researchers and funding agencies have, for a long time, been the ones to set research agendas for medical and health research, while the research process itself was mainly controlled by researchers. Two decades ago this became part of a public debate in various Western countries; the awareness of patients’ rights increased. The history of public involvement in research (agenda‐setting) goes partly back to the campaigning of the disability and advocacy movements for more control over the process of knowledge production.1 Transitions in society, including consumerism, public accountability and democratisation of science, further prompted disputes over the benefits and pitfalls of patient involvement, including the validity of experiential knowledge.2, 3, 4, 5, 6, 7, 8, 9

In the field of research agenda‐setting, also referred to as priority setting, various models and strategies have been developed since then, ranging from reasonably well structured approaches, from a one‐off consultation to regular advice from beginning to end.1, 2 In the UK, for instance, the James Lind Alliance3 has proposed a number of methods to engage patients, carers and clinicians in dialogue about uncertainties in medical treatment. The Delphi Method, originally used for expert consensus, has also been translated to agenda setting.4, 5 The Nominal Group Technique, also used in the UK, is a combination of the Delphi Method and focus group discussions to identify, discuss and prioritize research topics.6

In The Netherlands public involvement in research agenda‐setting became an issue after 2000. Several local charity/government funding agencies, in some occasions after pressure from local patient organizations, started to initiate (and fund) projects in which patients were engaged in agenda setting. These projects were facilitated by academic departments in the field of social sciences who collaborated in the development and validation of the Dialogue Model.7 The model operationalizes consultation of and collaboration among various stakeholders, and is grounded in the notion that involvement is an interactive process between stakeholders. Since its inception, the Dialogue Model has been used on fourteen different occasions.8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24

What all these agenda setting methodologies have in common is the belief that patients should be involved in the actual agenda setting, and that their involvement should be sustained in the programming and implementation phases. However, the systematic evaluation of patient involvement in these phases is lacking. Patients have been engaged in agenda‐setting,7 but it is unclear whether they are involved in later phases.25 Nor do we know under what circumstances and how the research agenda set by patients has actually been programmed and implemented. We also lack studies that reveal whether (limited) patient involvement and uptake of patient priorities is a matter of (poor) programming, or (poor) planning in earlier phases of the agenda setting process. Little is therefore known about the factors influencing patient involvement in programming and implementation.

The aim of this article is to contribute towards the knowledge gap concerning patient involvement in the phases of programming and implementation of agenda setting. The article specifically focuses on optimizing the Dialogue Model in order to sustain patient involvement as well as to provide tools to improve the programming and implementation of patients’ priorities. To this end, a responsive evaluation of nine agenda‐setting projects, which involved patients and used the Dialogue Model, has been conducted. The findings from this study are situated in The Netherlands and therefore context laden, but we expect that lessons learned can be transferred to other contexts.

Dialogue Model

The Dialogue Model is a multi‐phased participatory approach. The Model has been developed within academia in order to facilitate the involvement of patients in research agenda‐setting.7 It is based on the methodology of Responsive Evaluation26, 27, 28, 29, 30, 31, 32 and the Interactive Learning and Action (ILA) approach, which enables end‐users to have a role in decision‐making of innovation processes in various societal domains.33 The aim of the Dialogue Model is to create equal partnership among patients and other stakeholders (e.g. health care professionals, researchers) over the agenda‐setting process. Dialogue is seen as an ethical and fruitful way for agenda‐setting as it helps professionals understand the societal impact of their research interests, and integrates patients’ experiential knowledge and voice.22, 30 Dialogue fosters a mutual learning process which leads to a new shared perspective that is acceptable and recognizable for all involved.17, 27 Dialogue potentially stimulates the innovation of ideas and a sense of co‐ownership.14, 19, 28, 29

The Dialogue Model acknowledges power differentials and assumes that patients first need the safe environment of their own group to develop their own voice; patients’ priorities are not given, and patients require deliberation if they are to become aware of their priorities.22, 30 This deliberation equips patients for dialogue with professionals.22

There are four phases to the Dialogue Model: exploration, consultation, prioritization, and integration. They lead to an integrated and shared research agenda, followed by two phases of programming and implementation:

1. Exploration: the project team is established, key stakeholders are identified, an initial list of issues, perspectives and ideas of patients and other stakeholders is drawn up, and the social conditions for collaboration are created.

2. Consultation: the different stakeholder groups are consulted separately to develop a list of research topics from the perspective of each stakeholder group.

3. Prioritization: the stakeholders prioritize the research topics identified in the previous phase.

4. Integration: the prioritized research topics of each stakeholder group are integrated into one integral research agenda.

5. Programming: the integral research agenda is translated into a funding programme or plan.

6. Implementation: the research agenda is actively used, action is taken and results are evaluated.

The Dialogue Model uses a mix of methods, including interviews and focus groups in the consultation phase, and questionnaires or the Delphi Method in the prioritization phase. In the integration phase, dialogue is used to integrate the different perspectives and encourage mutual learning. An external facilitator creates the conditions for dialogue, stimulates mutual learning and, if necessary, acts as a mediator.

Methods

This article is based on the responsive evaluation of nine agenda‐setting processes carried out along Dialogue Model in the Netherlands since 2003 (Table 1). We followed a purposeful sampling strategy, and selected processes that varied in terms of: (i) type of funding agency (governmental; charity); (ii) number of stakeholder groups involved (at least patients); and (iii) disease‐specific research agendas. The agenda‐setting processes should also have been completed at least 1 year ago, otherwise there would have been insufficient time for programming and implementation. The following agenda‐setting processes were included: spinal cord injury; asthma and COPD (and revision); diabetes; neuromuscular diseases; renal failure; congenital heart disease; burns, and people with intellectual disabilities. The evaluated projects used external facilitators from academic institutes for the agenda setting. No external facilitator was present for programming and implementation. The evaluation took place between May 2010 and April 2012 and was carried out by five academic researchers (author team).

Table 1.

Overview of research agenda setting projects which used the Dialogue Model

Disease	Funding agency	Initiator	Consulted	Programmed	Role patients in implementation
Spinal cord injuries (April 2003–April 2004)	ZonMw	ZonMw	Patients	No, Though, one research topic of the agenda received funding	Within ZonMw patient involvement activities have been organised, e.g. patient partners in funded research, patient reviewers, patients in advisory committees
Asthma/COPD (August 2003–June 2004)	Lung Foundation and ZonMw	Lung Foundation	Patients, health care professionals and researchers	Yes, one‐on‐one translation, resulting in funding program active from 2005 to 2008	Reviewing of research proposals by patient reviewers
Revision Asthma/COPD and rare lung diseases (March 2009–September 2009)	Lung Foundation	Lung Foundation	Patients, health care professionals and researchers	Yes, one‐on‐one translation, resulting in new funding program active from 2009 to 2013	Reviewing of research proposals by patient reviewers
Renal failure (December 2005–May 2006)	Kidney Fund	Kidney Fund and renal patient organization	Patients	Yes, integration of research themes in their policy and program structure	None
Intellectual disabilities (December 2005–September 2006)	ZonMw	ZonMw	Patients	No	Within ZonMw patient involvement activities have been organised, e.g. patient partners in funded research, patient reviewers, patients in advisory committees
Diabetes (March 2006–July 2006)	Dutch Diabetes Foundation	Dutch Diabetes Foundation	Patients	Yes, one‐on‐one translation, resulting in funding program ‘Voice of the patient’ (Stem van de patient)	‘Reviewing committee’ (in which patient representatives are included) assesses relevance of submitted research for patients
Burns (February 2006–February 2007)	Dutch Burns Foundation	Dutch Burns Foundation	Patients, health care professionals and researchers	Yes, one‐on‐one translation in the funding program of the foundation	None
Neuromuscular diseases (April 2006–December 2007)	ZonMw	Patient organization neuromuscular diseases and Dutch Neuromuscular Research Centre	Patients and researchers	No, but, one research topic of the agenda received funding	Within ZonMw patient involvement activities have been organised, e.g. patient partners in funded research, patient reviewers, patients in advisory committees
Congenital heart diseases (June 2007–October 2007)	Heart Foundation	Heart Foundation	Patients, health care professionals and researchers	Yes, In 2008 a call for research proposals has been financed on research topics of the agenda. In 2012 new call will be organized	‘Selection committee’ (in which one patient representative is included) assesses research proposals

Design

Responsive evaluation is based on a constructivist theory, and assumes that all human beings endow meaning to their experiences.26, 27, 28, 29, 30, 31, 32 Construction is influenced by personal background, but outcomes of a socio‐cultural process are also influenced by socio‐structural and stakeholder positions. Together, the various complementary stakeholder perspectives will lead to a more informed understanding of what is evaluated. Within responsive evaluation the relation between science, policy and activists is understood as a complex interrelation of many stakeholders and contextual factors. In order to generate a holistic understanding of this complex interaction and the meanings endowed to these processes responsive evaluation generates multiple stakeholders perspectives located in a particular time‐and‐place constellation. The evaluation consisted of four, iterative phases, and results from previous phases were input for later phases. In the first, exploratory phase the nine agenda‐setting projects (phases 1–4 of the Dialogue Model) were described using desk study and eight interviews. In the second phase semi‐structured interviews were held with stakeholders involved in the agenda‐setting projects and/or subsequent programming and implementation phases (patients, policymakers, researchers, project leaders) to gain a detailed understanding of their experiences, the consequences and pitfalls of programming and implementation and the role of patients. Results were validated in the third phase through three focus groups (patients, funding agencies, researchers). Also improvements for the Dialogue Model were identified. Results were disseminated in the final phase at an invitational conference for participants and external audiences. A steering group of six people from science, research councils and patient organizations followed the study with a critical eye.

Sample, recruitment and participants

The sample is based on maximum variation. We wanted to involve input from as many stakeholders as possible in the research agenda‐setting processes. Inclusion criteria for the interviews were: (i) being involved in one of the nine agenda‐setting projects, and/or (ii) having knowledge of the phases after the agenda‐setting. The main inclusion criteria for the focus groups were: (i) not having taken part in an interview; (ii) affiliation with a patient organization, health funding agency or research; (iii) having experience with, knowledge of and interest in agenda setting processes. An overview of the 54 respondents is given in Table 2.

Table 2.

Consultation overview

Research agenda	No. of explorative interviews	No. of in‐depth interviews
Spinal cord injuries	Patient organization (p): 1	Patient organization (np): 1 Governmental funding agency: 1 Facilitator research agenda: 1
Asthma/COPD (including update and extension)	Health specific funding agency: 1	Health specific funding agency: 1 Researcher (members scientific committee): 2 Patient organization (p): 1 Patient organization (np):1
Renal failure	Health specific funding agency: 1	Patient organization (np): 1 Patient organization (p): 1 Facilitator research agenda: 2
Intellectual disabilities	Governmental funding agency: 1	Governmental funding agency: 2 Facilitator research agenda: 1
Diabetes	Health specific funding agency: 1	Health specific funding agency: 1 Patient organization (np): 1
Burns	Health specific funding agency: 1	Health specific funding agency: 2 Facilitator research agenda: 2 Patient organization (p): 4 Researcher: 6
Neuromuscular diseases	Patient organization (np): 1	Governmental funding agency: 2 Facilitator research agenda: 2 Researcher: 2
Congenital heart diseases	Health specific funding agency: 1	Facilitator research agenda: 1 Health specific funding agency: 1 Patient organization (np): 2 Researcher: 2
General	0	Governmental funding agency: 2 Facilitator research agenda: 1
Total	8	46

‘Patient organization (p)’ refers to respondents who are patients themselves and are aligned to a patient organization; ‘patient organization (np)’ refers to respondents who are no patients themselves, but who work as a policy maker for a patient organization.

The research team recruited the participants for the interviews and the focus groups, who were identified during the desk study and exploratory interviews. All participants, except one, agreed to participate. She felt she would not be able to provide any relevant information.

Data collection

A topic list was developed for semi‐structured interviews. The topic list covered the following topics: (i) experiences with initiating and formulating the research agenda (first four Dialogue Model phases); (ii) verification of the data obtained by the desk study; (iii) experiences with programming and implementation; and (iv) barriers and facilitators for patient involvement in programming and implementation. The interviews were conducted by the second and third authors, who had not been involved in the agenda‐setting projects. The interviews were held in a location indicated by the respondent, usually at work or home and lasted approximately 1.5 to 2 h. The focus groups covered three parts: (i) presentation of preliminary results; (ii) inventory of current programming and implementation activities, and (iii) discussion and reflection on practical guidelines, based on the preliminary results. The focus groups were moderated by the second, third and fourth authors, held at a central location and lasted 2.5 h.

Data analysis

The interviews and focus groups were audio‐recorded after consent, and transcribed in full. The data analysis was iterative, i.e. the data were analysed during the process and the outcomes steered the rest of the process. The content analysis was open and oriented towards the inductive analysis of themes per phase in the agenda‐setting process. The transcripts were read line by line and labels were attached to text fragments. The codes with labels were related to each other to form clusters. One cluster involved the factors that foster or impede agenda programming and implementation. Another cluster contained issues and questions regarding the Dialogue Model and how to improve it.

Quality procedures and ethical considerations

Participants received a summary of the interview transcript to check the validity of findings (individual member checks). Group member checks were run with focus group participants. All participants agreed on the summaries; only a few participants added small details or made minor changes. Three participants indicated that interview data should be considered strictly confidential; they agreed that the interview data were taken along in the analysis, but indicated that no quotes were allowed to be used. This was due to delicate relations with other organizations. The (intermediate) results were regularly presented to and discussed with the steering committee. The combination of interviews, desk study and focus groups enabled us to enrich our dataset and to follow up on any discrepancies in the data (triangulation). Theoretical saturation was reached, meaning that repetition of hypotheses formed in the process were robust and firmly established by the data. The team checked the codes and labels in the data analysis through discussion until consensus was reached (co‐checking; inter‐rater reliability).

Results

The nine agenda‐setting processes differ in many aspects, as is shown in Table 1. All processes were financed by a funding agency; three agenda setting processes were financed by the government funding agency ZonMw (spinal cord injuries, intellectual disabilities and neuromuscular diseases), the other agenda setting processes were financed by disease‐specific charity foundations. The agenda setting was in most processes initiated by the financing funding agency. For the agendas renal failure and neuromuscular diseases the patient organization and a research institute (agenda for neuromuscular diseases) were main initiators. In all agenda‐setting processes partnerships were formed with the specific funding agency and patient organization, and for the shared research agendas there was close contact with related research communities. There was no involvement of pharmaceutical industry.

Furthermore, the agenda‐setting processes differed in the way the agenda‐setting (Dialogue Model 1–4) had taken shape. Some projects only identified patients’ priorities for research (spinal cord injuries; renal failure; intellectual disabilities). The outcome in the other projects was a shared research agenda by patients, researchers and health professionals. In three projects patient research partners were engaged in the research team to collect and analyse data (spinal cord injuries; renal failure; intellectual disabilities).32 In three other projects, one or two patients were involved in the project team. They advised on the design and implementation of the agenda‐setting project (diabetes; asthma/COPD and revision).

Follow‐up after agenda setting: two patterns

When comparing the nine processes two follow‐up patterns emerged:

• 1 No formal programming, some patient involvement: This pattern was found in three of the nine agenda‐setting processes, all financed by ZonMw, the Dutch Council for Medical Research. Spinal cord injury is a good exemplar. This agenda‐setting project was the first initiated by ZonMw in 2003 at the time when a new Rehabilitation programme was formed. The local Spinal Cord Injury association had mixed feelings, some were enthusiastic, others shared doubts about the possibility to influence the new program. The facilitating department formed a research team with a woman with spinal cord injury, and along‐the‐way trust was build up among a vast group of patients who became owners of the agenda.8 The agenda was an eye‐opener for some researchers, others felt reinforced by the topics listed by patients, but at the time when the patients’ agenda was completed the new Rehabilitation programme was already established. One patient priority received funding.

Although there was no formal programming in these projects together these projects did encourage patient involvement in ZonMw, for example the inclusion of patient representatives in programme committees. Over the years the inclusion of patients in programmes has increased from 10 in the beginning years to 60 in 2010. In addition an annual meeting for patients at ZonMw was initiated and is still being continued. Furthermore, a specific programme on Patient and Citizen Participation was initiated (2009‐2013).

• 2 Formal programming and implementation, limited patient involvement: This pattern was found in six agenda‐setting processes. These concerned the projects financed by charity foundations. The agenda was sometimes translated into one specific programme (e.g. Burns Foundation, Diabetes Fund). In other instances, the agenda was found to be relevant for all programmes, and diffused throughout a whole set of programmes (e.g. Kidney Foundation). Patient involvement activities for most funding agencies were restricted to the appointment of patient reviewers or the inclusion of a patient representative in an advisory committee.

A good exemplar of this pattern is the case of asthma/COPD. In 2004, the Dutch Lung Foundation commissioned an agenda‐setting project and identified and prioritized a long list of research topics put forward by patients, researchers and health professionals. The top ten themes were adopted by the Dutch Lung Foundation. Two years later the Netherlands Respiratory Society (NRS) for researchers was set up. Patient involvement was encouraged in the NRS task forces that identify knowledge gaps in research. The review of research applications for grants presented to the Lung Foundation was completed by the Scientific Advisory Board and a patient pool. Patient pool criteria include clarity and relevance for the health condition of patients. The patient review was given full consideration in the overall review process.34 This pattern of formal programming and implementation can partly be explained by the relatively small‐scale and disease‐specific orientation of the charity funds compared to the many policy layers and broader scope of the governmental fund.

Factors influencing programming and implementation

A wide range of factors that influence programming and implementation was identified (Table 3). A detailed description of these factors can be found elsewhere.35 For the purpose of this article we give an overview, and focus particularly on the factors considered relevant for optimizing the Dialogue Model. In Table 4 quotes of consulted respondents can be found, which are referred to in the text.

Table 3.

Factors influencing programming and implementation of research agendas including the perspectives of patients

Context
Prior attitude	Stakeholder participation	Time and timing	Organization

Process of programming and implementation
Nature of research agenda	Research proposal assessment

Table 4.

Quotes of respondents

Quotes A	‘You should lose the thought that patients have nothing to tell. (…) They [patients] are not stupid!’ (policy maker funding agency, acting as change agent) ‘It [the involvement of patients in agenda‐setting] completely depends on individuals’. (researcher)
Quotes B	‘One patient was member of the committee. This person had useful comments and questions, but a long‐term vision on research was discussed too little. That hampered the discussion. Patients are positive when there is a very concrete outcome, for example a questionnaire measuring the burden of disease. The long‐term vision should include this type of short‐term topics’. (member of a Scientific Advisory Committee) ‘In the scientific committee 20 to 25 people voted for, because they thought it was a good research proposal. As a patient I voted against that proposal, but that doesn't have any effect. They voted and I had a say, but they didn't count my input […] One vote doesn't have any effect’. (patient representative).
Quote C	‘We supposed to integrate themes [based on the outcomes of the research agenda] in the research programme. This did not happen. That could also be an improvement. I think this did not happen, because the phases [of the research agenda setting project and the development of the research programme] were not running parallel. (…) It [the research agenda setting project] was somewhat delayed and we had to finish the programme proposal slightly earlier’. (policy maker funding agency)
Quotes D	‘We try to involve a minimum of employees in the agenda setting project. But if you enter such a trajectory, it is very labour intensive’. (policy maker funding agency) ‘If it is ending up in the drawers, that will absolutely not be a matter of unwillingness. It rather relates to the translation [of the research topics into a programme]’. (staff funding agency)
Quotes E	‘You should keep in mind that patient organizations do not receive subsidies. They do not have money. So, a whole bunch of stuff is only possible on project basis’. (patient representative patient organization) ‘I assume that there were plans [for implementation], otherwise we wouldn't have participated in the agenda‐setting. So I guess it is useful to join in, but then you face the fact that you do not have enough people [to carry out those plans]’ (staff patient organization)
Quotes F	‘When you formulate detailed research topics, things [research topics] become fragmented. The question is whether you want that. We have a four year policy mission. We have a horizon that is very distant, and that is where we are heading. All we are doing should contribute to that [the horizon]’. (policy maker funding agency) ‘It was a luminous idea and ultimately we made it very clear in our first agenda‐setting project that we wanted to gain more insight in the critical moments patients’ experienced. In their lives and medical history, because we expected that those are the moments when questions arise, and also issues for policy‐making’ (staff funding agency)
Quotes G	‘You should take care of the patient volunteers. If you ask people too much in too short a time, then people tune out. Not for just one moment, but then you lose them for your organization. When you, as a patient organization, would like to be an equal partner, then you probably need at some points a slightly modified schedule’. (patient reviewer) ‘Reviewing research proposals sometimes resulted in problems. Especially, when it concerned biomedical research. Researchers often use complicated jargon for patients. As a result the quality of the reviews of patients had a wide range’. (researcher)

Factors related to context: Four contextual factors have been identified:

• Prior attitude: The broader the degree of commitment for a shared research agenda among relevant stakeholders, the more supple the translation was of the agenda into a programme. Change agents who were dedicated to patient involvement were able to make a difference. For instance, in the Dutch Lung Foundation one person was able to promote patient involvement. Such change agents positively stimulated a higher commitment for patient involvement in programming and implementation (See: Quotes A). Existing partnerships between funding agencies, patient organizations and the research community were also found to be helpful for programming and implementation.

• Stakeholder participation: Crucial for programming and implementation was the involvement of researchers and their willingness to collaborate in multidisciplinary research projects and to focus on topics not directly related to their research profile. Furthermore, when researchers felt co‐owners of the agenda, there was a greater chance of implementation. The support and commitment of researchers in decision‐making positions, like members of scientific advisory boards, turned out to be crucial. If these boards were critical towards the involvement of patients this often hampered the programming and implementation (See: Quotes B).

• Time and timing: A misfit between the timelines of agenda‐setting and programming, like in the example of spinal cord injuries, hampered the uptake of agenda topics (See: Quote C)

• Organization: Many funding agencies did not have sufficient resources to support researchers to develop proposals in line with priorities set by patients. This struggle of funding agencies is emphasized in quotes D. Moreover, adaptations of internal structures and procedures were often found necessary for the research agenda to be implemented. This involved also effort from patient organizations, who not always had the resources (See: Quotes E).

Factors related to the process of programming and implementation:

In addition, we identified factors relating to the process of programming and implementation, including the nature of the agenda and research proposal assessment. We found that a broader definition of the agenda created more opportunities for multidisciplinary collaboration. However, when research topics became more specific, the chance increased that research proposals were more needs‐oriented. It appeared a challenge for funding agencies to find the right balance between a broad formulation of research topics to engage researchers, and specific research topics reflecting the priorities of patients (See: Quotes F). Furthermore, factors relating to research proposal assessment were identified. When patients reviewed research proposals this fostered the implementation of the patient agenda. A patient reviewer indicated that patients should be supported in reviewing by having assessment criteria specifically drawn up for that purpose. However, such support was often lacking (See: Quotes G).

Implications for the Dialogue Model

The above factors indicate that both contextual factors and factors relating to the nature of the agenda and review procedures could be steered in the direction of more patient involvement in programming and implementation. In this section we discuss implications for the Dialogue Model.

We found that in most instances participants and commissioners had, beforehand, thought little about how to programme and implement the research agenda. The involved organizations were initially not familiar with patient involvement in agenda setting. They were enthusiastic, but did not know what to expect and what aspects required specific attention. One important lesson for the Dialogue Model is therefore to pay attention to programming and implementation right from the start of agenda setting. Stakeholders in phase 1 (Exploration) should deliberate and determine how the research agenda will be used in phase 5 (Programming). In these deliberations the nature of the agenda should be taken into consideration. Topics might be listed one‐by‐one in a call, and/or agenda topics can be used in a more general sense to steer the programme. If the options are more explicit and discussed at the beginning we expect fewer agendas will remain unutilized. In addition, it is crucial that timelines and decision‐making procedures are matched to connect the agenda to the programming phase.

Balancing between a general and specific patient agenda appeared to be difficult for funding agencies. Therefore the nature of the agenda should again be subject of deliberation in phase 4 (Integration) including the pros and cons of a general versus detailed agenda for both patients and researchers. In this phase parties should be identified to pick up research topics. A particular group of researchers sometimes needs to be addressed to generate an interest in agenda issues. To reach intended users not everything needs to be carried out by the commissioner. Patient organizations and researchers can also act as ambassadors and lobby for certain topics. This also emphasizes the importance of good relationships among stakeholders.

The first four phases of the Dialogue Model were extensively facilitated by an academic institute, while programming and implementation activities were mainly the responsibility of the funding agencies. For successful programming and implementation more support should be provided to funding agencies in these phases by the facilitating academic institutes. We found the organization of patient involvement required a lot of efforts and deliberations. Stakeholders should therefore be engaged in discussion forums on the composition of their scientific advisory board, the review procedure, review criteria, and patient reviewers in early stages. Respondents made clear that researchers often lacked the expertise to involve patients in their research. Therefore, the Dialogue Model should stimulate communities of practice and training courses to increase researchers’ knowledge of patient involvement.

In all phases the attitude towards patient involvement in research deserved attention as there was a considerable amount of resistance and ignorance. It therefore seems fruitful to engage funding agencies and the research community earlier in discussions on patient involvement in research to raise their awareness and increase understanding of researchers’ motives for and against patient involvement, their ideals of (good) involvement, their current practice of patient involvement and the challenges experienced. It is clear enough that most professionals are willing to embrace patient involvement as long as it is an ideal. However, the real challenge comes when they need to put it into practice. Starting in phase 2 (Consultation) with a discussion based on their own experiences seems to be a good way to involve researchers in thinking about patient involvement.

Attention should also be given in all phases to issues related to stakeholder participation to foster broad commitment and co‐ownership. In phases 1‐4 of the Dialogue Model one should already more deliberately search for participant involvement in scientific advisory boards as this will increase their sense of ownership. It is also crucial that connections between patients engaged in agenda‐setting and their decision‐makers in the advocacy organizations are cherished. The Dialogue Model includes an inventory of researchers’ agenda in phase 2 (Consultation). In addition, researchers and their organizations should be informed about the agenda‐setting process and its outcomes, e.g. by attending research conferences and presenting findings.

Finally, in all phases the informal and formal relationships among the organizations involved requires focused attention, as described in the following section. A strong and enduring relationship with shared goals should be built. This will foster and sustain collaboration among parties beyond the agenda‐setting process. While these relationships tend to be vertical, with the funding agencies and researchers directing the agenda, the relationship with the patient organizations can, with time, become more horizontal.

Guidelines for funding agencies and patient organizations

Practical guidelines were developed (Table 5) to help patient organizations and funding agencies to include the patient perspective and patient involvement in the programming and implementation phases.

Table 5.

Guidelines for funding agencies and patient organizations

Starting position
Open attitude towards
Willingness towards
Adequate resources
Expertise
Collaboration between fund and patient organization with middle and long‐term goals

Funding agency	Patient organization
Appoint policy maker for inclusion patient perspective	Appoint patient representative to work with fund, and to provide patient perspective

Programming
Include patient representatives in program committee	Participate in program committee
Identify priorities based on research agenda	Participate in identification of priorities based on research agenda
Write scientific article on outcomes research agenda	Co‐author in scientific article on outcomes research agenda
Communicate the funding program and inclusion of patient perspective	Review and adjust program text

Implementation
Appointment patient reviewers	Recruit patient reviewers
Set up patient assessment criteria for reviewing	Provide input and feedback on patient assessment criteria for reviewing
Involve patients in writing and disseminating call for proposals	Support patient reviewers
Pay deliberate attention to patient issues
Request lay summary in proposals
Inform and train researchers about working with patient research partners	Organize pool of patient research partners
Evaluate patient involvement in process of programming and implementation	Evaluate patient involvement in programming and implementation from patient perspective
Disseminate evaluation to broad public	Disseminate evaluation to patients community

Organizations with an open attitude and a willingness to adopt patient involvement and with adequate resources and expertise are in a good starting position for complex, dynamic activities like patient involvement in agenda‐setting. Patient organizations and funding agencies should also consider building a relationship with each other, and set middle and long‐term goals together. Organizations lacking abovementioned conditions may first invest in optimizing them, and/or should start with small initiatives regarding patient involvement, and build on these initiatives as their position continues to strengthen.

Discussion

Research agenda‐setting with and for patients is gaining in popularity. Several formal methodologies have been adopted3, 4, 6, 36, 37, 38, 39, 40 including the Dialogue Model.7 Scientific literature on the programming and implementation of agendas is still rare. This evaluation study is among the first to assess the influence of agenda‐setting through the Dialogue Model on the programming and implementation phases.

We conclude on the basis of nine agenda setting processes that patient involvement in programming and implementation and the uptake of the patient priorities did not happen automatically, and was influenced by a wide range of factors relating to the context and the process of programming and implementation. Six out of the nine agenda‐setting processes initiated by disease‐specific charity funds led to formal programming and implementation with limited patient involvement. The other three agenda setting processes, granted by a governmental fund, involved more participatory activities, but less formal programming and implementation. The pattern of no formal programming, but stimulation of patient involvement can be partly explained by the many policy layers of the governmental fund; the policy makers who commissioned the research agendas and were dedicated to patient involvement, were not necessarily directly involved in the related research programmes. It is likely that this pattern is also related to the broader scope of this governmental fund compared to disease‐specific charity funds. Further research is required to verify these explanations.

Many differences in programming and implementation were observed. A commonality was, however, the unfamiliarity of funding agencies with involving patients in agenda‐setting. Beforehand, commissioners did not consider what they wanted to do with the patient priorities and how to sustain patient involvement, resulting in limited results. Patient involvement activities in the evaluated projects were mainly restricted to a few activities, but we did observe attempts to improve the degree of patient involvement. Although this is far from ideal, there was a heightened awareness among participants that standard procedures and organizational routines needed to be adjusted in order to guarantee patient involvement. This is in line with the findings presented by O'Donnell (2004) about the involvement of patients in charity funds in the UK.41, 42 They report that many funding agencies (two‐thirds) somehow involved patients in research aspects with a wide variety of involvement activities, such as proposal development and review, patients as members in committees and dissemination of results, but that there were many differences in involvement activities.

Based on the above findings, suggestions have been put forward to optimize the Dialogue Model. Attention should first be given to the intended use of the agenda in phase 1 (Exploration). In phase 4 (Integration) more attention should be paid to potential users of the agenda. The relevance of advance assessment, the intended use and potential users of findings in decision‐making procedures is well‐known in the utilization literature.44, 45 Tailoring the agenda‐setting process to intended use and the needs of potential users will foster the uptake of the outcomes. Adjusting the timelines and procedures of agenda setting to the decision‐making processes of policymakers is also well‐known in the utilization literature.46 Furthermore, the research agenda should be formulated comprehensively enough to leave some room for reinterpretation by researchers,47, 48 and simultaneously be specific enough to preserve the patient perspective.

Deliberate attention needs to be paid in all phases of the Dialogue Model to the attitude towards patient involvement, stakeholder participation and formal and informal relationships. The Dialogue Model was originally grounded in the notion of partnership and dialogue. In order to balance power asymmetries between researchers and patients the model pays a lot of attention to generating a genuine patient agenda and empowerment in a safe environment.22, 30 Until now a lot of attention and time in those trajectories has been given to the least heard group. We should, however, bear in mind that researchers and funding agencies also need to be involved in those processes in order to change through it. This is in line with other literature emphasizing ‘Don't forget the professional’ in processes of patient involvement in research.49 Engaging these stakeholders actively in discussions on patient involvement early in the agenda‐setting process and keeping them informed about the outcomes help to build enduring partnerships. During the programming and implementation phases support is needed to involve patients in writing the programme, in the programme committee and in adjusting review procedures. Guidelines have been developed to assist funding agencies and patient organizations in the process.

Although the data in this article pertain to the Dutch context, and the experiences only involve agenda‐setting in line with the Dialogue Model, we believe readers might benefit from what is presented here, and transfer some ideas to their own context, based on patterns they recognize or consider relevant. Moreover, international literature describes corresponding challenges in research agenda‐setting and follow‐up; in most approaches for agenda‐setting, patient involvement is limited to the actual agenda‐setting and there is little understanding of what happens next1, 2 and/or how to shape patient involvement activities in follow‐up phases.41 The objective of the presented study was to add to that emerging knowledge base, and open new horizons for patient involvement in programming and implementation of patient‐driven research agendas.

Funding

This work was supported by the Netherlands Organization for Health Research and Development, ZonMw (415011003).

Conflicts of interest

The authors declare that they have no competing interests.