Abstract
Background
Showing how engagement adds value for all stakeholders can be an effective motivator for broader implementation of patient engagement. However, it is unclear what methods can best be used to evaluate patient engagement. This paper is focused on ways to evaluate patient engagement at three decision‐making points in the medicines research and development process: research priority setting, clinical trial design and early dialogues with regulators and health technology assessment bodies.
Objective
Our aim was to review the literature on monitoring and evaluation of patient engagement, with a focus on indicators and methods.
Search strategy and inclusion criteria
We undertook a scoping literature review using a systematic search, including academic and grey literature with a focus on evaluation approaches or outcomes associated with patient engagement. No date limits were applied other than a cut‐off of publications after July 2018.
Data extraction and synthesis
Data were extracted from 91 publications, coded and thematically analysed.
Main results
A total of 18 benefits and 5 costs of patient engagement were identified, mapped with 28 possible indicators for their evaluation. Several quantitative and qualitative methods were found for the evaluation of benefits and costs of patient engagement.
Discussion and conclusions
Currently available indicators and methods are of some use in measuring impact but are not sufficient to understand the pathway to impact, nor whether interaction between researchers and patients leads to change. We suggest that the impacts of patient engagement can best be determined not by applying single indicators, but a coherent set of measures.
Keywords: evaluation, framework, impact, literature review, medicines development, metrics, patient and public involvement, patient engagement, patient participation, research
1. INTRODUCTION
There is increasing consensus among stakeholders that patient engagement in research and development (R&D) of medicines provides benefits for patients, researchers, industry, regulatory bodies, payers and policy makers.1, 2, 3 The case for patient engagement is often presented in ethical and political terms referring to fairness, transparency and accountability.4, 5 Methodological arguments consider the experiential knowledge of patients—acquired by their personal experience of a condition—as valuable to improving the quality and relevance of the research.6, 7, 8 The inclusion of patients in decision making about the development of new innovative medicines is a substantial change, requiring time and (financial) commitments from researchers, industry and patients.2, 4 Despite efforts to promote and support patient engagement in research, the prevalence of patient engagement in medicines research and development remains low.9, 10 Patient engagement has not been fully embedded in the health research system, partly because it is not yet clear to all involved what the added value is.11 To address this need, an increasing number of studies aim to evaluate the impact of patient engagement, underscoring the growing interest in the “return on engagement,” or why it makes sense for patients, society and industry.2, 12
The perceived value of patient engagement practices can vary for different stakeholder groups, and the metrics of interest will therefore differ accordingly.13 For example, for researchers and industry partners it might be about evidence that patient engagement improves the quality and efficiency of research and the uptake of findings, whilst for patients it might be more about influencing the R&D agenda to develop medicines for unmet needs. Some argue that evidence is needed to justify the ‘business case’ for engagement. This could also help to establish a financial model to support engagement.2, 14, 15 Evaluation could also define the genuine value of patients’ contributions, contributing to valued rather than tokenistic inclusion for appearances’ sake.16 There is also some resistance; people are concerned about assessing impact too simplistically. Some question whether it is fair to evaluate the value of patient input in isolation, and not that of others such as key scientific leaders,12 not least because it may be the synergy of working in partnership that produces benefit.17 As mentioned by Staniszewska, it is important to recognize that “any form of measurement sits within a political or ideological context that cannot be ignored.”13 Nonetheless, there is a desire to assess the impact of patient engagement, to demonstrate better decision making, avoidance of previous errors and a contribution to continuous efficiency and quality improvement.15, 16, 18
Despite this perceived importance of assessing the return on patient engagement, little is known about “what” to evaluate, and even less about “how.”19, 20, 21 A number of researchers have tried to assess how patient engagement makes a difference.3, 5, 8, 12, 22, 23, 24, 25, 26, 27 Although there is no standardized way to assess the impact of patient engagement, very similar benefits, costs and challenges are reported in literature reviews.4, 17, 19, 20, 28, 29, 30, 31, 32 The current assessment of patient engagement is considered weak, partly because much of the evidence is mainly anecdotal17 and because methods used have not captured the complexity, context or mechanisms of change.17, 33 Previous studies have identified a number of gaps in the literature and identified challenges such as the delayed nature of impact, inconsistent terminology, absence of accepted criteria for judging the success or quality of research, no agreed evaluation methods or framework and few reliable measurement tools. The absence of a control group—identical research carried out without patient engagement—is problematic too, particularly in an area of science where direct comparison to an existing standard is routinely demanded.8, 16, 34, 35 It is argued that to build an evidence base, some level of consensus on measurable impacts is needed, whilst others state that the outcomes of engagement cannot easily be quantified.13, 30, 36 In sum, it remains unclear what methods can be best used to evaluate patient engagement.
To address the need for means of determining the “return on engagement,” the aim of this paper was to scope, review and summarize the literature on monitoring and evaluation of patient engagement. Many publications present useful guidance for conducting patient engagement and assessing the quality.37, 38, 39 Evaluation studies focus mainly on qualitative methods and only occasionally link to specific outcomes.12, 33, 40, 41 Therefore, this paper is focused on ways to evaluate patient engagement with both qualitative and explicitly quantitative methods.
This work is part of the PARADIGM project, a public‐private partnership that is developing ways to ensure that patients are always meaningfully involved in the development of medicines. The impact of patient engagement may differ at different points in the development of a medicine. Accordingly, PARADIGM focuses on three decision‐making points during R&D at which point integration of the patient perspective is considered likely to be valuable, specifically as part of research priority setting, design of clinical trials and at early dialogues with regulators and health technology assessment bodies. Each of these represents a point at which engagement can influence effective planning and implementation, and demonstrate impact on the final product.
2. METHODS
We undertook a scoping review of published academic and grey literature as recommended by Arksey and O’Malley, also drawing on Mays et al and Peters et al42, 43, 44 Scoping reviews are similar to systematic reviews in that they follow a structured search process; however, they are performed for different reasons.45 Our aim was not to answer a precise question addressing the effectiveness of a certain practice, as in a meta‐analysis, but to provide an overview of the breadth of the available literature about evaluating patient engagement.
Whilst the review is concerned with patient engagement at the three key decision‐making points, we used broader search limits to ensure capture of related publications in other areas of health research. One of the challenges was the variety of terminology. For example, the words “measure,” “metric” and “indicator” are often used interchangeably and their definitions may vary, if they are stated at all. Furthermore, the terms used for “patient engagement” differ globally. In this paper, we use the term patient engagement; in our search, we included terms such as public involvement, patient participation, community engagement and user involvement. In Table 1, we provide definitions of terms developed by the authors and as we used them in this review.
Table 1.
Concept | Description |
---|---|
Patient engagement | The effective and active collaboration of patients, patient advocates, patient representatives and/or carers in the processes and decisions within the medicines lifecycle, along with all other relevant stakeholders when appropriate1 |
Patient partner | A patient, patient advocate, patient representative and/or carer who contributes to any level of patient engagement activities; this can also be substituted for other terms such as patient contributor82 |
Research participant | A person who participates in human subject research, also called a subject, study participant or volunteer of an experiment or trial |
Society | Includes all members of the public and patients who use health‐care services |
Research priority setting | Any process aimed at constructing priorities or agendas for health research and medicines development, to raise awareness and change the way research funding is allocated |
Design of clinical trials | Any process aimed at the development or design of clinical trials for medicines development at any stage of that process. One example is changes made to inclusion and exclusion criteria for trial participants |
Early dialogues with regulators and Health Technology Assessment (HTA) bodies | Any process in which medical technology developers communicate with regulatory bodies and/or HTA bodies prior to health technology assessment. Early dialogue can happen only with regulators (eg scientific advice), jointly with regulators and HTA bodies (to discuss data requirements to support decision making on marketing authorization and reimbursement simultaneously) or only with HTA bodies (eg EUnetHTA multi‐HTA dialogues) |
Benefit | An advantage of engagement for research and development and stakeholders involved |
Costs and challenges | The expenditure and/or effort of engagement for research and development and the stakeholders involved |
Outcomes | Decisions made and things produced as a direct result of patient engagement practices. One example is changes made in the design of a clinical trial resulting in a more relevant and appropriate research protocol. Outcomes may lead to impact on research and development |
Impacts | Broader effect of outcomes, both positive and negative, of patient engagement. Impact may be direct or indirect, intended or unintended. For example, this may include study quality benefits such as improved recruitment and retention of study participants |
Value | The benefits of patient engagement (in relation to the direct and indirect costs) for individuals and organizations involved |
Monitoring | The formative evaluation of patient engagement practices in order to strengthen them |
Evaluation | The ‘systematic acquisition and assessment of information to provide useful feedback about …’ patient engagement practices.83 Summative evaluation examines the effects of patient engagement practices on various measures including outcomes, impact and cost‐benefit |
Criteria | Dimensions or parameters used for evaluation. These need to be translated into measurable entities called ‘indicators’ and indicators are measured with ‘metrics’ |
Indicator | Qualitative or quantitative measure that provides a means of expressing achievement of a goal or ascertaining the consequences of a specific change. Quantitative indicators are reported as numbers, such as rates of change and ratios. Qualitative indicators are reported as words, in statements, paragraphs and reports84 |
Metrics | Observations based on standardized data sources or agreed techniques for gathering information. Metrics could consist of an agreed set of quantitative and/or qualitative indicators to measure evaluation criteria, with a set of agreed methods/tools to collect this information |
Methods | Ways to collect information for monitoring and evaluating the outcomes and impact of patient engagement practices, for example quantitative, qualitative or mixed methods |
Tools | Instruments to collect information about patient engagement practices. For example, interview guides, questionnaires, log sheets and observation forms are all tools |
2.1. Search methods
Prior to the database search, we did a search to identify a tentative sample set of relevant studies for a snowballing exercise. Using broad key words, we searched Google Scholar for published articles and Google for grey literature. We also searched the Patient‐Centered Outcomes Research Institute (PCORI) database46 and the INVOLVE evidence library.47 A snowballing exercise using references and citations from these articles provided a starting set of publications that informed the protocol for the main review. This is recommended for the clarification of concepts and search terms when interrogating large, diverse fields of literature.48
Accordingly, with the assistance of a specialist librarian, we searched CINAHL, Embase, Medline, PsychINFO and PubMed databases for peer‐reviewed published literature. The following key words were used “patient engagement” combined with « AND» “research” « AND» “outcomes,” including a variation of terms combined with « OR». An overview of all search terms can be found in Table 2.
Table 2.
Patient engagement (title only) | Research (title only) | Outcomes (title/abstract only) |
---|---|---|
Patient participation [MeSH] | Comparative effectiveness research [MeSH] |
Outcome(s) Impact |
Patient engagement | Research | Measurement(s) |
Public engagement | Clinical trial | Metrics |
Client engagement | Study design | Framework(s) |
Community engagement | Trial design | Assessment |
Public participation | Research design | Criteria |
Patient participation | Health technology assessment | Indicator(s) |
Public involvement | Agenda setting | |
User involvement | ||
Client involvement | ||
Consumer involvement |
Grey literature was searched using the same terms; items recommended by consortium partners and external stakeholders were added manually, and reference lists of items included were searched for additional publications. All searches were conducted between 1 May 2018 and 31 July 2018. The search was limited to publications in English. We excluded articles that did not provide information on possible evaluation approaches or outcomes associated with patient or public engagement. No date limits were applied other than a cut‐off of publications after 31 July 2018. Following completion of the search, duplicated items were removed.
2.2. Study selection and data extraction
Two researchers (TF, LV) independently screened all items’ title and abstract. To ensure inter‐rater reliability, items were marked for inclusion or exclusion with each researcher's initials, discrepancies were discussed and consensus reached. Both researchers read all the selected items in full and followed up references for final inclusion. At this stage, further exclusions were made of items that did not include methods for evaluating outcomes and/or impact of patient engagement practices in health research or health technology assessment—discrepancies were discussed, and consensus agreed for final inclusion in the data extraction and analysis. Figure 1 demonstrates the number of articles identified, screened, selected and reviewed.
TF and LV developed a data extraction sheet to record relevant information from each item, including the publication year and focus, country of origin, methodology, patients involved as partners, use of a framework or model, definitions included, outcome and/or impact on research, benefits and costs per stakeholder group, measurement or evaluation methods suggested or applied. When available, context and process criteria were also included. Both researchers extracted data independently from 30% of the scientific articles and then compared their findings to agree the approach to data extraction. Thereafter, all the peer‐reviewed published papers’ data were extracted by LV and the grey literature by TF and LV.
2.3. Analysis
Data were thematically analysed following Braun and Clarke's approach.49 To achieve the summary, we coded data using the review question, aim and objective as included on the data extraction sheet. Codes used were benefits (B), costs and challenges (C), outcome or impact on research (O or I) and types of benefits or costs and challenges. Codes were then clustered into themes, which were agreed by TF and LV; themes were identified deductively and include benefits, costs and challenges for each stakeholder group, benefits, costs and challenges for research per decision‐making point, indicators, methods or tools. LV and NG clustered the indicators, methods and tools into qualitative and quantitative types. Benefits and costs were mapped to suggested or applied indicators and tools or methods. The decision‐making point focus of articles was interpreted by the researchers if not defined in the article. Benefits and costs that could not easily be linked to one particular decision‐making point were analysed separately. LV and TF agreed on the data analysis strategy, and sections of the analysis were cross‐checked by comparing interpretations of results; inconsistencies were discussed and agreed.
2.4. Consultation and validation
The preliminary results of the review were presented and discussed during a PARADIGM meeting held in London, 18 July 2018. This session provided valuable input on how best to present and categorize the results. Participants in the meeting included representatives from patient organizations, pharmaceutical companies and academia with an interest and considerable expertise in patient engagement. Based on the discussions during the meeting, it was agreed to structure the results per key decision‐making point and per stakeholder, including benefits and costs. These members of the PARADIGM consortium were involved in writing this article; their interpretation of results informed the discussion and conclusion. Furthermore, their contributions to the entire research process informed the direction of research, the terminology and definitions used in this article.
3. RESULTS
A total of 91 documents met the eligibility criteria (academic literature n = 77 and grey literature n = 14). Included documents were published between 2000 and 2018 and focused mainly on the health research field. We found limited documents in the field of regulation and health technology assessment. Most documents were published in the United Kingdom in an academic setting. We found largely qualitative studies and literature reviews. Sixteen studies reported that patients were involved in the study as partners. Table 3 provides an overview of the characteristics of included documents.
Table 3.
Characteristic | Output |
---|---|
Year | Last 8 y (2010‐2018) (n = 69) |
10 y (2000‐2010) (n = 22) | |
Focus | Clinical trial (n = 24) |
Health research (n = 47) | |
Regulation and HTA (n = 8) | |
Other (n = 12) | |
Country of origin | Canada (n = 11) |
United Kingdom (n = 40) | |
Canada and the United Kingdom (n = 1) | |
United States (n = 27) | |
Europe (n = 6) | |
Netherlands (n = 4) | |
Germany (n = 1) | |
Denmark (n = 1) | |
Setting | Academia (n = 38) |
Health care (n = 19) | |
Industry (n = 3) | |
Mixed (n = 14) | |
Other (n = 17) | |
Methodology (academic literature only) | Quantitative study (n = 6) |
Qualitative study (n = 23) | |
Mixed method study (n = 15) | |
Literature review (n = 25) | |
Commentary/Editorial/Opinion/other (n = 8) | |
Patients involved as partners in the study (academic literature only) | Yes (n = 16) |
Unspecified (n = 61) |
In this section, we present the findings of our review, first considering the three decision‐making points, which were relevant to our search. Not all reported benefits and costs could easily be linked to one decision‐making point. These referred to overall benefits and costs for stakeholders and general costs or challenges for research and development. Therefore, we report them separately.Additionally, reported benefits and costs were omitted where they related to other phases of the research process (such as interpretation of research findings or dissemination of results).
3.1. Benefits, costs and challenges for research and development
A total of 18 benefits and five costs of patient engagement at three R&D decision‐making points were identified. These were grouped into 11 domains and mapped with 28 possible indicators for their evaluation. Tables 4 and 5 provides an overview of indicators per domain. Please refer to Appendices S3–S5 for more detailed indicators, evaluation methods and tools.
Table 4.
Research priority setting | |
Usability benefits (1) | Examples of indicators related to usability benefits (total: 6) |
More relevant research topics and priorities, based on patients’ needs3, 4, 15, 17, 20, 22, 23, 29, 30, 50, 51, 52, 53, 54, 55 |
Rating of influence of patients and other stakeholders61 Rating of relevance or importance of studies23, 59 Perceptions or degree of contentment/satisfaction with the topic generation and prioritization process96 Similarities and differences in research priorities between stakeholder groups15 Types of research gaps reported that were not previously identified61 Perceptions on how patients’ experiential knowledge helped shaped the research question30 |
Research questions, hypothesis, interventions and medical technologies become more relevant and usable for patients24, 30 | |
Societal benefits (2) | Examples of indicators related to societal benefits (total: 3) |
More appropriate resource allocation, based on patients’ needs30 |
Comparison of academic and lay scores assigned to research proposals60 Perceptions of public influence on funding decisions60 Indicators of dynamics in the panel discussion61 |
Funding benefits (3) | Examples of indicators related to funding benefits (total: 1) |
Improved fundability and credibility of research proposals25, 29, 30, 31, 32, 56, 57, 58 |
Number of studies that had success in gaining research funding12 |
Design of clinical trials | |
Ethical benefits (4) | Examples of indicators related to ethical benefits (total: 1) |
More appropriate, inclusive and sensitive research design8, 17, 29, 30, 52, 55, 58 |
Number of studies that had success in gaining ethics approval12 |
Methodological benefits (5) | Examples of indicators related to methodological benefits (total: 4) |
More appropriate wording and timing of research instruments and interventions17, 20, 22, 24, 25, 27, 29, 31, 55, 56, 64, 65, 66, 67, 68 |
Number of changes made to clinical trial communication as a result of study participant feedback59 |
Increased readability and accessibility of research materials4, 20, 24, 25, 29, 31, 40, 55, 56, 67, 68 |
Reading level of research documents/instruments70 Rating or perceptions of understanding of the consent form70 |
More relevant research outcomes/endpoints32, 41, 93 |
Number and type of patient‐reported outcomes61 |
Study quality benefits (6) | Examples of indicators related to study quality benefits (total: 7) |
Improved recruitment and retention23, 24, 29, 40, 69 |
Number of study participants who dropout for reasons other than adverse reactions59 |
Increased diversity of study participants66 |
Recruitment and retention rates among hard‐to‐reach population, level of diversity61 |
Improved trial experience/satisfaction by study participants2, 80 |
Rating or explore feelings of satisfaction among study participants15, 70 Rating convenience of study visits and procedures by study participants59 |
More adherence to the research protocol93 |
Number of protocol amendments59 |
Faster study completion2, 23 |
Number of studies completed within a particular timeframe3, 61 |
Regulatory and HTA processes | |
Instrumental benefits (7) | Examples of indicators related to instrumental benefits (total: 1) |
Higher accuracy in measuring needs and preferences of patients71, 72 |
Perceptions on how patient input was used and added value for assessment75, 76 |
Better quality of assessment (in terms of relevance and reliability to local context)71, 72 | |
Study uptake benefits (8) | Examples of indicators related to study uptake benefits (total: 2) |
Uptake of evidence/approval by regulators and HTA bodies2, 73 |
Time to approval/response of regulators52 Changes in the proportion of drugs recommended for reimbursement36 |
Developmental benefits (9) | Examples of indicators related to developmental benefits |
Knowledge and public awareness of products72 |
None reported |
Democratic accountability and transparency72 |
Table 5.
Various decision‐making points | |
Non‐financial costs (10) | Examples of indicators related to non‐financial costs (total: 2) |
Biases in recruitment or findings24, 67 |
Perceived negative impacts of patient engagement for research and development24 Total hours spent on engagement24 |
Scientific and ethical conflict in protocol design20 | |
Power struggles20 | |
Increased time20 | |
Financial costs (11) | Examples of indicators related to financial costs (total: 1) |
Increased costs20 |
Total monetary costs of engagement for research and development24 |
3.1.1. Benefits of patient engagement in research priority setting
Literature suggests that patient engagement in research priority setting has several benefits. We identified four unique benefits and nine possible indicators. We clustered the benefits into three domains: usability benefits, societal benefits and funding benefits. Usability benefits refer to impact on the topic generation and prioritization process, for example more relevant topics and priorities based on patients’ needs3, 4, 15, 17, 20, 22, 23, 29, 30, 50, 51, 52, 53, 54, 55 and the relevance of studies, for example more relevant research questions and medical interventions or technologies.30 Societal benefits refer to the way public and private resources are allocated, for example more appropriate resource allocation based on patients’ needs.30 Funding benefits refer to new funding and funding opportunities, for example success in gaining funding due to enhanced credibility of research proposals.25, 29, 30, 31, 32, 56, 57, 58
In the literature, quantitative methods are used to collect information about the perceived importance of studies by patients, the perceived influence of stakeholders in research priority setting,23, 59 or to compare academic and lay scores assigned to research proposal evaluation.60 For example, studies suggest rating the importance or influence of partners in developing the research topics.23, 59 Qualitative methods are used to explore the relevance of research topics and how patients’ experiential knowledge helped shape the research question.30 The Patient‐Centered Outcome Institute (PCORI) uses mixed methods (survey, focus groups, database review) to explore the perceptions incorporated into the topic selection process and the kinds of research gaps documented as important to patients and other stakeholders that were not previously identified.61 Quantitative methods could also be used for comparison of academic and lay scores assigned to research proposals.60 Qualitative methods are suggested for exploring similarities and differences in research priorities.15 For example, Brown et al invited patients with diabetes to focus groups to identify research priorities. Results were analysed using the constant comparative method and compared with current expert‐led research priorities in diabetes.62 Additionally, documentary analyses (eg review of minutes, grant applications, reports) may be conducted to compare patient input and responsiveness to patients’ ideas.54, 61, 63
3.1.2. Benefits of patient engagement in the design of clinical trials
We identified ten unique benefits of patient engagement for the design of clinical trials, including 13 possible indicators. We clustered the benefits into three domains: ethical benefits, methodological benefits and study quality benefits. Several studies described ethical benefits such as a more appropriate, inclusive and sensitive research design.8, 17, 29, 30, 52, 55, 58 Furthermore, studies described methodological benefits such as more appropriate wording and timing of research instruments and interventions,17, 20, 22, 24, 25, 27, 29, 31, 55, 56, 64, 65, 66, 67, 68 and improved consent forms and accessible recruitment materials.4, 20, 24, 25, 29, 31, 40, 55, 56, 67, 68 Study quality benefits are also reported, for example improved trial recruitment and retention.23, 24, 29, 40, 69
The literature suggests several indicators and methods for the evaluation of patient engagement in the design of clinical trials. For example, Guarino et al measured participants’ understanding of the study consent form, using the Informed Consent Questionnaire‐4 questionnaire. The reading levels of the consent forms were assessed using Flesch‐Kincaid reading level scores.70 Rating the impact of patient engagement on study volunteer attitudes about aspects of the participation process (eg ease of understanding the informed consent form; convenience of study visits and procedures) is also suggested.59 Other studies suggest collecting data on the number of studies that gain research ethics committee approval,12 the number of protocol amendments59 and the number and type of patient‐reported outcomes.61 Furthermore, several studies have assessed study quality benefits, for example recruitment rates, using different quantitative methods.40, 69, 70 Iliffe, McGrath and Mitchell40 compared recruitment levels before and after the involvement of the public. Guarino et al70 also conducted a comparison; they assessed the effect of two different consent documents on recruitment levels using one consent form developed by a consumer focus group compared with another developed by the study investigators. Ennis and Wykes conducted a quantitative analysis of successful recruitment by studies where patient engagement was undertaken. A change in patient engagement over time was assessed by correlating study entry order (studies were ordered by the date identified) with the level of patient engagement. Additionally, suggested indicators include recruitment and retention rates among hard‐to‐reach populations,61 the number of dropouts for reasons other than adverse reactions, the total number of changes made to clinical trial communications as a result of patient feedback,59 and the number of studies completed within a particular time frame.3, 61 Validated questionnaires such as the Client Satisfaction Questionnaire‐8 measure overall satisfaction of study participants.70 Qualitative methods are mostly suggested for gathering information about participants’ experiences of taking part in a clinical trial.15
3.1.3. Benefits of patient engagement in regulatory processes and health technology assessment (HTA)
We identified five unique benefits of patient engagement in regulatory processes and HTA, including four possible indicators. The benefits can be categorized into three dimensions: instrumental benefits, study uptake benefits and developmental benefits. Instrumental benefits are related to improving the relevance of assessment to making better quality decisions, for example higher accuracy in measuring needs and preferences of patients and better quality of assessment and relevance of reports to the local context.71, 72 Study uptake benefits refer to the usefulness of assessments for decision makers and the uptake of evidence by decision makers, for example gaining regulatory approval.2, 73 Developmental benefits include, for example, increasing the public's understanding of HTA and openness of decision processes.72
Literature suggests a few methods to evaluate the benefits of patient engagement in regulatory processes and HTA. Quantitative methods are suggested to assess study uptake benefits such as the time to response/approval of regulators and a change in the proportion of drugs recommended for reimbursement.3, 74 Furthermore, quantitative methods could be used to assess the perceived impact. For example, the European Medicines Agency has used a survey to assess the perceived added value of patient input in scientific advice processes and feedback.75 Qualitative methods can also be used to explore measures of change or uptake of patients’ input. For example, Abelson et al76 assessed how patients’ input informed the HTA process through document analysis, interviews and observations. Dipankui et al77 used semi‐structured interviews and document analysis (eg HTA reports, minutes) to evaluate how patient engagement changed the HTA report and its recommendations.
3.2. Costs and challenges of patient engagement in research and development
Limited studies have published costs and challenges. Of those which have, most studies reported increased time and costs for researchers and research institutions due to the practical aspects of planning and managing patient engagement. For example, there are increased time and financial costs from building relationships with the relevant community, setting up user groups, organizing and providing training and education for users and researchers, and the additional time needed for users to read and comment on documentation.20 Only two studies suggest that patient engagement could potentially result in a more homogenous sample or biases in recruitment.24, 67 For example, Blackburn et al24 reported that a more homogenous study sample may have been recruited, since the young contributors encouraged their friends to participate in a study on reproductive health in young people. Furthermore, Brett et al found that studies indicated that patient engagement led to scientific and ethical conflict in protocol design. Also, patient engagement may lead to tokenistic engagement and can lead to power struggles between researchers and patient partners.20 Furthermore, stakeholders have raised concerns that engaged patients may want to see their clinical trials succeed, and as a result, these patients may bias the study findings.59 It was also reported that a number of clinical research professionals fear that patient centricity is pushing them to discard traditional practices, including the use of blinded, randomized controlled clinical trials.59
Methods to assess costs include qualitative methods to gather insights into the perceived effort of engagement as well as a quantitative method to gather insights into financial costs. For example, the costs and consequences framework developed by Blackburn et al includes questions about costs for researchers such as total costs associated with recruiting patients involved, the total costs associated with training patients involved, the total costs associated with supporting patients, financial payment/rewards, total costs of expenses reimbursed to all patients for their involvement and other costs (including parking permits, room booking, audio‐visual, equipment). A separate questionnaire developed for patients includes questions about the hours spent on engagement, the costs they incurred (eg travel, child care, food and drinks, accommodation) and any costs related to arrangement and planning (for instance changed shifts at work or arranged care for a relative).24 Log sheets are also used to gather insights into time and costs.27 Open questions are used to gather insights into (non‐financial) negative impacts.24
3.3. Benefits, costs and challenges for stakeholders
Studies that assessed patient engagement for individuals and organizations mostly highlighted benefits, costs and challenges for patients engaged, with comparatively less published on the benefits and costs for other groups. Based on our review, suggested dimensions to measure the benefits, costs and challenges for the individuals and organizations involved relate to personal development, skills and knowledge, emotions and meaningful relationships, financial, performance and strategic value, transparency and awareness, trust and mutual respect. A summary of reported benefits and costs for stakeholders can be found in Table 6. Please refer to Appendices S1 and S2 for more detailed information on benefits, costs and challenges for patients and other stakeholders.
Table 6.
Individuals and organizations | Benefits | Costs and challenges |
---|---|---|
Patient partners |
|
|
Society | ||
Research participants | ||
Researchers | ||
Research institutes | ||
Research funders |
|
|
Industry |
|
|
Regulators and health technology assessment bodies |
|
|
Others (decision makers and health‐care providers) |
|
|
Multiple tools have been developed to assess the benefits and costs for stakeholders. The Evaluation Toolkit is a resource designed for practitioners of the health sector, produced after the completion of a rigorous systematic review of patient and public engagement evaluation tools.78 Boivin et al reviewed the tools and concluded that most tools were designed to collect information from patients and the public; very few instruments measure the perspectives of other stakeholder groups. The authors of the review reported that the outcomes of patient engagement were least often evaluated (55.6% of the tools), in contrast to the engagement process and context. The most common focus of tools that measure outcomes was on perceived, self‐reported impacts. Methods are qualitative (eg interviews, focus groups) and quantitative for perceived self‐reported benefits (eg surveys using Likert scales). Self‐administered questionnaires and surveys were the most common types of tools identified.21
4. DISCUSSION
To address the need for means of determining the “return on engagement,” the aim of this paper was to review the literature on monitoring and evaluation of patient engagement. This review identified a range of benefits, costs and challenges that patient engagement can have on R&D and describes several indicators associated with their monitoring and evaluation. In addition, we summarized the overall reported benefits, costs and challenges for stakeholders involved in patient engagement initiatives. In this section, we reflect on the indicators and methods found in this review and consider the review's methodological strengths and limitations.
4.1. Reflection on our findings
A total of 18 benefits and five costs of patient engagement at the three decision‐making points were identified in this review. These were grouped into 11 domains and mapped with 28 possible indicators for their evaluation. Little is known about the validity and performance of these indicators as most were suggested rather than applied, or used in single studies. Those studies mostly considered a single indicator (eg recruitment rate) for trying to answer a single question (eg Does patient engagement in research lead to better recruitment?). Measuring this may be feasible but may not be useful in predicting impact for other studies, as the factors influencing impact may differ. This has been noted by other authors.36 We argue that currently available indicators are of some use in measuring benefits, but are not sufficient to understand the pathway to impact, or whether the interaction between researchers and patients involved could lead to change in the external environment (eg research culture, structure and practice). We argue that the impacts of patient engagement can best be determined not by applying a single indicator, but a coherent set of measures. Given the importance of context and the complexity of evaluating patient engagement that this review illustrates, we are developing a monitoring and evaluation framework that considers various indicators for patient engagement practices in medicines research and development. This framework is informed by other frameworks and being tested in practice. We will publish our findings of working with a more coherent evaluation approach in medicines research and development shortly.
This review also looked at methods for evaluation. We identified several quantitative methods to measure the benefits of patient engagement; these mostly assess the benefits on study quality, study uptake and self‐reported benefits. Qualitative methods are mostly suggested for gathering information about experiences, attitudes and perceptions. We agree with others that there is a need for new evaluation methods and tools that focus on observable impact on the research process and benefits for those involved.4, 21, 79, 80 Some argue for broadly applicable, quantitative methods whilst others contend that more subjective, qualitative methods are necessary to capture the nuances of outcomes and impacts of patient engagement.13, 17, 59, 76 Universally applicable evaluation criteria that capture all aspects of engagement are supported for reasons of consistency, reliability and comparison across different projects.16 To build an evidence base, conceptual and practical guidance and some level of consensus on measurable impacts are needed. This has also been suggested by other authors.13, 30 However, whilst a standardized approach may be appealing to health research and development communities, it is problematic in the complex and contextually dependent arenas of patient engagement.81 It might inhibit capacity‐building in projects and makes changes difficult; arguably, this undermines the original rationale for patient engagement. The tension between obtaining comparable data on patient engagement by using metrics (standardized or agreed techniques for gathering information) and tailored participatory evaluative approaches should not be overlooked. By implication, it should be recognized that measures can be valued and applied differently in different contexts; therefore, we recommend discussing relevant and feasible indicators and methods per setting.
4.2. Strengths and limitations of this review
To our knowledge, this is the first literature review that attempts to capture the existing publications about the evaluation of patient engagement practice as it relates to medicines development. It both maps outcomes and impacts of patient engagement with suggested measures for each decision‐making point in R&D.
Very few publications refer to costs or negative impact of engagement, compared with positive findings. This may be because people tend not to report negative outcomes and impacts despite their being just as important. There were very few studies that considered patient engagement in the HTA process, and only, three publications were authored by (and for) the pharmaceutical industry. Furthermore, of the papers included in our review, very few reported that they had involved patients; therefore, the conclusions derived from the studies may be based on the perspectives of researchers. For this review, a meeting was held to discuss preliminary findings with a broad range of stakeholders in our project and the co‐authors of this paper work for patient representative groups and industry. We therefore feel that our findings may be considered relevant to a broader audience than a predominantly academic one.
Our focus on the measurement of impact of patient engagement in the development of medicines has resulted in several limitations to our review. Because this is a scoping review rather than a systematic review, we may have missed relevant articles. Our search focused on titles and abstracts of publications and three decision‐making points, which means that some articles (eg related to other time points) have been excluded. We specifically searched for outcomes and impact of patient engagement in the R&D of medicines; therefore, our paper does not include context or process indicators, or the indicators per stakeholder group. Furthermore, we cannot draw hard conclusions about the relationship between input, outcomes and impact with respect to the benefits and costs for the people and organizations involved in patient engagement. Finally, we had to exclude articles not published in English. Whilst we are aware that most publications on this topic are written in English originating from the UK and North America, we acknowledge that we may have missed relevant publications in other languages.
5. CONCLUSIONS
For patient engagement in the development of medicines to become standard practice at the key decision‐making points of priority setting, clinical trial design and regulatory and HTA processes, benefits need to be demonstrable to all stakeholders. This literature review has mapped benefits, costs and challenges with indicators in current literature. Discrete tools and methods for evaluation are less apparent, as is evidence of their application. The approaches to evaluation we found are largely qualitative, and our review suggests that there are few quantitative tools and no standardized approaches to assessing the outcomes and impact. The reported costs, challenges and benefits are largely congruent, with agreement that there is a need for consensus‐based monitoring and evaluation frameworks that include metrics.
We suggest that the development of a coherent set of measures warrants further investigation and that the benefits, costs and challenges of patient engagement for all stakeholders should be given more consideration (rather than the current focus on benefits for research). To this end, we will co‐develop and test an evaluation framework with stakeholders using a reflexive monitoring approach in real‐life cases of patient engagement in medicines research and development.
CONFLICT OF INTEREST
None declared.
Supporting information
ACKNOWLEDGEMENTS
We want to thank Nicole Goedhart from the Athena Institute for assisting with the analysis. We also would like to thank Carina Pittens from the Athena Institute for reviewing the first drafts. We are grateful for the support of Nia Roberts, librarian at the University of Oxford. Furthermore, we very much appreciate the feedback from Joanna Crocker from the University of Oxford and María José Vicente Edo from Instituto Aragonés de Ciencias de la Salud.
Vat LE, Finlay T, Jan Schuitmaker‐Warnaar T, et al. Evaluating the “return on patient engagement initiatives” in medicines research and development: A literature review. Health Expect. 2020;23:5–18. 10.1111/hex.12951
Funding information
This study was funded by PARADIGM. PARADIGM is a public‐private partnership and is co‐led by the European Patients’ Forum and the European Federation of Pharmaceutical Industries and Associations (EFPIA). PARADIGM is receiving funding from the Innovative Medicines Initiative Joint Undertaking 2. This Joint Undertaking receives support from the European Union's Horizon 2020 research and innovation programme and EFPIA.
DATA AVAILABILITY STATEMENT
The data that support the findings of this study are available from the corresponding author upon reasonable request.
REFERENCES
- 1. PARADIGM . Patients active in research and dialogues for an improved generation of medicines. PARADIGM. https://imi-paradigm.eu/our-work/. Published 2018. Accessed October 6, 2019.
- 2. Levitan B, Getz K, Eisenstein EL, et al. Assessing the financial value of patient engagement: a quantitative approach from CTTI’s patient groups and clinical trials project. Ther Innov Regul Sci. 2018;52(2):220‐229. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Geissler J, Ryll B, di Priolo SL, Uhlenhopp M. Improving patient involvement in medicines research and development: a practical roadmap. Ther Innov Regul Sci. 2017;51(5):612‐619. [DOI] [PubMed] [Google Scholar]
- 4. Duffett L. Patient engagement: what partnering with patient in research is all about. Thromb Res. 2017;150:113‐120. [DOI] [PubMed] [Google Scholar]
- 5. Barber R, Boote JD, Parry GD, Cooper CL, Yeeles P, Cook S. Can the impact of public involvement on research be evaluated? A mixed methods study. Health Expect. 2012;15(3):229‐241. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Francisca Caron‐Flinterman J, Broerse J, Teerling J, Bunders J. Patients’ priorities concerning health research: the case of asthma and COPD research in the Netherlands. Health Expect. 2005;8(3):253‐263. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Pittens C, Noordegraaf AV, van Veen SC, Anema JR, Huirne J, Broerse J. The involvement of gynaecological patients in the development of a clinical guideline for resumption of (work) activities in the Netherlands. Health Expect. 2015;18(5):1397‐1412. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Wilson P, Mathie E, Keenan J, et al.Research with patient and public involvement: a realist evaluation – the RAPPORT study. Southampton (UK): NIHR Journals Library; 2015. https://www.journalslibrary.nihr.ac.uk/hsdr/hsdr03380/#/plain-english-summary. Accessed June 10, 2019. [PubMed]
- 9. Evans D, Bird E, Gibson A, et al. Extent, quality and impact of patient and public involvement in antimicrobial drug development research: a systematic review. Health Expect. 2018;21(1):75‐81. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Fergusson D, Monfaredi Z, Pussegoda K, et al. The prevalence of patient engagement in published trials: a systematic review. Res Involv Engagem. 2018;4(1):17. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Schölvinck A.Towards meaningful and sustainable patient involvement in health research decision‐making. 2018. https://research.vu.nl/en/publications/towards-meaningful-and-sustainable-patient-involvement-in-health-. Accessed February 13, 2019.
- 12. Crocker JC, Boylan A, Bostock J, Locock L. Is it worth it? Patient and public views on the impact of their involvement in health research and its assessment: a UK‐based qualitative interview study. Health Expect. 2017;20(3):519‐528. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Staniszewska S, Adebajo A, Barber R, et al. Developing the evidence base of patient and public involvement in health and social care research: the case for measuring impact: Patient and public involvement in health and social care research. Int J Consum Stud. 2011;35(6):628‐632. [Google Scholar]
- 14. Staniszewska S, Herron‐Marx S, Mockford C. Measuring the impact of patient and public involvement: the need for an evidence base. Int J Qual Health Care. 2008;20(6):373‐374. [DOI] [PubMed] [Google Scholar]
- 15. Collins M, Long R, Page A, Popay J, Lobban F. Using the public involvement impact assessment framework to assess the impact of public involvement in a mental health research context: a reflective case study. Health Expect. 2018;21:950‐963. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Rowe G, Frewer LJ. Evaluating public‐participation exercises: a research agenda. Sci Technol Human Values. 2004;29(4):512‐556. [Google Scholar]
- 17. Staley K. “Is it worth doing?” Measuring the impact of patient and public involvement in research. Res Involv Engagem. 2015;1:6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Kok M.Guidance document: evaluating public involvement in research. 2018. http://eprints.uwe.ac.uk/36354/1/UWE_PHWE%2520guidance%2520doc%2520on%2520evaluating%2520PI%2520in%2520research.pdf. Accessed June 7, 2018.
- 19. Manafo E, Petermann L, Mason‐Lai P, Vandall‐Walker V. Patient engagement in Canada: a scoping review of the ‘how’ and ‘what’ of patient engagement in health research. Health Res Policy Syst. 2018;16:5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Brett JO, Staniszewska S, Mockford C, et al. Mapping the impact of patient and public involvement on health and social care research: a systematic review. Health Expect. 2014;17(5):637‐650. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Boivin A, L'Espérance A, Gauvin F‐P, et al. Patient and public engagement in research and health system decision making: a systematic review of evaluation tools. Health Expect. 2018;21:1075‐1084. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Mann C, Chilcott S, Plumb K, Brooks E, Man M‐S. Reporting and appraising the context, process and impact of PPI on contributors, researchers and the trial during a randomised controlled trial – the 3D study. Res Involv Engagem. 2018;4:15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Forsythe L, Heckert A, Margolis MK, Schrandt S, Frank L. Methods and impact of engagement in research, from theory to practice and back again: early findings from the Patient‐Centered Outcomes Research Institute. Qual Life Res. 2018;27(1):17‐31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Blackburn S, McLachlan S, Jowett S, et al. The extent, quality and impact of patient and public involvement in primary care research: a mixed methods study. Res Involv Engagem. 2018;4:16. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Gordon J, Franklin S, Eltringham SA. Service user reflections on the impact of involvement in research. Res Involv Engagem. 2018;4(1). 10.1186/s40900-018-0095-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Vale CL, Tierney JF, Spera N, Whelan A, Nightingale A, Hanley B. Evaluation of patient involvement in a systematic review and meta‐analysis of individual patient data in cervical cancer treatment. Syst Rev. 2012;1:23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Evans D, Coad J, Cottrell K, et al. Public involvement in research: assessing impact through a realist evaluation. Southampton (UK): NIHR Journals Library; 2014. http://www.ncbi.nlm.nih.gov/books/NBK260168/. Accessed October 6, 2019. [PubMed]
- 28. Staley K. Changing what researchers “think and do”: is this how involvement impacts on research? Res All. 2017;1(1):158‐167. [Google Scholar]
- 29. Staley K, INVOLVE (Organization) , National Institute for Health Research (Great Britain) . Exploring impact: public involvement in NHS, public health and social care research. Eastleigh, England: National Institute for Health Research; 2009. http://www.invo.org.uk/pdfs/Involve_Exploring_Impactfinal28.10.09.pdf. Accessed October 6, 2019.
- 30. Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in research: moving from theory to practice. J Comp Eff Res. 2015;4(2):133‐145. [DOI] [PubMed] [Google Scholar]
- 31. Price A, Albarqouni L, Kirkpatrick JO, et al. Patient and public involvement in the design of clinical trials: an overview of systematic reviews. J Eval Clin Pract. 2017;24(1):240‐253. [DOI] [PubMed] [Google Scholar]
- 32. Domecq JP, Prutsky G, Elraiyah T, et al. Patient engagement in research: a systematic review. BMC Health Serv Res. 2014;14:89. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Jagosh J, Macaulay AC, Pluye P, et al. Uncovering the benefits of participatory research: implications of a realist review for health research and practice. Milbank Q. 2012;90(2):311‐346. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34. Dillon EC, Tuzzio L, Madrid S, Olden H, Greenlee RT. Measuring the impact of patient‐engaged research: how a methods workshop identified critical outcomes of research engagement. J Patient Cent Res Rev. 2017;4(4):237‐246. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Gagnon M‐P, Dipankui MT, DeJean D. Evaluation of patient involvement in HTA In: Facey K, Ploug Hansen H, Single A, eds. Patient Involvement in Health Technology Assessment. Singapore: Adis; 2017:201‐213. [Google Scholar]
- 36. Staley K, Barron D. Learning as an outcome of involvement in research: what are the implications for practice, reporting and evaluation? Res Involv Engagem. 2019;5(1):14 10.1186/s40900-019-0147-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37. Greenhalgh T, Hinton L, Finlay T, et al. Frameworks for supporting patient and public involvement in research: systematic review and co‐design pilot. Health Expect. 2019;1-17. 10.1111/hex.12888 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38. Standards Development Partnership . UK standards for public involvement in research. https://sites.google.com/nihr.ac.uk/pi-standards/home. Published March 2018. Accessed June 7, 2019.
- 39. Patient Focused Medicines Development . Patient engagement quality guidance. 2018. https://patientfocusedmedicine.org/the-patient-engagement-quality-guidance/. Accessed October 6, 2019.
- 40. Iliffe S, McGrath T, Mitchell D. The impact of patient and public involvement in the work of the Dementias & Neurodegenerative Diseases Research Network (DeNDRoN): case studies. Health Expect. 2013;16(4):351‐361. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41. Dudley L, Gamble C, Preston J, et al. What difference does patient and public involvement make and what are its pathways to impact? Qualitative study of patients and researchers from a cohort of randomised clinical trials. PLoS ONE. 2015;10(6):e0128817. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42. Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19‐32. [Google Scholar]
- 43. Mays N, Roberts E, Popay J. Synthesising research evidence In: Fulop N, Allen P, Clarke A, Black N, eds.Studying the Organisation and Delivery of Health Services: Research Methods. London: Routledge; 2001:188-220. [Google Scholar]
- 44. Peters M, Godfrey C, Khalil H, McInerney P, Parker D, Soares C. Guidance for conducting systematic scoping reviews. Int J Evid Based Healthc. 2015;13(3):141‐146. [DOI] [PubMed] [Google Scholar]
- 45. Munn Z, Peters M, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18(1):143. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46. PCORI . Patient‐Centered Outcomes Research Institute. Research and results. https://www.pcori.org/research-results-home. Published 2018. Accessed October 6, 2019.
- 47. INVOLVE . INVOLVE evidence library. http://www.invo.org.uk/resource-centre/libraries/evidence-library/. Published 2018. Accessed October 6, 2019.
- 48. Wohlin C. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering ‐ EASE ’14 London, UK: ACM Press; 2014:1‐10. [Google Scholar]
- 49. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77‐101. [Google Scholar]
- 50. Manafò E, Petermann L, Vandall‐Walker V, Mason‐Lai P. Patient and public engagement in priority setting: a systematic rapid review of the literature. PLoS ONE. 2018;13(3):e0193579. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51. Sofolahan‐Oladeinde Y, Newhouse RP, Lavallee DC, Huang JC, Mullins CD. Early assessment of the 10‐step patient engagement framework for patient‐centred outcomes research studies: the first three steps. Fam Pract. 2017;34(3):272‐277. [DOI] [PubMed] [Google Scholar]
- 52. Ray KN, Miller E. Strengthening stakeholder‐engaged research and research on stakeholder engagement. J Comp Eff Res. 2017;6(4):375‐389. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53. de Wit M, Abma TA, Loon M, Collins S, Kirwan J. What has been the effect on trial outcome assessments of a decade of patient participation in OMERACT? J Rheumatol. 2014;41(1):177‐184. [DOI] [PubMed] [Google Scholar]
- 54. Caron‐Flinterman JF, Broerse J, Teerling J, et al. Stakeholder participation in health research agenda setting: the case of asthma and COPD research in the Netherlands. Sci Public Policy. 2006;33(4):291‐304. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55. Brett J, Staniszewska S, Mockford C, Seers K, Herron‐Marx S, Bayliss H. The PIRICOM study. 2010. http://www.ukcrc.org/wp-content/uploads/2014/03/Piricom+Review+Final+2010.pdf. Accessed October 6, 2019.
- 56. Wyatt K, Carter M, Mahtani V, Barnard A, Hawton A, Britten N. The impact of consumer involvement in research: an evaluation of consumer involvement in the London Primary Care Studies Programme. Fam Pract. 2008;25(3):154‐161. [DOI] [PubMed] [Google Scholar]
- 57. Lindenmeyer A, Hearnshaw H, Sturt J, Ormerod R, Aitchison G. Assessment of the benefits of user involvement in health research from the Warwick Diabetes Care Research User Group: a qualitative case study. Health Expect. 2007;10(3):268‐277. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58. Public Involvement Impact Assessment Framework (PiiAF) Study Group . Database – methods and tools to assess impacts: signposting resource to published case examples of methods and tools. 2013. http://www.piiaf.org.uk/documents/impacts-database.pdf. Accessed August 6, 2018.
- 59. Getz KA. Establishing return‐on‐investment expectations for patient‐centric initiatives. Ther Innov Regul Sci. 2015;49:745‐749. [DOI] [PubMed] [Google Scholar]
- 60. Andejeski Y, Bisceglio IT, Dickersin K, et al. Quantitative impact of including consumers in the scientific review of breast cancer research proposals. J Womens Health Gend Based Med. 2002;11(4):379‐388. [DOI] [PubMed] [Google Scholar]
- 61. Patient‐Centered Outcomes Research Institute (PCORI) . PCORI‐Evaluation‐Framework‐3.0. 2017. https://www.pcori.org/sites/default/files/PCORI-Evaluation-Framework-3.0.pdf. Accessed October 6, 2019.
- 62. Brown K, Dyas J, Chahal P, Khalil Y, Riaz P, Cummings‐Jones J. Discovering the research priorities of people with diabetes in a multicultural community: a focus group study. Br J Gen Pract. 2006;56(524):206‐213. [PMC free article] [PubMed] [Google Scholar]
- 63. Caron-Flinterman Francisca. A New Voice in Science. Patient Participation in Decision-Making on Biomedical Research. Amsterdam: Vrije Universiteit; 2005. [Google Scholar]
- 64. Forsythe LP, Ellis LE, Edmundson L, et al. Patient and stakeholder engagement in the PCORI pilot projects: description and lessons learned. J Gen Intern Med. 2016;31(1):13‐21. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 65. McKevitt C, Fudge N, Wolfe C. What is involvement in research and what does it achieve? Reflections on a pilot study of the personal costs of stroke. Health Expect. 2010;13(1):86‐94. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 66. De Las ND, Hacker K, DiGirolamo A, Hicks LS. A systematic review of community‐based participatory research to enhance clinical trials in racial and ethnic minority groups. Health Serv Res. 2012;47(3 Pt 2):1363‐1386. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 67. Public Involvement Impact Assessment Framework (PiiAF) Study Group . Tabular summary of findings from a review of reported impacts of public involvement in research from PiiAF study group’s evidence review. 2013. http://piiaf.org.uk/documents/impacts-summary-1113.pdf. Accessed August 23, 2018.
- 68. Boote J, Baird W, Sutton A. Public involvement in the design and conduct of clinical trials: a review. Int J Interdiscip Soc Sci. 2011;5(11):91‐111. [Google Scholar]
- 69. Ennis L, Wykes T. Impact of patient involvement in mental health research: longitudinal study. Br J Psychiatry. 2013;203(05):381‐386. [DOI] [PubMed] [Google Scholar]
- 70. Guarino P, Elbourne D, Carpenter J, Peduzzi P. Consumer involvement in consent document development: a multicenter cluster randomized trial to assess study participants’ understanding. Clin Trials. 2006;3(1):19‐30. [DOI] [PubMed] [Google Scholar]
- 71. European Patients Forum . Patient involvement in health technology assessment in Europe. Results of the EPF survey. 2013. http://www.eu-patient.eu/globalassets/projects/hta/hta-epf-final-report2013.pdf. Accessed October 6, 2019.
- 72. Abelson J, Wagner F, DeJean D, et al. Public and patient involvement in health technology assessment: a framework for action. Int J Technol Assess Health Care. 2016;32(4):256‐264. [DOI] [PubMed] [Google Scholar]
- 73. Borup G, Bach KF, Schmiegelow M, Wallach‐Kildemoes H, Bjerrum OJ, Westergaard N. A paradigm shift towards patient involvement in medicines development and regulatory science: workshop proceedings and commentary. Drug Inf J. 2016;50(3):304‐311. [DOI] [PubMed] [Google Scholar]
- 74. Berglas S, Jutai L, MacKean G, Weeks L. Patients’ perspectives can be integrated in health technology assessments: an exploratory analysis of CADTH Common Drug Review. Res Involv Engagem. 2016;2(1):21. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 75. European Medicines Agency (EMA) . EMA stakeholder engagement report 2017. 2017. http://www.ema.europa.eu/docs/en_GB/document_library/Report/2018/06/WC500251085.pdf. Accessed October 6, 2019.
- 76. Abelson J, Bombard Y, Gauvin F‐P, Simeonov D, Boesveld S. Assessing the impacts of citizen deliberations on the health technology process. Int J Technol Assess Health Care. 2013;29(3):282‐289. [DOI] [PubMed] [Google Scholar]
- 77. Dipankui MT, Gagnon M‐P, Desmartis M, et al. Evaluation of patient involvement in a health technology assessment. Int J Technol Assess Health Care. 2015;31(03):166‐170. [DOI] [PubMed] [Google Scholar]
- 78. Boivin A, Abelson J, L’Espérance A, et al.Evaluation toolkit. CEPPP | Centre of Excellence on Partnership with Patients and the Public. https://ceppp.ca/en/our-projects/evaluation-toolkit/. Accessed June 7, 2019.
- 79. Brett JO, Staniszewska S, Mockford C, et al. A systematic review of the impact of patient and public involvement on service users, researchers and communities. Patient. 2014;7(4):387‐395. [DOI] [PubMed] [Google Scholar]
- 80. Pushparajah DS. Making patient engagement a reality. Patient. 2018;11(1):1‐8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 81. Staley K, Buckland SA, Hayes H, Tarpey M. “The missing links”: understanding how context and mechanism influence the impact of public involvement in research. Health Expect. 2014;17(6):755‐764. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 82. Karazivan P, Dumez V, Flora L, et al. The patient‐as‐partner approach in health care: a conceptual framework for a necessary transition. Acad Med. 2015;90(4):437. [DOI] [PubMed] [Google Scholar]
- 83. Trochim W.Introduction to evaluation. Web Center for Social Research Methods. http://www.socialresearchmethods.net/kb/intreval.php. Published 2006. Accessed October 6, 2019.
- 84. Simister N. Indicators: monitoring‐and‐evaluation‐series. 2015. https://www.intrac.org/wpcms/wp-content/uploads/2016/06/Monitoring-and-Evaluation-Series-Indicators-8.pdf. Accessed September 24, 2018.
- 85. Oliver SR, Rees RW, Clarke‐Jones L, et al. A multidimensional conceptual framework for analysing public involvement in health services research. Health Expect. 2008;11(1):72‐84. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 86. Morrow E, Ross F, Grocott P, Bennett J. A model and measure for quality service user involvement in health research. Int J Consum Stud. 2010;34(5):532‐539. [Google Scholar]
- 87. Minogue V, Boness J, Brown A, Girdlestone J. The impact of service user involvement in research. Int J Health Care Qual Assur Inc Leadersh Health Serv. 2005;18(2–3):103‐112. [DOI] [PubMed] [Google Scholar]
- 88. TwoCanAssociates . An evaluation of the process and impact of patient and public involvement in the advisory groups of the UK Clinical Research Collaboration. TwoCanAssociates; 2009. http://www.ukcrc.org/wp-content/uploads/2014/03/PPI_Evaluation_Report.pdf. Accessed May 3, 2018.
- 89. Cossar J, Neil E. Service user involvement in social work research: learning from an adoption research project. Br J Soc Work. 2015;45(1):225‐240. [Google Scholar]
- 90. Vale CL, Thompson LC, Murphy C, Forcat S, Hanley B. Involvement of consumers in studies run by the Medical Research Council Clinical Trials Unit: results of a survey. Trials. 2012;13:9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 91. Drug Information Association (DIA) . Capturing the value of patient engagement. 2017. https://www.diaglobal.org/en/resources/press-releases/2016/10-31-patient-engagement-study. Accessed June 7, 2018.
- 92. Patrick‐Lake B. Patient engagement efforts with the clinical trial enterprise. 2017. https://www.fda.gov/downloads/AdvisoryCommittees/CommitteesMeetingMaterials/PatientEngagementAdvisoryCommittee/UCM580041.pdf. Accessed October 6, 2019.
- 93. Ashkenazy R, Schneider RF. A patient centricity team tool to enable patient‐focused drug development. Ther Innov Regul Sci. 2016;50(5):577‐580. [DOI] [PubMed] [Google Scholar]
- 94. FasterCures . Expanding the science of patient input: pain points and potential. 2016. http://www.fastercures.org/reports/view/58. Accessed May 18, 2018.
- 95. European Medicines Agency (EMA) . Involvement of patient representatives in scientific advice procedures at EMA. 2018. http://www.ema.europa.eu/docs/en_GB/document_library/Other/2013/08/WC500148296.pdf. Accessed August 2, 2018.
- 96. Sibbald SL, Singer PA, Upshur R, Martin DK. Priority setting: what constitutes success? A conceptual framework for successful priority setting. BMC Health Serv Res. 2009;9(1). 10.1186/1472-6963-9-43 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 97. Price A. Public led online trials and participatory action research: why do wez need them? Eur J Pers Cent Healthc. 2016;4(2):340. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The data that support the findings of this study are available from the corresponding author upon reasonable request.