A Framework to Ensure Patient Partners Have Equal and Contributing Voices Throughout the Research Program Evaluation Process

Abstract

Purpose of program:

Traditionally, peer review was a closed process conducted only by individuals working in the research field. To establish a more integrated and patient-centered approach, one of Canada’s largest kidney research networks (Can-SOLVE CKD) has created a Research Operations Committee (ROC) that includes patients as key members. The ROC represents one way for achieving meaningful patient-oriented research (POR).

Source of information:

Can-SOLVE CKD, a network created as part of the Canadian Institutes of Health Research (CIHR) Strategy for Patient-Oriented Research (SPOR).

Methods:

The ROC consists of patients, physicians, scientists, Indigenous partners, experts in research methodology, and a member of Can-SOLVE CKD’s operational team. On an annual basis, Can-SOLVE CKD’s research teams provide the ROC with a review package, which incorporates information from patient engagement check-in calls and surveys, the project’s knowledge translation plan and products, and a progress report written by the project team. The ROC evaluates the review package and provides feedback and recommendations accordingly.

Key findings:

The transparent nature of the process, regular feedback and review, along with an overt accountability and scoring system, has been embraced by both patients and researchers. As a result of the ROC process, the number of patient leads for each project has grown over a 3-year period and more researchers have received POR and cultural sensitivity training.

Limitations:

While anecdotal evidence suggests this approach is beneficial for achieving POR, formal mechanisms of evaluation are currently lacking.

Implications:

This ROC framework ensures patients are active contributors throughout the research process and could be adopted by other organizations to achieve a more patient-centered approach to research.

Background

Peer review is a critical component of the scientific process; it supports scientific excellence and integrity across the span of research processes including, but not limited to, study design, research relevance and impact, and publication of results. Traditionally, peer review was a closed process conducted only by individuals working in the research field, but more recently, institutions have recognized the benefit of including patient partners and the public.¹ This shift toward public involvement in research benefits all stakeholders, as it encourages researchers to think about the relevance and accessibility of their work and fosters relationships between patients and researchers.² Critically, inclusion of patient partners helps to ensure that research is humanized and remains focused on what matters the most: improving patient outcomes and quality of life.

Frameworks for involving patient and public voice in research exist in the United Kingdom through the National Institute for Health Research (NIHR) and in the United States through the Patient-Centered Outcomes Research Institute (PCORI). Similarly, journals such as the British Medical Journal (BMJ) and Research Involvement and Engagement (RIE) have initiated programs in which patients and the public are involved in the peer-review process of journal articles.¹ The Canadian Institutes of Health Research (CIHR) established the Strategy for Patient-Oriented Research (SPOR), which encourages and supports the shift toward patient engagement by creating space for patient involvement to shape research and health care.³ While there has been some progress toward more involvement of patient partners in the research process, a recent survey by the International Society of Nephrologists revealed there is a lack of formal mechanisms for patient involvement.⁴

Canadians Seeking Solutions and Innovations to Overcome Chronic Kidney Disease (Can-SOLVE CKD) is 1 of 5 pan-Canadian chronic disease networks supported through SPOR. The vision of Can-SOLVE CKD is that every Canadian with or at high risk for chronic kidney disease (CKD) will receive the best recommended care, experience optimal outcomes, and have the opportunity to participate in studies with novel therapies, regardless of age, sex, gender, location, or ethnicity.⁵

The organizational structure of Can-SOLVE CKD puts patients at the center of all activities, including the development, implementation, and evaluation of its 18 patient-centered projects (Figure 1). These projects, which span the research domains of basic science, clinical, and population health (Figure 2), were selected and vetted using a James Lind Alliance methodology⁶ to specifically align with the priorities of kidney patients. In the development of the Can-SOLVE CKD network, the Research Operations Committee (ROC) was created with the purpose of supporting the project teams in actualizing the value of patient-oriented research (POR), through structured rigorous and regular review (Supplemental Material 1)

Figure 1.

Can-SOLVE CKD network overview of committees and councils.

Note. Can-SOLVE CKD = Canadians Seeking Solutions and Innovations to Overcome Chronic Kidney Disease.

Figure 2.

Canadians Seeking Solutions and Innovations to Overcome Chronic Kidney Disease map of research projects and key players.

The ROC consists of patients, physicians, scientists, Indigenous partners, experts in research methodology, and a member of Can-SOLVE CKD’s operational team. The ROC performs annual reviews on all projects to evaluate and provide guidance for successful implementation of the Can-SOLVE CKD’s research program. Whereas patient input has traditionally been completed at a single point during the research review process, if at all, this novel approach using the ROC ensures better integration of patient input throughout the entire research process. A similar approach could be adopted by other organizations interested in creating a more patient-centered research approach.

The Structure of the ROC

The structure of the ROC is intended to capture diverse perspectives and areas of expertise. The current structure includes 1 chair-elect; 3 patient partners based in Canada; 4 researchers/clinicians; and a member of the Can-SOLVE Operations team. These members meet a minimum of 3 times a year to complete their annual reviews, usually involving 6 research projects at a time. As well, 6 international reviewers, 1 international patient partner, and 1 sex and gender lead are asked to provide feedback for each review. It is standard procedure that all ROC members take POR training, as well as Indigenous cultural sensitivity training.

The purpose of the ROC is to assess key elements of Can-SOLVE CKD research projects and advise the project leads. On a quarterly basis, the ROC reports to Can-SOLVE CKD’s Steering Committee (Figure 3), which is ultimately accountable to the Board of Directors of the Canadian Society of Nephrology and the Kidney Foundation of Canada. On an annual basis, or as requested, the ROC provides an update on the research projects to the Patient Governance Circle (PGC), the network’s governance committee led by patient partners.

Figure 3.

The Research Operations Committee oversees the progress of Canadians Seeking Solutions and Innovations to Overcome Chronic Kidney Disease’s 18 research projects, and reports accordingly to the Steering Committee.

The ROC Review Process

Key Elements

The ROC review process incorporates 6 key elements: patient engagement, sex and gender considerations, scientific design, project progress and risk mitigation, Indigenous cultural sensitivity, and knowledge translation (Figure 4). These elements were inspired by the SPOR and developed to reflect the foundational values of Can-SOLVE CKD.

Figure 4.

Canadians Seeking Solutions and Innovations to Overcome Chronic Kidney Disease 6 elements of The Collaborative Peer-Review Model.

The first element, patient engagement, is based on a growing body of research that supports the engagement of patients in setting research agendas and incorporation of patient values in the research review process.⁵ Thus, patients are present throughout the peer-review process and involved in assessment, review, and discussions.

The second element of the review process focuses on sex and gender considerations, in response to the imbalance of women enrolled in clinical trials in Canada. Under-representation in research, specifically clinical trials, disadvantages women by exposing them to potentially ineffective therapies studied predominantly in men and putting them at greater risk for adverse drug reactions.³ Similarly, preclinical studies most commonly include only male animals, thus creating a bias in the scientific literature and knowledge. In view of this, all study protocols are reviewed by the ROC, with attention to sex and gender.

The third and fourth elements include more traditional elements of research review such as the scientific design, project progress, and risk mitigation. Peer researchers contribute to the review by providing feedback on scientific design and prioritizing participant safety. Patient partners are educated as to types of study design, and supported in questioning various projects. Importantly, patient partners are provided with lay summaries describing research processes and outcomes, to support better participation and involvement in the review process. Project progress is often measured alongside the team’s work plan to determine if outcomes and goals align with the project timeline.

A fifth element included in all projects is that of Indigenous cultural sensitivity, as Indigenous peoples have often been underrepresented in research and experience additional barriers to health care in Canada.⁵ This element aims to foster partnerships across communities, advocating for the needs and values of Indigenous peoples throughout the process.

Finally, the peer-review process assesses knowledge translation aspects of the project, to ensure better communication of the research results to the public. With the patient voice present at the review table, and in the context of knowledge translation, the intention is that researchers can improve their ability to share findings to key stakeholders, including patients, which in turn further engages the public in the research process.³ These 6 key elements shape the review process and are integrated into the committee’s assessment criteria (Table 1, evaluation form in Supplemental Material 2).

Table 1.

Can-SOLVE CKD Research Operations Committee Assessment Criteria.

Accomplishments to date

• Has the project team performed as per Work plan/Gantt chart? If not, please provide recommendations.

• Is the project on schedule? If not, please provide recommendations.

• For clinical trials, is enrollment on target? If not, please provide recommendations and indicate if a referral to Clinical Nephrology Trials Network is needed.

Design of work proposed for next 12 month

• Is the scope of work proposed feasible within 12-month period? If not, please provide recommendations.

• Have there been significant changes to the project plan? If so, please identify them and indicate if there are any concerns.

Issues that limit productivity to date and mitigation strategies

• Are the mitigation strategies fitting for the issues identified? If not, please provide recommendations.

• Has the project team identified all foreseeable issues? List any issues that you have identified and potential mitigation strategies.

Deliverables/outputs expected and alignment with Can-SOLVE CKD objectives

• Do the project team members engaged reflect a patient-oriented research team? (ie, Patient partners, policy makers, diverse clinicians, and nonclinicians) If not, please provide recommendations.

• Has there been an effort in completing Patient-oriented Research Training for the project team?

• Do the abstracts/presentations/articles/communication tools align with Can-SOLVE CKD objectives?

Achieved and planned patient engagement

• Are patient partners involved in the following research process?

○ Study design

○ Development of study protocol

○ Preparation for study execution

○ Data collection

○ Analyzing and interpretation data

○ Dissemination

○ Implementation

○ Monitoring and evaluation

• Has a Patient Lead/Co-Leads been identified?

• Has there been patient co-authorship on abstracts/presentations/articles/communication tools?

• How would you rate the quality of engagement out of 10? (ie. Relationship between researchers and patient partners?) please indicate your recommendations for further improvement.

Note. Refer to The Research Operations Committee Standard Operational Procedures supplemental material for more information. Can-SOLVE CKD = Canadians Seeking Solutions and Innovations to Overcome Chronic Kidney Disease.

An Integrated and Collaborative Evaluation Process

On an annual basis, the research teams are requested to complete a review package, which incorporates information from the patient engagement check-in calls and surveys (detailed further below), the project’s knowledge translation plan and products, and a progress report written by the project team (Figure 5). These 3 components of the review packages are submitted to the ROC, allowing the committee members to measure tangible outputs of the project teams and evaluate their success in actualizing the vision of Can-SOLVE CKD and the SPOR Networks. Three ROC members are assigned to evaluate each research project, including a researcher who focuses on scientific methods, a patient partner who focuses on patient engagement, and an additional reader who contributes to the larger discussion. Each reviewer completes an evaluation checklist and then all 3 ROC reviewers, including other committee members, hold a 30-minute session. The ROC Chair-Elect begins the session by summarizing the review package, and the group discusses until it reaches a consensus on recommendations, which are then collated in a letter back to the project team. In addition, each reviewer has the opportunity to provide a specific recommendation to Can-SOLVE CKD leadership on whether funding for the project should continue. Annual review by the ROC allows for course correction, which ensures more successful execution of the project and better understanding of all the partners of the value of the program of work. Ad hoc meetings are scheduled if needed.

Figure 5.

Can-SOLVE CKD collaborative peer-review model process.Note. ICF = Informed Consent Form.

The project team may be asked to provide clarification on specific aspects of the project or respond to the recommendations of the committee. In the event that a project team needs further support from the ROC to align their outputs with the vision of the network, an ROC member and patient engagement liaison may be dispatched to the site to facilitate strategic planning regarding the recommendations. If major concerns persist, the ROC may ask the project team to continue with interim reporting of the team’s progress. To ensure accountability and transparency, the ROC developed an “Escalation” process which clearly defines the steps to be followed if projects are not achieving required standards, up to and including the cessation of funding for the project.

Beginning in 2020, the ROC has decided to hold annual retreats for all its members. The aim is to build group cohesiveness and to discuss strengths, weaknesses, goals, opportunities, and technical aspects of ROC operations, such as the review process for the coming year.

Evaluation of Patient Engagement in Can-SOLVE CKD Research Projects

To evaluate patient engagement, the ROC assesses qualitative aspects of the working relationship between researchers and patient partners for each research project. Two approaches are employed to capture this: patient engagement check-in calls and surveys given to both the researchers and patient partners. Both the survey and check-in calls have been iteratively revised and improved upon with feedback from the ROC and the PGC.

Patient engagement check-in calls take place annually, whereby the key project team members and patient partners participate in a teleconference. The purpose of this call is to provide a safe space to intentionally explore the successes and challenges that exist for partners in research and enhance collaboration as a team. A report of this check-in call is written and included as part of the ROC package to describe the working relationship within the team. Along with highlighting the successes and challenges, the report outlines recommendations for improvement.

The surveys for patients and researchers, respectively, are conducted annually to collect information on personal experiences and the working relationship within the research team. These surveys (Supplemental Material 3) were adapted from the 2016 version of the Patients as Partners in Research: Patient/Caregiver Surveys, developed by Patients Canada.⁷ Both the patient partner and researcher surveys have corroborating questions aimed to identify the productivity of their partnership. Responses from these surveys are kept anonymous and a summary of the results are included in the ROC review package, as discussed above.

Supporting Infrastructure

Some members of the ROC also contribute as members on other Can-SOLVE CKD committees, including the PGC and another patient governance group led specifically by Indigenous People, called the Indigenous Peoples’ Engagement & Research Council (IPERC). Currently, 1 member of IPERC and 2 members of PGC also serve on the ROC. This cross-over helps foster consistency and connectivity across the network’s operations (Figure 2).

As well, the International Research Advisory Committee provides strategic guidance, oversight, and opportunities to collaborate or share knowledge on the research projects with international leaders in public and patient engagement in nephrology.

Can-SOLVE CKD’s Knowledge Translation and Knowledge User (KT/KU) Committee plays a valuable role in supporting KT activities of the ROC. A member of the KT/KU Committee sits in on patient engagement check-in calls each year to ensure that there are adequate KT activities being developed and incorporated into the project plan, and the committee is called upon for KT support throughout the year as needed. In the context of the ROC review process, these supporting committees play a valuable role.

Results

Between December 2017 and December 2019, the ROC has held quarterly review meetings and 3 ad hoc meetings to perform a total of 45 project reviews. Formative feedback on all 6 key elements was provided in each review. As a result of the ROC process and dedicated sessions to evaluate patient engagement, the number of patient partners involved in each project has increased over time. For example, by Year 4 of network operations, there was a total of 94 patient partners across all research projects, an increase from 71 patient partners in Year 2. As well, the number of research projects with a designated patient lead has increased dramatically over a 3-year period⁸; in Year 4, there were 16 projects with a unique patient lead, whereas in Year 2, there were only 3 projects that had a patient lead. Recommendations by the ROC have also driven an increase in POR and Indigenous cultural training among research teams.

Over time, the ROC has worked with research teams to address a variety of issues. In one specific case, a research project was lacking quality patient engagement and struggled to identify patient partners and complete POR training. Following recommendations and guidance from the ROC, there has been an improvement in patient engagement, with 3 patient partners identified and almost half the research team completed POR training.

The ROC process was also beneficial in helping project timelines. For example, one project was struggling to keep to timelines and there was a risk that this project would not be completed in the allotted grant period. Funding was temporarily withheld, and the project went through the escalation process and ultimately worked collaboratively with the Steering Committee across several meetings and constructive brainstorming sessions. This valuable project is now back on track as per the timeline. Notably, the ROC annual reviews allow successes to be recognized and applied to other research projects.

Throughout this process, an evolving understanding of the value, roles, skills, and comfort level of patient partners in the ROC has occurred. It is evident that these learnings have been bidirectional: patient partners have learned from researchers about science, scientific language, study design, and rigor, while researchers have learned about patient perspectives, priorities, and world views. This bidirectional learning has catalyzed a deeper mutual understanding and partnership between researchers and patients. Selected perspectives from members who served on the ROC members can be found in Table 2. These patient partners have shared that in order to be an effective patient partner, reviewers must be comfortable voicing their opinions and must be motivated and interested in the work. In recognition and acknowledgment of the commitment to the committee and the value provided, patient partners were compensated for their time.

Table 2.

Can-SOLVE CKD Patients as Equal Partners—Perspectives From ROC Members.

Chantel Large

ROC Member & Indigenous Peoples’ Engagement and Research Council Member

“The Research Operations Committee was very intimidating at the beginning but the committee is so supportive I never felt incompetent or ‘less than’. It helped me to realize that I have so much knowledge to contribute on patient engagement and I feel my input is really valued and makes a difference for the research teams. I love that it gives me a greater understanding of the research projects and overall the experience has been really positive.”

David Hillier

ROC Member & Patient Council Executive

“To be a member of the ROC is not only an amazing learning opportunity but also having the ability to input, from a patient perspective, into the research projects is incredibly fulfilling. The Committee has allowed me to gain a far better understanding of ‘health research’ including such challenges as Research Ethic Board approvals and multi-centre funding requirements. I believe patient engagement is critical in Patient Oriented Research. As a patient partner on the committee, I have had the opportunity to provide meaningful input into each of the research projects.”

Norman Rosenblum

Past ROC Chair & CIHR INMD Scientific Director

“As a researcher, it is common to participate in review processes whereby we attempt to constructively improve the quality of research projects. ROC was a distinctly different and enriching experience for me because it embraces two major values: research projects must be constructed towards benefit for patients, and (ii) each aspect of review and advice must be generated in true partnership with patients and patient partners. These values brought me to the coal face of ‘social responsibility in research’ in a unique way that had a major positive impact. I feel great appreciation for the opportunity to focus my efforts and to have engaged with fellow researchers and patients in this way.”

Note. Can-SOLVE CKD = Canadians Seeking Solutions and Innovations to Overcome Chronic Kidney Disease; ROC = Research Operations Committee; CIHR INMD = Canadian Institutes of Health Research Institute of Nutrition, Metabolism and Diabetes.

Discussion

Applications and Originality

As POR becomes more mainstream, there is a need for clear mechanisms to support the inclusion of patient partners in research. While some research organizations include patient partners on a preplanning basis or postcompletion basis, the approach used by the ROC allows for involvement of patient partners throughout the research process via its annual check-ins. This means that patient partner feedback can help identify any problems in the midst of a project and allow for timely course correction.

Including both a researcher and patient partner in the review process also ensures that patients have equal contributing voices and provides an accountable framework for achieving POR across Can-SOLVE CKD’s 18 research projects.

Mechanisms for Success and Broader Implementation

As a SPOR initiative, Can-SOLVE CKD benefits from having a clearly defined overarching purpose from its inception: to place patients at the center of research planning and innovation.

Importantly, the diversity and expertise of ROC members brings value into the research process. Through structured meetings and an inclusive environment, committee members can discuss key issues and identify solutions. A respectful and inclusive environment at ROC meetings keeps reviewers engaged and motivated to take part in the review process. The committee built the structure of the review process together and through a mindset of listening, learning, and leading, which allows for opportunities to improve the structure and function of the committee to be continuously identified. Furthermore, there was an understanding early on that the activities, processes, and structure of the ROC would evolve and change over time, allowing a high degree of flexibility and adaptability.

The success of the committee is also built upon the readiness of the renal research community in centering the patient perspective and fostering relationships across various backgrounds and skill sets. Training programs, such as the CIHR foundations in POR and St. Michael’s Hospitals online course titled Partners in Research, forged a path for the work of the ROC. These programs equipped the committee and project teams with a framework for how to work in collaboration with one another.

While implementation of the ROC review process has been successful so far in the context of Can-SOLVE CKD because of leadership, inclusiveness and the patient-centered approach of the network, the overarching framework developed here could be applied to other networks and research institutions. In Canada, the other 4 SPOR initiatives are perhaps best poised to adopt this framework because they have similar structures and goals, but analogous approaches using an ROC could be more widely applicable to research institutions across Canada and beyond.

Of note, Can-SOLVE CKD’s 18 research projects are ongoing, as is evaluation of the ROC. Can-SOLVE CKD will continue to monitor the success of the ROC review process and its outcomes. Previous studies have highlighted the positive impact of including patients in the peer review,⁸ which is also reflected in this interim assessment of the ROC.

Feedback and Improvements

Suggestions for improving the experience and outcomes of the ROC included beginning the review process earlier in the design phase of the project. This would ensure alignment of patient-centered values and foster the relationship between projects and the ROC earlier on in the process. In addition, reviewers suggested that there be training of new patient partners to create sustainability of the committee and extend the opportunity of reviewing to others.

As the POR projects move through their lifecycle, the ROC fully anticipates that the review process will require modification and changes to respond to key aspects of the research work plan. This is in alignment with the core mandate of Can-SOLVE CKD (“Listening, Learning and Leading”) and enables the evolution of impactful POR research.

Evaluating the efficacy of the ROC in the research process remains a challenge. The ROC and even POR is constantly evolving. While anecdotal evidence suggests that the ROC process has been well received by both patients and researchers, we plan to seek formal feedback from all participants through surveys to determine if the ROC process and recommendations have been beneficial.

Conclusions

With patients as key members of the ROC, this committee and its review process furthers the Can-SOLVE CKD mission of causing a paradigm shift toward patient-centered research and care. This framework ensures that patient involvement and input occurs more consistently throughout the research process. Strong leadership within the committee and a readiness for patient-centered research within the wider health care community has allowed the ROC to establish a model for patients as equal and contributing partners in the review process and incorporate patient feedback throughout the research process. A similar ROC framework could be considered by and adapted for other organizations to provide patient partners the platform to shape the research that ultimately impacts them.