Involving Patient Partners in the KRESCENT Peer Review: Intent, Process, Challenges, and Opportunities

Abstract

Purpose of review:

The Kidney Research Scientist Core Education and National Training (KRESCENT) is a national Canadian training program for kidney scientists, funded by the Kidney Foundation of Canada (KFOC), the Canadian Institutes of Health Research (CIHR), and the Canadian Society of Nephrology (CSN). We describe our first year of incorporating patient partners into a scientific peer-review committee, the 2017 committee to select senior research trainees and early-career kidney researchers for funding and training, in the hope that it will be helpful to others who wish to integrate the perspective of people with lived experience into the peer-review process.

Sources of information:

Other peer-review committees, websites, journal articles, patient partners, Kidney Foundation of Canada Research Council, Canadians Seeking Solutions and Innovations to Overcome Chronic Kidney Disease (Can-SOLVE CKD) Patient Council, participants in the 2017 Kidney Foundation of Canada KRESCENT peer-review panel.

Methods:

We describe our motivation, rationale, guiding principles, plans, feedback, implementation, and response.

Key findings:

We disseminated a “call for patient partners” 8 weeks before the meeting, seeking patients or their care givers to partner with the KRESCENT peer-review panel; we defined these people with lived experience of kidney disease as patient partners. Eight patient partners came forward and all participated as reviewers. Patient partners first participated in a webinar to learn about the function, structure, and processes of a peer-review committee. They practiced reviewing plain language summaries and giving feedback. In a subsequent teleconference, they shared and discussed their reviews. Plain language summaries were scored, overall, on the same 0-5 quality scale used by scientific reviewers. Three patient reviewers participated in some or all of the 6-hour meeting, which was conducted as usual, for this panel, by teleconference (initially audio only; from 2020 onwards by videoconference). In the meeting, the 2 assigned scientific reviewers first gave their scores, followed by the patient reviewers giving their scores, and discussion (mostly scientific, and conducted in usual scientific language). Scientific reviewers then negotiated a consensus score based on their initial scores, the discussion, patient reviewers’ scores and statements, and the scientific officer’s notes. Patient reviewers, scientific reviewers, and the Kidney Foundation of Canada (KFOC) were generally positive about the process. The increased length of the meeting (estimated at 1 hour) was generally thought to be acceptable. Patient reviewers also provided feedback on the methods used to incorporate patients into the research under review. These comments were concrete, insightful, and helpful. The patients did not uniformly recommend that basic scientists involve patients in their work. We did not detect bias against preclinical science, work that did not involve patients, or rarer diseases. Some patients found participation inspiring and enlightening. All participants appreciated the idea of patient partners as community witnesses to a group process committed to fairness and supportiveness. We discussed assigning formal meaningful weight to patient reviewers’ assessments. Most, but not all, patients thought that the scientific reviewers were ultimately the best judges of the allocation of scarce research resources.

Limitations:

Patient participants tended to be Caucasian, middle class, and well educated. Because of the difficulties of travel for some people living with or supporting those living with kidney disease, our findings may not generalize fully to peer-review meetings that are conducted face to face. This is explicitly a supportive panel, committed to reviewing junior scientists with kindness as well as rigor; our findings may not generalize to panels conducted differently. We did not use formal qualitative methodology.

Implications:

Inclusion of patient partners as patient reviewers for the KRESCENT program peer-review panel was feasible, added value for scientific and patient reviewers, and for the funding stakeholders (CIHR, KFOC, and CSN). We were glad that we had taken this step and continue to refine the process with each successive competition.

Purpose of Review

In 1995, the BMJ published a recommendation to researchers to involve patients in the planning and dissemination of health research.¹ In the same issue, Ian Chalmers persuasively argued, with examples, that health research with patient involvement would align better with what patients considered important, and indeed, that the meaning and implications of health research cannot be fully understood without a patient perspective.² Sandra Oliver added that patients are uniquely placed to bridge the gap between the new knowledge and its implementation, by presenting the needs and views of patients to researchers.³ Uptake of these ideas in the United Kingdom and the United States has led to structural changes involving patients or consumer groups in the design and dissemination of clinical studies.⁴ In Canada, the Canadian Institutes of Health Research (CIHR) launched the multi-million-dollar Strategy for Patient-Oriented Research (SPOR) initiative in 2010, seeking to “transform the role of patient from a passive receptor of services to a proactive partner who helps shape health research and, as a result, health care.”⁵ One of the main tenets of this SPOR program is that patients must be included in all aspects of research, including the peer review. Canadians Seeking Solutions and Innovations to Overcome Chronic Kidney Disease (Can-SOLVE CKD) is the kidney-centered SPOR chronic disease network funded through this program. CIHR has included patients as peer reviewers for all SPOR-related research programs since SPOR’s inception. Patients provide feedback and a score on the appropriateness and level of patient engagement in each research project, but do not review scientific content of the applications.

The Kidney Research Scientist Core Education and National Training (KRESCENT) Program, a national partnership of the KFOC, Canadian Society of Nephrology, and CIHR, was launched in 2005 with 2 major objectives: to enhance kidney research capacity in Canada and to foster collaborative research and knowledge translation across the 4 pillars of health research.⁶ In 2016, the KRESCENT program partnered with Can-SOLVE CKD, to promote and enhance patient-oriented research in Canada, and to deliver education and coaching in how to involve patients in research as part of the curriculum for KRESCENT trainees.

The KFOC administers the KRESCENT program and coordinates the peer-review process that selects applicants for funding and training. The KFOC believes that incorporating patients in its work, including its research programs, is an important aspect of being responsive to its community. Given this, and the existing partnership between KRESCENT and Can-SOLVE CKD, the KRESCENT program peer-review committee was chosen, in 2017, to initiate the involvement of patients.

We describe the process used to incorporate patient reviewers within the KRESCENT peer-review process, how scientific reviewers and patients responded to the experience, and what has been learned from this experience. We involved patients or care givers with lived experience of kidney disease; we use the term “patient partners” to describe their partnership with the academic and administrative elements of the KRESCENT peer-review panel.

Sources of Information and Methods

The National Director of Research of the KFOC (“the director,” E.A.F.) worked closely with the Chair of the KRESCENT peer-review committee (“the chair,” C.M.C.) and several patient partners to determine the process for including patient partners in peer review and to create feedback forms, background and educational materials (Table 1). The process began with an online search regarding the incorporation of patients into peer review to determine best practices. We searched using PubMed and Google, using variants of the keywords “incorporating patients,” “including patients,” and “peer review.” The director spoke with other health charities that had already incorporated patients into the research review process and initiated a discussion at the CIHR-led “Community of Practice in Peer Review” (a gathering of Canadian research funders, including health charities and provincial funding agencies).

Table 1.

Process and Timeline.

Event	Timing
Environmental scan and discussions with other organizations	6 months in advance
First draft of documents, educational and coaching materials and process created	6 months in advance
Evaluation and feedback	5 months in advance
Second drafts created	4 months in advance
Process and documents finalized	3 months in advance
Call for patient partners sent	8-10 weeks in advance
Coaching webinars and teleconferences, practice summaries provided and reviewed	6 weeks in advance
Assignments sent to patient and scientific reviewers	4 weeks in advance
Preliminary scores received	2 days in advance
Peer-review meeting	May 2018
Scores and written reviews finalized	1 week post meeting
Debrief teleconferences with patient and scientific reviewers	2 weeks post meeting
Applicants informed	4 weeks post meeting

A first draft of the proposed process for incorporating patients in the peer review, patient partner review documents, educational materials, evaluation forms, and a coaching webinar were created. The educational materials took the form of power point slides explaining the role of the KRESCENT program, the role of the peer-review committee, the detailed processes of the peer-review committee before the meeting and during the meeting, the criteria by which the applications would be judged scientifically, some possible criteria for judgment by patient partners, and the nature of peer review. Five patient partners, the KFOC Research Council, and the Curriculum Chair of the KRESCENT program (A.L.) provided unstructured feedback and advice. A second round of feedback was provided by the Patient Council of Can-SOLVE CKD, and its executive, and 2 individual patient partners. The chair underwent Foundations of Patient Oriented Research Training (a 1-day workshop provided by Can-SOLVE CKD).^7,8

Throughout, our guiding principles were those articulated by the CIHR SPOR initiative: inclusiveness, support, mutual respect, and co-building.⁹

There was debate in the working group about how to incorporate patient partners’ ratings. In the first iteration, patient partners were to provide a score, on the usual 5-point CIHR scale, that would affect the final scientific reviewers’ score by a proportion of ±0.2, which can be the difference between a funded and non-funded application.¹⁰ This was not endorsed by participating patient partners, however. A minority of patient partners thought this weighting insufficient, among whom one suggested that the weighting should be equal between scientific reviewers and patient partner reviewers. However, the prevailing view among the patient partners was that their ability to change scores by 0.4 (±0.2) was too high a weighting, given that smaller differences than this usually separate fundable from non-fundable applications. These patient partners’ main idea was that the best applications should be funded, based on the scientific review, and they did not want their judgment, based on the lay summary, to promote a flawed application over one judged as excellent by scientific reviewers. Patient partners also did not think they had the skill set to make a weighted or numeric contribution. After discussion, it was decided that patient partners, like scientific reviewers, would use the 5-point CIHR scale, with the same anchors,¹⁰ to rate the domains, to have their feedback included in the overall discussion, and that the scientific reviewers would take their comments into account when they were coming to consensus on the meeting score.

In response to patient feedback, the evaluation form also changed again at this point: additional language and clarification about the domains were included and patients changed the wording significantly to make it more accessible, clear, and appropriate.

Key Findings

Preparation

A “call for patient partners” was made through various web and social media platforms (KFOC, Can-SOLVE CKD & the Regional Patient Networks), asking for interested patients living with CKD, and their caregivers, to come forward as patient reviewers. All those who came forward were identified through Can-SOLVE CKD. Six weeks before the meeting, 8 patient reviewers completed a 1-hour coaching webinar led by the director and the chair who explained the peer-review process, introduced the evaluation forms, and explained the task of reviewing the lay summary (with the full application available) for the domains of clarity, relevance, and level of patient involvement in the research program, if appropriate. Patient reviewers then reviewed lay summaries from successful applicants in the previous competition (with the applicants’ permission) and rated them for practice; a second coaching webinar was held to discuss ratings and to improve calibration. Feedback from patient reviewers was sought at the end of the webinar, via email and telephone calls. At this point, patient reviewers expressed their readiness to begin the process for the 2017-2018 competition. All peer reviewers (scientific and patient) were asked to sign a confidentiality form in advance of receiving their applications for review. Owing to scheduling issues, 1 patient partner did not participate further, and 7 patient partners continued forward from this point.

The director and chair explained the intent and process of involvement of patient reviewers to the scientific reviewers as part of their orientation to the meeting. We did this through emails. Intent was defined as the desire and plan to add the perspective of patient partners to the existing peer-review process.

About 4 weeks before the meeting, each patient reviewer was provided with lay summaries from 3 to 5 applicants as well as feedback forms and asked to review the summaries for clarity, relevance, and where applicable, the patient engagement plan. Patient reviewers were also provided with access to the full application if they wished to gain more information. Patient reviewers were asked to provide a score from 0 to 5 that aligned with the scientific reviewers’ scoring system. All applications were reviewed by 2 patient reviewers. There were 7 patient partners for about 20 applications, in duplicate, which resulted in about 6 applications each. All scores and reviews were submitted to the KFOC research associate a minimum of 2 days in advance of the meeting.

No financial compensation was provided to patient partners. The academic members of the panel were also unpaid volunteers.

Peer-Review Meeting

Patient reviewers were polled to see if anyone was interested in attending the peer-review meeting: 3 accepted and participated for some or all the meeting. The KRESCENT peer-review meeting takes place by teleconference call and included 10 scientific reviewers in addition to the chair, scientific officer, CIHR observer, KFOC research associate, KFOC director of research, and patient reviewers. Initially this meeting was audio only; from 2020 onwards it was conducted by videoconference. At the beginning of the meeting, the chair explained the process and responded to any questions. She reminded the patient reviewers of the domains that the scientific reviewers would be addressing to provide their scores, and the need for the discussions to remain at a technical level. She also explained to the scientific reviewers the domains that the patient reviewers would be assessing and reminded the committee that the patient reviewers would not be voting at the end of each application.

For each application, the 2 scientific reviewers assigned the application provided their scores, then the 2 patient reviewers provided theirs (Figure 1). The first and second scientific reviewers discussed the rationale for their score and discussion from all reviewers ensued. Both patient reviewers then provided their feedback on clarity and relevance. Patient feedback scores and text were read by the chair if the patient reviewer was not present, this happened in about half of cases, overall. For some grants, neither of the patient partners who had reviewed the grant was present. The scientific officer (N.J.) read back her notes, and there was further discussion if needed. A consensus score was determined by negotiation between the 2 scientific reviewers, taking both scientific and patient feedback into account. All scientific reviewers voted ±0.5 around the consensus score (meeting score). The final score was determined by taking the average of all the meeting scores. We did not assess the agreement between scores or attempt to determine formally whether either the patient partners’ scores or their contributions to the discussion affected the scientific reviewers’ consensus score or their meeting score.

Figure 1.

KRESCENT peer-review meeting process.

At the end of the peer-review meeting, the group had a discussion and provided a recommendation for funding.

Debriefing

Two weeks after the peer-review meeting, separate debriefing teleconferences were organized with both the patient and scientific reviewers.

Response

In general, scientific and patient reviewers were very positive and supportive of the inclusion of patients into the peer-review process. However, both scientific and patient reviewers indicated that the process could be improved and provided suggestions.

The most common concern expressed by the scientific reviewers was their fear that patient reviewers would consider their participation tokenistic or disappointing since patient reviewers were not asked to participate in the “meat” of the discussion, which remained at the scientific level: “I found that it was probably hard for patients to be there, because at the end we did not do that much with their reviews, although they worked very hard on it.” Another scientific reviewer reflected that perhaps with more experience, practice and coaching patients’ judgment could be given more weight, adding “Patients have a lot to say about the relevance of a project (if scientifically feasible).” Overall, the scientific reviewers did not consider it appropriate for patient reviewers to have a great impact on who was funded, because of the specialized knowledge required to identify a good candidate proposing good science in a good environment.

Most patient reviewers expressed gratitude for the work done by the scientific reviewers. They expressed that they felt confident that the funding was being spent appropriately after seeing the amount of effort and detailed review of the scientific reviewers. They also recognized that it was difficult for scientific reviewers to take their feedback into consideration. Patients acknowledged that they are unable to appropriately assess a scientific application by looking only at the lay summary: “As a patient, how can I judge who their mentors are, and what publications they have written? I relate to it as how it would impact me.” “I feel it is appropriate that patients don’t score. It is not fair to the applicant if we do the scoring. Patients don’t have the understanding or scientific knowledge to provide a score. Patients can’t possibly know everything. It would take away hugely from the process if patients were to add their score.” Some patient reviewers felt that it was an inspiring and enlightening experience.

The patient reviewer who had thought that patient input should be weighted at 50% continued to participate fully. In debriefing, this reviewer also acknowledged the breadth of knowledge of the scientific reviewers and, while continuing to advocate for more weighting in the future, was content with the process as delivered.

Some scientific reviewers expressed concern about the length of the peer-review meeting. The peer-review meeting takes place by teleconference call and normally lasts between 4 and 5 hours. The inclusion of patient reviewers added about 60 minutes to the meeting, which increased the time commitment.

All patient reviewers agreed that the applicants to the KRESCENT program needed help writing a summary of their work in language that was clear and understandable to the public. They also requested that the name be changed from “lay summary” to the more descriptive “plain language summary.” As a result, a “Best Practices for Writing Plain Language Summaries” document was created by the patient reviewers. This document is available on the KFOC website and is referenced in all KFOC application forms (Appendix A). Applicants to the KRESCENT program in the first year were not aware that their plain language summary would be specifically reviewed by patients, though they knew that it would be read by the general public if their application was successful, and that it would be considered by the peer-review panel, which formerly had consisted entirely of academics.

Scientific reviewers and patient partners both endorsed the value of “witnessing” the work of the committee. Patient partners also thought that their involvement had improved their knowledge and understanding of kidney research and thought they would be better research advocates as a result.

We observed no bias against preclinical research or against rarer diseases, nor did we encounter suggestions from any patient reviewer strongly advocating for research on one type of kidney disease over another.

Patient reviewers also asked to have the coaching webinar expanded to include more fully the details of how the scientific reviewers judge applications. In addition, they suggested that a new section to the peer-review application be created in which the applicant outlines how they have engaged patients in the creation of their research question, where relevant.

The patient reviewers requested that they be allowed to provide 2 scores: one for the clarity of the lay summary, and a separate score for the relevance of the research project. In many cases, the summary was not clear; however, the project was highly relevant, and the patient reviewer had to choose how to provide a single score for these conflicting parameters.

There was clear tension between our aims. Everyone involved, including the scientific reviewers, wished to involve patient partners meaningfully. The need for increased time on each application was understood and accepted by all. However, leadership (E.A.F. and C.M.C.) were aware that as the duration of the time commitment to the panel increased, there was an increasing possibility that it might be difficult to recruit scientific reviewers or patient partners. If everyone had been prepared to spend yet more time, it would have permitted deeper and more impactful conversations.

Further Consultation and Feedback

The results were presented to the KFOC Research Council. A small working group comprised of KFOC Research Council members and patient partners reviewed the feedback provided and recommended a course of action for the inclusion of patient partners in the 2018-2019 KRESCENT competition.

Discussion

We found it feasible to include patient partners in a peer-review process and meeting. They expressed their thoughts on the clarity, relevance, and (where applicable) integration of patients into research. No numerical weight was assigned to their contribution, and the merits of the application were judged on the basis of applicant, environment, and project as assessed by the scientific reviewers, along with incorporation of patient reviewers’ statements and relevance scores. To our knowledge, this is the first description of methods and results from including patient partners in a peer-review panel.

Patients have also been incorporated into the peer review of manuscripts for several years. For example, the BMJ has had patient reviewers as part of its Patient and Public Partnership strategy since 2014.^11,12 These reviewers are asked to provide feedback to the authors on: relevance and importance of the study’s aims; the feasibility of the treatment or intervention being studied; whether the outcomes being studied were appropriate; the level of patient engagement; and any suggestions to the authors. The Canadian Medical Association Journal open incorporates patient review in manuscripts considered for its patient-oriented clinical research collection.¹³

In the United States, patient partners were incorporated into peer review at the Patient-Centered Outcomes Research Institute (PCORI) since 2016.¹⁴ Their rationale is: “The idea is that, along with asking scientists about the studies, we consult others with practical expertise and lived experience about the relevance, usefulness, and patient-centeredness of the studies.” However, PCORI asks patient reviewers to assess completed PCORI-funded studies by analyzing the draft final research reports that PCORI’s awardees submit when they finish their projects, rather than being included in formulation of the questions or the peer-review process of deciding which awardees are funded.

In Canada, some health charities have been including patient reviewers in their peer-review processes for the past few years. For example, the Alzheimer Society has included 2 community representatives in their peer-review committees whose role is to comment on the “intent, purpose and on the clarity of the language used within the lay summaries” of all applications received.¹⁵ The Alzheimer Society indicates that they feel the inclusion of community members in peer review serves as a mechanism for public accountability.¹⁵ However, this experience has yet to be formally written up as a peer-reviewed manuscript.

Since we completed this work, a peer-review group from Alberta have published their experience.¹⁶ The Addiction and Mental Health Strategic Clinical Network (AMH-SCN) used a standard peer-review process to identify grants that were in the fundable range and then asked a panel of patients with lived experience of addiction and mental health problems to rank the applications. The steering committee took the ranking into account when making the final funding decisions, but it is not clear whether they followed the rank order exactly. Patient partners provided feedback that they worried that they had requisite skills and worried whether they might be biased; they asked for feedback on their reviews.

We also later became aware of the model used by Capital Health in Alberta, which involves researchers, administrators, and patients in the review as follows (Nancy Verdin, personal communication): Patients are given the full application for funding, but do not write a review. They are asked to attend the meeting to discuss how they reacted to the application. Scientific reviewers then take this information into account in the discussion, very similar to the approach we had taken.

We must be aware of the dichotomy of opinions with respect to patient involvement in research within the patient community. Some patients feel that they are being pushed too much and do not wish to contribute more, while others feel that their ability to participate in research is still too limited. One of the biggest challenges facing organizations that wish to engage patient partners in research will be to manage expectations and to take the time to work with individual patients to find the role that suits them best. Everyone involved will need to support future patient partners who may feel “unnecessary pressure”¹⁶ if they have concerns that they are unequal to the task in any way.

In addition, care must be taken to be realistic and open about the potential impact a patient partner might have on peer-review decisions. Patient engagement in research is a relatively new endeavor and while patients are experts in living with their condition, and the impact of kidney disease on themselves, their families and caregivers, they usually do not have the scientific background required to assess a scientific research application. When patient partners become more knowledgeable about the research process, meaning that they understand research frameworks and have familiarity with common processes, and when patients are empowered to participate to a greater degree, they will understandably expect to have a greater impact on the outcome. Therefore, great care must be taken to manage expectations, and not to “over-promise.” Those who choose to participate in the research process are often educated, passionate, and driven individuals, and there is a risk of disenchanting them. Scientific and patient reviewers must continue to work collaboratively, to openly discuss options and to continuously improve the proposed processes.

At the same time, some patient reviewers expressed their discomfort with being asked to provide a score at all: “I was scared when I realized that my words might make the difference between funded and not funded. That is a lot of power and I didn’t feel qualified to make that assessment.” The role of the patient reviewer must be clearly explained at every step, and must remain clear throughout the process, to avoid feelings of being overwhelmed or unequal to the task on the part of the patient partners. The patient partners’ unique perspective and the value of that perspective must be reinforced and embraced, ie, that their lived experience is what is important. “Reinforce to patients that they already have the skills that they need to do this properly. The content expertise is what you are; simply being a patient.” Providing more education around research topics to patient partners, or specifically selecting patient partners to be involved in peer review from those who have experience in research through, for example, serving on steering committees, would result in more knowledgeable participants at peer review, but is in tension with the idea that the value that the patient partner brings is from their lived experience itself.

In the education and coaching we provided, we were explicit that we did not expect patients to represent others: to do this adequately would require consultation. We were also clear that this was not an advocacy role for any particular subset of patients with kidney disease. We asked patients to be who they are; to allow their lived experience to inform their thoughts about a particular application and to share that with us. When the plain language summary clearly described the problem to be addressed, patient partners appreciated the relevance of the work to people living with kidney disease. Sometimes, but not always, they reflected on their own experiences.

Where applicable, patient partners made helpful and concrete suggestions for improvement of patient involvement in the planned research. The applications that described patient involvement were mostly clinical. The patient partners did not suggest that basic science programs should uniformly adopt patient partnerships.

During the de-brief calls, patient and scientific reviewers were asked how to improve the process. There was no consensus. Suggestions for alternative methods of weighting the patients’ contributions included: (1) asking patient reviewers to provide a score for the clarity of the lay summary and allowing this score to contribute a percentage of the overall score (10%) and (2) implementing a 2-stage process, whereby scientific reviewers perform the initial review to determine which applications are in the fundable range, and patient reviewers determine the order in which the applicants are funded. For the second iteration of the process in 2019, we decided to keep the process the same as before, but to separate the domains of relevance and clarity in terms of scoring as well as feedback. We discussed the relevance score at the meeting. We thought it was very important for the applicants to get feedback, including a score, on the clarity of their plain-language summary since these are often used by funding agencies to explain funded scientific work to the general public. However, we decided that compared with the merits of the candidate, the environment, and the scientific proposal, the quality of the writing of lay abstract should not affect fundability. Based on this decision, and our concerns about time constraints for all the volunteer panelists, we did not discuss the lay abstract at the meeting.

While a lot of effort was expended speaking with the patient reviewers to discuss the process, using educational materials and formal coaching, less effort was expended discussing the process and expectations with the scientific reviewers. In the future, a greater effort will be made to include the scientific reviewers in these discussions.

Strengths

Through CanSolve, KFOC leadership was able to engage with patient partners at an early stage in the process and their input was used and useful at every stage. We provided explicit education and coaching on the peer-review process and collected feedback for further improvement. Stakeholders worked together to write and improve this document. To our knowledge, there is one previous report of such a process.¹⁶

Challenges and Limitations

Although we started with an informal literature review and outreach to other organizations, we did not have the resources to begin with a formal systematic review of the literature or to conduct a formal scoping review of the landscape in health research funding in Canada. One of the most cited limitations to including patient reviewers is that patients who wish to engage in research tend to be quite homogeneous: we were unable to find any data addressing the question, but there is a concern in the research community that participants tend to be Caucasian, middle class, well educated, and older individuals. This was largely true of the group that participated in the KRESCENT peer review. Policies that improve health literacy may bring a number of benefits, one of which might be to facilitate patient engagement in research.¹⁷ We sometimes observed quite large differences between patients’ scores for the same application. Scientific reviewers also sometimes start far apart (called the “initial score”); the gap is narrowed by discussion and consensus is reached (“final score”). We do not know whether patients’ scores would be more different still if patients were drawn from a more diverse population. Finally, because this is a voluntary process, consuming time and energy, the patient partners participating may be healthier than average; or, recognizing that many have been through darker days, they are perhaps at a stable phase of their illness, but may have had experience of other, more challenging illness in the past.

The KRESCENT peer-review meeting is organized as a teleconference for economy and feasibility, facilitating the participation of people who have many demands on their time, by reducing travel time. The participation of patients in meetings would be very different for other committees which meet face to face, particularly considering the practical difficulties of travel for patient partners who are currently on dialysis or who support someone on dialysis. Some of our findings may not be generalizable to other committees for this reason.

Although we had placed calls for partners through various web and social media platforms (KFOC, Can-SOLVE CKD and the Regional Patient Networks), all the patient partners who came forward were active in Can-SOLVE CKD, a national program which explicitly aims to involve patients in research at all levels. Without this program, it is possible that our call would have been unsuccessful. Other disciplines may not have such a valuable resource available. We recognize the problems and biases that arise in any method of seeking patients and their caregivers. Two additional options that we did not explore were word-of-mouth from clinicians to patients and caregivers, and poster advertising in nephrology clinics and dialysis units.

This is a program report which aims to share our processes and experiences with others who may be contemplating involving patients in peer review. We recognize that this could have been approached as original research using a variety of designs, but we had no resources for this, and it was not our intent. Our work therefore falls short of the expectations of formal research. We had no formal process for feedback on our processes, but rather asked for it in an unstructured way. We did not have a formal project plan, but rather created the materials that we needed as we needed them. We did not have a formal process for the decision on how to incorporate the patients’ scores. We did not have the resources for either qualitative or quantitative formal evaluation of the projects processes or outcomes.

The KRESCENT peer review committee is a panel dealing with applications for training from junior researchers: because of this, the atmosphere and philosophy of the committee is explicitly supportive and helpful, and scientific reviewers are chosen for their ability to express critical rigor in language that is thoughtful and kind. Our findings may not be generalizable to committees operating on different principles. It is also possible that these scientific reviewers may be more tolerant than the average scientific reviewer of the increased time commitment resulting from the incorporation of patient partners.

This work is a narrative review, including some direct quotations from participants. We did not use formal qualitative methods to classify and understand all the ideas expressed by participants. This paper was drafted by the research director and chair; we have attempted to control bias by involving all scientific reviewers and patient partners in the revision and refinement of the manuscript.

Conclusions

Inclusion of patient reviewers into the peer-review process for the KRESCENT program was challenging, but feasible, and added value for some scientific and all patient reviewers, and for the KFOC. We all acknowledge that it was a first attempt and recognize room for improvement. There are advantages and disadvantages to each possible idea for further change. We hope that our experiences help others embarking on the same path.

Appendix A

Best Practices for Writing a Plain Language Summary

Below you will find suggestions and recommendations from patient partners who have experience reviewing plain language summaries. Please use the following as your guide:

• Consider writing your plain language summary using one or a combination of the following formats:

First suggested format:

• Purpose: What is your research about?

• Procedure: What are the steps required to accomplish the goals of your proposed research?

• Outcome: What are the results you are expecting?

• Relevance to patients: How will your research improve or impact the lives of patients? What is the nature and extent of the problem you are trying to address?

Second suggested format:

• Background

• Method

• Conclusion

• Please use headings to guide context and understanding. They also break up the text and make your summary easier to read.

• Indent and separate new ideas. When changing ideas or topics use an indent or a space or begin a new paragraph.

• Please use plain language. This will allow lay people insight into your ideas. (Please refrain from clarifying to the point of patronizing). Keep in mind that the reader will not be familiar with technical terms. Write as you would for a newspaper.

• Legend/glossary: Consider including a legend of terms and acronyms.

• Acronyms: Acronyms must not stand alone. Include a brief explanation of the acronym and terms that you are using. This will bring clarity to your document.

• Use of analogies, similes, and metaphors: Using a metaphor or analogy to help explain your document may be helpful. These tools are only helpful when they are relevant to pertinent concepts.

• Provide context: It is helpful to include information about the depth or extent of the issue your research will address. In addition, include information related to how your research will improve patient lives.

• Choose a reader: Consider asking someone who does not work in research to read your plain language summary before finalizing it.

• If someone doesn’t understand what you are writing about, it becomes ineffective.

We are looking forward to receiving your summary!

This document was created thanks to the insights and contributions of the patient partners who participated in the KRESCENT peer review for the 2017-2018 competition: Karin Bell, Angela Chiazzese, Gwen Herrington, Sandi Kidston, Anne MacPhee, Shanda McCutcheon, Nancy Verdin, and Hans Vorster.

Compiled by Catherine Clase and Lis Fowler

Appendix B

Evaluation Form, Final, After Feedback From 2017 Competition, and Used in 2018 Competition

Applicant Name:

Project Title:

Plain Language Summary:

1. Clarity: Is this research summary clear? Did the research applicant clearly explain the intent and importance of the research?	Yes___ Somewhat/Partially____ No___ I’m not sure____ Comments:
2. Relevance: In your opinion, do you believe that patients will consider this research important? Is the research summary missing any key components? Do you feel you or other patients will benefit from this research?	Yes___ Somewhat/Partially____ No___ I’m not sure____ Comments:
3. Patient Engagement: Does this project have a patient engagement component?	A. Yes___ B. Somewhat/Partially____ C. No____ D. I’m not sure____
Patient engagement: If you responded to Question 3 checking “A” or “B”. . . . . . feel this project has a patient engagement component, we welcome your opinion: Is the engagement appropriate? Is there a way patients could be more involved?	Comments:
Patient engagement: If you responded to Question 3 checking “C” or “D”. . . You feel this research does not have a patient engagement component, we welcome your opinion: Should patients be involved in this research? If yes, how many patients be more involved?	Comments:
4. Feedback to the applicant: Is there anything you would like the researcher(s) to know?
Any comments, compliments, questions or concerns for the Peer-Review Committee?

YOUR SCORE For Clarity:________(out of 5—see table below)

YOUR SCORE FOR RELEVANCE:______(out of 5—see table below)

Descriptor	Score	Explanation
Outstanding	4.5-5.0	The summary was outstanding in all aspects: • The summary needs no changes: it is nearly perfect “as is” and is written in clear, understandable language. • The project will produce results that are very important to patients.
Excellent	4.0-4.4	The summary was excellent in all aspects: • The summary is very clearly written and understandable, but would be even better with a few small changes. • The project will produce results that are important to patients but could have been more clearly explained.
Very Good	3.5-3.9	The summary was very good in most aspects: • The summary is clearly written but needs some revisions and a number of areas need to be changed. • The project will produce results that are somewhat important to patients, and/or could have been explained more clearly.
Good	3.0-3.4	The summary is good in most aspects: • The summary is ok, but needs to be read multiple times before it is understandable. It needs some changes and revisions to be clear to everyone. • The project will produce results that are important only to some patients, and/or it is not clear how patients will be impacted.
Fair	2.5-2.9	The summary is only fair in most or all aspects: • The summary needs major revisions and changes. It was a struggle to read the summary. • The project will produce results that are of minimal importance to most patients and/or the importance is not clearly explained.
Unacceptable	0-2.4	The summary is not acceptable: • The summary is written in language that is incomprehensible. • The project will produce results that are not important to patients, and/or I could not figure out what the impact of this research will be.