Abstract
Background
How best to involve patients in the development of clinical practice guideline (CPG) recommendations is not known. We sought to determine the feasibility and value of developing CPG recommendations based on a voting panel composed entirely of patients, with the ultimate goal of comparing the patients’ recommendations to ones developed by a physician-dominated voting panel on the same clinical questions.
Methods
Ten patients with rheumatoid arthritis completed eight hours of training on evidence-based medicine and guideline development. They constituted a voting panel and, with two American College of Rheumatology staff with expertise in CPG development and a physician facilitator, subsequently met at a face-to-face meeting to develop recommendations. They applied the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) methodology to formulate recommendations on 18 questions for which there was evidence warranting moderate or high confidence.
Results
The patient panel developed recommendations for 16 of the 18 questions; for the other two, the panel thought there were insufficient data to support a recommendation. For 13 of the 16 questions, the patient panel recommended the same course of action as did the physician-dominated panel. Differences were due to how the two panels valued the balance between benefits and harms.
Conclusion
Patient and physician-dominated panels developed the same recommendations for most questions for which there was evidence warranting moderate to high confidence. Additional experiences are necessary to advance the evidence necessary to determine what panel composition is optimal to produce the best guidelines.
Clinical practice guidelines (CPGs) are valuable tools targeted at improving patient outcomes and decreasing unwarranted variability in the delivery of healthcare. Most CPGs are grounded in evidence, but their development involves tradeoffs between potential benefits, possible harms, and burden of treatment, which involve judgment. Because physicians and patients may value these factors differently, the National Academy of Medicine (formerly the Institute of Medicine) (1), Guidelines International Network (2), and the Appraisal of Guidelines Research and Evaluation in Europe all recommend including patients in the development and implementation of CPGs (3).
Several methods of including patients in the development of CPGs have been described. Some health services research experts advocate using preference data generated from cost-effectiveness models or derived from health-related quality of life measures (4). Studies quantifying preferences using other approaches, such as conjoint analysis, may also be suitable sources of patient preference data (5-9). Qualitative studies provide insights into the patient's perspective (4) but cannot quantify maximum acceptable risk or preference heterogeneity. The preceding approaches are all of limited practical use, however, because published preference data are rarely available for the specific clinical scenarios included in CPGs. Others advocate holding meetings to obtain patient feedback on preliminary draft versions of the guidelines (10), though it is unclear how patient feedback can be meaningfully incorporated once the literature search has been completed and the votes taken.
Many societies now include one or two patient representatives on their voting panels. This approach, while giving patients a voice, may have limited impact because patients do not have a sufficient number of votes to change the direction or strength of a recommendation. Moreover, patients may not feel comfortable defending a point of view that contrasts with those of the physician, health professional, and methodology “experts” who make up 80% or more of the voting panel. Lastly one or two patients may not be able to adequately represent the views of a heterogeneous population.
In this pilot project, we sought to determine the feasibility and value of developing CPG recommendations based on a voting panel composed entirely of patients, with the ultimate goal of comparing the patients’ recommendations to ones developed by a physician-dominated voting panel on the same clinical questions. Our goal was to better understand if there are differences between how a patient panel versus a physician-dominated voting panel weigh the information presented in evidence tables when developing recommendations.
The American College for Rheumatology (ACR) uses the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) method (11) to develop CPGs. The GRADE approach proceeds according to the following steps (12):
Generate relevant clinical questions.
Describe each question using the PICO format (Population, Intervention, Comparison, and Outcomes). For example: In newly diagnosed treatment-naïve adult patients with moderately active rheumatoid arthritis (RA) (Population), how effective are biologics (Intervention) compared to non-biologic disease modifying anti-rheumatic drugs (DMARD) (Comparison) in preventing erosions over a two year period (Outcome)?
Perform a systematic literature review.
For each PICO question, synthesize data from relevant studies by outcome, including benefits and harms (13).
Rate the confidence in the evidence for each outcome and for the PICO question overall, according to pre-specified criteria (14).
Create an evidence summary table for each PICO question that includes information about benefits and harms as well as the quality of evidence.
Develop specific recommendations based on data presented in the summary tables (15). Recommendations are described as 1) for or against a specific action and 2) being strong or conditional (16). For example, “In patients with established RA, we strongly recommend using a treat to target strategy over a non-targeted approach.”
The strength of each recommendation is based on the panel's evaluation of the balance between potential benefits and harms, and on the quality of evidence. Strong recommendations refer to those in which the benefits clearly outweigh the harms (or in the case of a negative recommendation, the harms clearly outweigh the benefits) and almost all patients in a specific situation would want the recommended course of action. In contrast, conditional recommendations are made when the balance between the benefits and harms is less certain. Uncertainty in the estimates of outcomes, magnitude of treatment effect, impact of specific risks, burden of treatment, and patient values may all influence the strength of a recommendation (17).
Given the large body of literature demonstrating that physicians do not accurately predict their patients’ values or priorities (5, 6, 18-20), we propose that the strength of each recommendation consider patients’ values, as directly expressed by patients with the disease or condition under consideration. To meet this objective, we assembled a panel of 10 patients and asked them to vote on a subset of recommendations for the treatment of rheumatoid arthritis (RA) also presented to an ACR voting panel that was comprised of nine physicians and two patient representatives. The patient panel followed the same voting procedures as the physician-dominated panel. However, for this pilot study, we included only the subset of PICO questions that were supported by moderate to high quality evidence, based on the assumption that patients would not feel prepared to make judgments in the absence of sufficiently robust data.
Methods
The RA patients who comprised the voting panel were identified primarily from the Patient Governor group of the Arthritis Patient Partnership With Comparative Effectiveness Researchers (AR-PoWER) Patient Powered Research Network (PPRN). AR-PoWER (now referred to as Arthritis Power) is a partnership of the Global Healthy Living Foundation's CreakyJoints® online arthritis support community and rheumatology researchers at the University of Alabama at Birmingham, and is part of the National Patient-Centered Outcomes Research Network (PCORnet), developed with support from the Patient-Centered Outcomes Research Institute (PCORI).
The 10-member patient panel included three men (all Caucasian) and seven women (five Caucasian, one African American, one biracial); mean (SD) age of 38.1 (9.0) years, range 29-56 years. All were college educated and six were currently employed. The average duration since diagnosis was 10.5 (7.5) years; four were on a traditional DMARD only, and six were on a biologic with or without a DMARD. The project was facilitated by the same two ACR staff persons who facilitated the ACR physician-dominated CPG development project (which took place two weeks prior to the patient in-person meeting) – a senior director responsible for all ACR guideline projects, and a project coordinator who oversaw administrative details – as well as a rheumatologist, all with expertise in CPG development and experience with GRADE methodology.
Prior to arriving at the in-person meeting, the 10 patients completed the Collaborative Institutional Training Initiative (CITI) Human Subjects Protection training and the Cochrane Collaboration's Understanding Evidence-Based Healthcare and Serving on a Clinical Practice Guideline modules, totaling approximately eight hours of training. In addition, patients received and confirmed access to an online polling mechanism that was used for voting during the meeting (Poll Everywhere).
Patients knew each other by name, as they had participated in several group conference calls related to their AR-PoWER activities, prior to the in-person meeting. The meeting was organized as follows: 1) a 2-hour introductory session during which “ground rules” were agreed upon (including: take care of yourself; there are no dumb questions; step up, step back (i.e., draw out others’ ideas); agree to disagree; acknowledge that everyone's experience is different; and one person speaks at a time) and an overview of the GRADE methodology and voting procedures were discussed, 2) a 7-hour voting session the next day, and 3) a 3-hour voting session and wrap up on the third day. Patients were invited to ask clarifying questions throughout each session. A 2-hour conference call was later held to address six specific PICO questions that had been revised or added by the physician-dominated panel. The patient panel did not have any knowledge of the physician-dominated panel's votes throughout the process.
Voting on the recommendations followed standard ACR procedures. For each recommendation, the panel used the following process: 1) the PICO question and accompanying evidence summary table was reviewed and clarifications of specific data were made, if necessary; 2) the panel then voted on the direction (for or against the recommendation); and 3) the panel voted on the strength of the recommendation (strong or conditional). Voting was anonymous and was performed using an online polling mechanism (Poll Everywhere) that was accessed via participants’ personal electronic devices. Additional rounds of voting were conducted until consensus was met (defined as a minimum of 70% of votes for the recommendation direction and then for the recommendation strength).
Participants were asked to make the same assumptions as the physician-dominated panel, i.e., that they were developing recommendations for patients who were candidates for the proposed therapies (i.e., without contraindications) and not to consider cost and access to therapy in their decisions. The materials used (PICO questions, evidence tables) were the same as those used by the physician-dominated group. The patient-panel did not have access to any of the physician-dominated panel's recommendations or discussions. The physician-dominated panel included two patient experts, neither of whom participated in the patient-panel. Details regarding the methods used (i.e., development of PICO questions, literature search, data abstraction, rating the quality of evidence, and development of the evidence summary tables) are detailed in the ACR RA guideline manuscript (21).
The patient panel was presented with all PICO questions associated with moderate to high quality evidence (n= 18). Some of the specific recommendation statements in this paper were not ultimately included in the final ACR RA guideline because of modifications made after the RA guideline physician-dominated panel meeting. The recommendations described in this manuscript are presented to illustrate a novel approach of including patients in the CPG development process and should not be interpreted as ACR recommendations for the management of RA.
Results
After the introductory session, the patient panel confirmed that they required at least moderate quality evidence to be able to generate recommendations, because they did not have the requisite medical expertise to use as evidence (which GRADE would classify as low quality). The patient panel was able to develop recommendations for 16 out of the 18 PICO questions with this level of evidence (See Table 1). They chose not to vote on two questions (#10 and #13) because they felt that they did not have enough direct data to support a recommendation. For 13 out of the remaining 16 questions, the patient panel voted in the same direction as the physician-dominated panel. The strength of the recommendation was the same across both panels for 10 out of these 13 recommendations.
Table.
Physician and Patient Panel Votes for PICO Questions Supported by Moderate or High Quality Evidence*
| 1. In patients with early RA with moderate or high disease activity, who have never taken a DMARD medication, we (strongly/conditionally) recommend (using/ against using) one DMARD medication over using combination double DMARD therapy. | |||
|---|---|---|---|
| Physician-dominated Panel Votes | Patient Panel Votes | ||
| Direction | Strength | Direction | Strength |
| Use a single DMARD over two DMARDs (Majority**) | Conditional (Unanimous) | Use a single DMARD over two DMARDs (Majority) | Strong (Unanimous) |
| 2. In patients with early RA and moderate or high disease activity, who have never taken a DMARD medication, we (strongly/conditionally) recommend (using/against using) one DMARD medication over using combination triple DMARD therapy. | |||
|---|---|---|---|
| Physician-dominated Panel Votes | Patient Panel Votes | ||
| Direction | Strength | Direction | Strength |
| Use a single DMARD over triple therapy (Unanimous) | Conditional (Unanimous) | Use triple therapy over a single DMARD (Majority) | Conditional (Majority) |
| 3. In patients with established RA with moderate or high disease activity who have never taken DMARD medications, we (strongly/conditionally) recommend (using/against using) one DMARD medication over using more than one DMARD medication. | |||
|---|---|---|---|
| Physician-dominated Panel Votes | Patient Panel Votes | ||
| Direction | Strength | Direction | Strength |
| Use one DMARD over two or three DMARDs (Unanimous) | Conditional (Unanimous) | Use two or three DMARDs over one DMARD (Majority) | Conditional (Majority) |
| 4. In patients with established RA with moderate or high disease activity who have failed DMARD monotherapy including methotrexate, we (strongly/conditionally) recommend (using/not using) combination DMARD therapy OR a non-TNF biologic OR TNF inhibitors OR tofacitinib (all choices without or without methotrexate). | |||
|---|---|---|---|
| Physician-dominated Panel Votes | Patient Panel Votes | ||
| Direction | Strength | Direction | Strength |
| Use additional traditional DMARDs or a biologic (Unanimous) | Strong (Unanimous) | Use additional traditional DMARDs or a biologic (Unanimous) | Strong (Majority) |
| 5. In patients with established RA and moderate or high disease activity who have never taken methotrexate, we (strongly/conditionally) recommend (using/against using) oral tofacitinib over methotrexate alone. | |||
|---|---|---|---|
| Physician-dominated Panel Votes | Patient Panel Votes | ||
| Direction | Strength | Direction | Strength |
| Use methotrexate over tofacitinib (Majority) | Conditional (Unanimous) | Use tofacitinib over methotrexate (Unanimous) | Conditional (Unanimous) |
| 6. In patients with established RA and moderate or high disease activity who have not responded to DMARD therapy, we (strongly/conditionally) recommend (using/against using) oral tofacitinib and methotrexate over methotrexate alone. | |||
|---|---|---|---|
| Physician-dominated Panel Votes | Patient Panel Votes | ||
| Direction | Strength | Direction | Strength |
| Use tofacitinib + methotrexate over methotrexate alone (Unanimous) | Strong (Majority) | Use tofacitinib + methotrexate over methotrexate alone (Unanimous) | Strong (Majority) |
| 7. In patients with established RA and moderate or high disease activity, who have not responded to DMARD medications, we (strongly/conditionally) recommend (using/against using) oral tofacitinib over TNF inhibitors.1 | |||
|---|---|---|---|
| Physician-dominated Panel Votes | Patient Panel Votes | ||
| Direction | Strength | Direction | Strength |
| Use a TNF inhibitor over tofacitinib (Majority) | Conditional (Majority) | Use a TNF inhibitor over tofacitinib (Unanimous) | Conditional (Unanimous) |
| 8. In patients with established RA and moderate or high disease activity, who have not responded to DMARD medications, we (strongly/conditionally) recommend (using/against using) oral tofacitinib and methotrexate over TNF inhibitors and methotrexate.1 | |||
|---|---|---|---|
| Physician-dominated Panel Votes | Patient Panel Votes | ||
| Direction | Strength | Direction | Strength |
| Use a TNF inhibitor + methotrexate over tofacitnib+ methotrexate (Majority) | Conditional (Majority) | Use a TNF inhibitor + methotrexate over tofacitnib+ methotrexate (Majority) | Conditional (Unanimous) |
| 9. In patients with established RA and moderate or high disease activity who are not on DMARD therapy, we (strongly/conditionally) recommend (using/against using) one or two DMARD medications in combination with TNF inhibitors over TNF inhibitors alone. | |||
|---|---|---|---|
| Physician-dominated Panel Votes | Patient Panel Votes | ||
| Direction | Strength | Direction | Strength |
| Use DMARD(s) + a TNF inhibitor over a TNF inhibitor alone (Unanimous) | Strong (Majority) | Use DMARD(s) + a TNF inhibitor over a TNF inhibitor alone (Unanimous) | Strong (Majority) |
| 10. In patients with established RA with moderate or high disease activity, who also have congestive heart failure, we (strongly/conditionally) recommend (using/against using) combination DMARD therapy over a TNF inhibitor. | |||
|---|---|---|---|
| Physician-dominated Panel Votes | Patient Panel Votes | ||
| Direction | Strength | Direction | Strength |
| Use combination DMARD therapy over a TNF inhibitor (Unanimous) | Conditional (Majority) | Patients refused to vote because all data were indirect. | |
| 11. In patients with established RA with low disease activity, we (strongly/conditionally) recommend (tapering/continuing) traditional DMARD therapy. | |||
|---|---|---|---|
| Physician-dominated Panel Votes | Patient Panel Votes | ||
| Direction | Strength | Direction | Strength |
| Continue DMARD (Majority) | Strong (Majority) | Continue DMARD (Unanimous) | Conditional (Unanimous) |
| 12. In patients with established RA and low disease activity who are currently on methotrexate and a TNF inhibitor, we (strongly/conditionally) recommend (tapering/continuing) TNF inhibitors. | |||
|---|---|---|---|
| Physician-dominated Panel Votes | Patient Panel Votes | ||
| Direction | Strength | Direction | Strength |
| Continue TNF inhibitor (Unanimous) | Strong (Majority) | Continue TNF inhibitor (Majority) | Conditional (Unanimous) |
| 13. In patients with established RA in disease remission who are currently on methotrexate and a TNF inhibitor, we (strongly/conditionally) recommend (tapering/continuing) TNF inhibitors. | |||
|---|---|---|---|
| Physician-dominated Panel Votes | Patient Panel Votes | ||
| Direction | Strength | Direction | Strength |
| Tapering TNF inhibitor (Unanimous) | Conditional (Unanimous) | Patients refused to vote because all data were indirect. | |
| 14. In patients with early RA and moderate or high disease activity, we (strongly/conditionally) recommend (using/against using) low dose glucocorticoid therapy in combination with DMARD therapy over DMARD therapy without glucocorticoids. | |||
|---|---|---|---|
| Physician-dominated Panel Votes | Patient Panel Votes | ||
| Direction | Strength | Direction | Strength |
| Use low dose glucocorticoids (Majority) | Conditional (Unanimous) | Use low dose glucocorticoids (Unanimous) | Conditional (Majority) |
| 15. In patients with early RA and moderate or high disease activity who are on a TNF inhibitor or non-TNF biologic, we (strongly/conditionally) recommend (adding/not adding) low dose glucocorticoid therapy over no glucocorticoids. | |||
|---|---|---|---|
| Physician-dominated Panel Votes | Patient Panel Votes | ||
| Direction | Strength | Direction | Strength |
| Use low dose glucocorticoids (Unanimous) | Conditional (Majority) | Use low dose glucocorticoids (Unanimous) | Conditional (Majority) |
| 16. In patients with established RA and moderate or high disease activity, we (strongly/conditionally) recommend (using/against using) low dose glucocorticoid therapy in combination with DMARD therapy over DMARDs without glucocorticoids. | |||
|---|---|---|---|
| Physician-dominated Panel Votes | Patient Panel Votes | ||
| Direction | Strength | Direction | Strength |
| Use low dose glucocorticoids (Majority) | Conditional (Unanimous) | Use low dose glucocorticoids (Unanimous) | Conditional (Unanimous) |
| 17. In patients with established RA and moderate or high disease activity who are on a TNF inhibitor or non-TNF biologic, we (strongly/conditionally) recommend (adding/not adding), low dose glucocorticoid therapy over no glucocorticoids. | |||
|---|---|---|---|
| Physician-dominated Panel Votes | Patient Panel Votes | ||
| Direction | Strength | Direction | Strength |
| Use low dose glucocorticoids (Unanimous) | Conditional (Unanimous) | Use low dose glucocorticoids (Unanimous) | Conditional (Majority) |
| 18. In patients with established RA, we (strongly/conditionally) recommend (using/against using) a treat to target strategy over a non-targeted approach. | |||
|---|---|---|---|
| Physician-dominated Panel Votes | Patient Panel Votes | ||
| Direction | Strength | Direction | Strength |
| Use a treat-to-target strategy (Unanimous) | Strong (Unanimous) | Use a treat-to-target strategy (Unanimous) | Strong (Majority) |
Patient panel: The panel noted that overall there is no difference in efficacy or safety, and that the burden of two drugs is greater than one.
Physician-dominated panel: The panel noted little difference in efficacy or safety, but because the burden of two drugs is greater than one, they concluded that the recommendation should be conditional because preferences might vary.
Patient panel: Strong advocates felt that an increased chance of remission justified triple therapy, given a lack of significant difference in toxicity. Conditional advocates felt that some patients would attach significant weight to the burden of taking three medications over one. Some patients also felt that starting three medications at once would make it difficult to tell which medication was responsible for possible side effects.
Physician-dominated panel: The panel concluded that the increased burden of three medications over one is not justified because the overall efficacy is similar over time (this fact was not described in the evidence table, and therefore was not discussed by the patient panel). However, it was recognized that triple therapy might be preferred by some patients who desire more rapid short term benefits (e.g., earlier resumption of work activities).
Patient panel: Patients felt that the increased ACR50 responses associated with double/triple therapy and lack of additional significant toxicity, favored recommending combination over monotherapy.
Physician-dominated panel: Panelists felt that the increased burden of double/triple therapy was not justified by the minimal difference in benefit that was shown in (indirect) evidence from early RA trials.
No discussion by either panel.
Patient panel: Patients retrieved the original study to obtain further information on the adverse events. They felt that the gastrointestinal side effects were worse with methotrexate, but that other side effects were of greater significance with tofacitinib, including the risks of elevated blood pressure and hypercholesterolemia. Patients also noted that tofacitinib was a newer agent and that less was known about this medication compared to methotrexate. However, ultimately, the panel voted in favor of tofacitinib because of fewer gastrointestinal side effects and strong evidence of improved outcomes. The vote was conditional based on the panel view that patients’ preferences should ultimately be the deciding factor.
Physician-dominated panel: Panelists felt that the balance of benefit and potential harms associated with methotrexate outweighed those of tofacitinib, particularly because the long-term safety of tofacitinib is not yet well known.
Patient panel: Patients had reservations about recommending tofacitinib, but did so based on the constrained choice presented in the PICO question. They noted that future studies might change this recommendation.
Physician-dominated panel: Panelists assumed that patients who had “not responded to DMARD therapy” would have taken methotrexate at some point. If the patients’ disease activity was still moderate or high despite this methotrexate use, then panelists felt it was inappropriate for them to continue on methotrexate alone.
Patient panel: The panel was uncomfortable voting on this recommendation, and they would not have voted if they had been given that option. Patients felt that although the ACR20 response was statistically significant for using a TNF inhibitor over tofacitinib, an ACR20 response is only marginally important to patients. The more meaningful ACR50 and ACR 70 responses did not differ. Patients also felt that the short duration of the study lowered the impact of the results.
Physician-dominated panel: There was no discussion prior to this vote.
Patient panel: As in the preceding question, the panel was uncomfortable voting on this recommendation, and they would not have voted if they had been given that option, because they felt there were insufficient data. Patients were frustrated that the available study didn't address an ACR50 response. The panel felt strongly that this decision should be completely dependent on each patient's preferences given the lack of data supporting the superiority of either option.
Physician-dominated panel: There was no discussion prior to this vote.
Patient panel: Some patients voted “conditionally” because of the increased risk of hepatotoxicity with combination therapy.
Physician-dominated panel: There was no discussion prior to this vote.
No discussion by either panel.
Patient panel: All patient panelists felt that preventing flares was important; however, the vote was conditional because the panel felt that patient preferences were important in determining whether to continue medications or to risk having a flare.
Physician-dominated panel: Panelists expressed concern about the possibilities of flares after tapering, noting that it was often difficult to get a patient's disease activity back under control after a flare.
No discussion by either panel.
No discussion by either panel.
No discussion by either panel.
No discussion by either panel.
No discussion by either panel.
No discussion by either panel.
Patient panel: While the panel was instructed to assume that patients would have access to treat-to-target strategies, they raised concerns regarding the feasibility of being able to arrange more frequent contact with their rheumatologists. The minority of patients who voted conditionally felt strongly that the decision to implement a treat-to-target strategy should be based on a shared decision making process between rheumatologists and their patients because stricter protocol might not be the preferred approach for a significant number of patients.
Physician-dominated panel: Panelists thought that patients have better outcomes when a targeted approach is used.
Note the recommendations described in this manuscript are presented to illustrate a novel approach of including patients in CPG development process and should not be interpreted as ACR recommendations for the management of RA.
Majority: 70% or more of the voting panel.
For Question #1, both panels voted for using monotherapy over two DMARDs, given the lack of evidence documenting an incremental benefit for two versus one DMARD. However, patients voted that this should be a “strong” recommendation while the physician-dominated panel voted to label this recommendation as “conditional”. The patients felt that this recommendation was applicable to the vast majority of patients, whereas the physician-dominated panel thought there might be more variability in treatment choices (see Evidence Table for Question #1 in the Appendix).
In the second and third questions, the physician-dominated panel voted conditionally against triple (versus mono DMARD) therapy. In contrast, patients voted conditionally for using triple therapy for DMARD-naive RA patients with at least moderate disease activity. The patient panel noted that the increased chance of significant improvements (i.e., remission and ACR 50 in the second and third questions, respectively) associated with triple therapy and the lack of significant added toxicity found in studies between the two strategies justified the use of three medications (see Evidence Tables for Questions #2 and #3 in the Appendix). The patient panel voted to label the recommendation as conditional given the recognition that patients would vary in how they weighed both the practical and psychological burdens associated with taking three medications compared to one. Physicians’ voted conditionally against triple therapy based on their knowledge that the benefits of both strategies converge by two years.
The two panels also differed in the direction of their recommendations for Question #5. In this case, the patient panel voted for using tofacitinib over methotrexate in DMARD-naïve RA patients, whereas the physician-dominated panel voted against using tofacitinib in this population. For this question, patients asked for the original study to be retrieved so that they could examine the risks of adverse events in greater detail. They acknowledged the contrasting risk profiles between the two drugs, but ultimately voted in favor of tofacitinib because of the statistically significant incremental benefits associated with tofacitinib and its lower risk of gastrointestinal side effects (a side effect felt to have a significant impact on quality of life) compared with methotrexate.
The two panels voted in the same way for all remaining recommendations with the exception of the two focused on tapering (Questions #11 and #12), for which the physician-dominated panel voted strongly and the patient panel conditionally against tapering methotrexate or TNF inhibitors in patients with low disease activity. In both cases, the patient panel felt that continuing medications to prevent flares was very important, but that patient preferences would likely vary and should be taken into account (see Evidence Table for Question #11 in the Appendix).
In addition to the comments provided in response to specific PICO questions, the patients also noted that the evidence tables did not include some of the adverse events (such as gastrointestinal side effects, lightheadedness, and general malaise) that they argued should be considered when weighing benefits versus harms. They stated that the data included in the evidence tables were not always sufficiently detailed in order for them to accurately gauge harm. The patient panel also felt that quality of life (reflecting domains besides physical function) should be included as an outcome in the evidence tables.
Discussion
This pilot study demonstrated that patients can develop recommendations based on evidence summary tables generated using the GRADE methodology. The main caveat is that we included only PICO questions with evidence warranting moderate or high confidence. GRADE allows for recommendations to be developed based on lower quality evidence based on the rationale that 1) physicians’ clinical experience is considered evidence (albeit of low quality); 2) physicians and patients want recommendations for difficult decisions, even when robust data are not available; and 3) the transparency of GRADE allows end users to distinguish between the quality of evidence and rationale underlying each recommendation and its associated strength.
In this project, all patients were able to access online materials in order to familiarize themselves with some of the principles of evidence-based medicine and to subsequently meaningfully participate in the face-to-face voting meeting. Moreover, the lived experience with RA is itself a valuable form of expertise, as patients must live with the disease, mobilize resources to implement treatments, and make treatment decisions based on their own preferences, values and biopsychosocial exigencies. Patients strongly endorsed the importance of meaningful patient input in CPG development, and they felt that the amount of time and effort required to prepare and participate were appropriate and worthwhile. Moreover, they thought that patients would be more likely to endorse guidelines that had been developed with meaningful patient input. It must be noted, however, that the participants were all college educated, and that including less well educated patients would require additional time and effort to prepare the panel members to effectively participate.
The panel members were able to effectively use the evidence summary tables to inform their votes. Furthermore, they were comfortable stating when they did not have sufficient information to make a recommendation and to ask for further information, when needed. They did note that the tables did not include many of the adverse events they required to weigh benefits versus harms. In general, the evidence tables reported serious adverse events, but patients also wanted to consider reversible side effects that do not necessarily require treatment or result in hospitalization (e.g. dyspepsia, nausea, headache, lightheadedness, general malaise), but do affect quality of life. They also emphasized the importance of including quality of life as an outcome measure. This assertion highlights the importance of obtaining significant patient input when deciding on the outcomes to be included for each PICO question.
The two panels developed the same recommendations for the majority of the questions considered. However, consistent with an extensive literature documenting differential weighting of specific harms and benefits by patients and their physicians (20), there were some noteworthy differences. In contrast to the physician-dominated panel, the patient panel voted in favor of triple therapy. Patients felt that the incremental benefits associated with triple therapy presented in the evidence table, outweighed the burden of taking two additional medications. They emphasized the importance of achieving “remission,” a viewpoint consistent with previous studies documenting the value that patients attach to “feeling normal” (22-24). The patient panel did not consider the equivalence of long-term outcomes associated with the two strategies in their recommendation, as this piece of information was not included in the evidence table. This example underscores the importance of having both patients and physicians on the panel, as they bring different (but complementary) expertise and values.
The differences in the panels’ recommendations considering the use of tofacitinib reflect differences in how patients and physicians view medications with which we have extensive experience vs. new medications with unknown long-term toxicity. The recent overview of advances in rheumatology published in the Annals of Internal Medicine reflects the physician-dominated panel's opinion (25). In this review, the authors concluded that methotrexate would likely remain the most commonly prescribed initial DMARD despite data from a randomized controlled trial finding tofacitinib to be more efficacious (26), because of potential toxicity related to Janus kinase inhibitors (25). While patients recognized the risks associated with taking new medications, they were willing to consider tofacitinib as a first line agent, whereas the physician-dominated panel was not. This difference was explained by the patient panel's view that some patients might prefer to try a medication that might have less impact on their quality of life, which is consistent with a previous study examining patient preferences for RA treatment (6).
The two panels also differed in the strength (but not direction) of the recommendation regarding tapering. The physician-dominated panel felt that the vast majority of patients with established RA and low disease activity should, and would want to, continue their medications given the known outcomes associated with tapering, whereas the patient panel felt that patients’ preferences for tapering are likely to vary and that this recommendation should, therefore, be conditional.
Based on the insights gained from this pilot experiment, the ACR is deliberating on how to modify current procedures. For example, one option is to have a separate patient panel vote on the subset of PICO questions with moderate to strong evidence prior to the ACR physician panel (as we did in this pilot project), and to then include the patient recommendations for this subset of PICO questions in the evidence tables used by the physician-dominated panel. In this case, two members of the patient panel could sit on the physician-dominated panel to 1) act as liaisons between the two groups, 2) better represent the patient perspective, and 3) increase the likelihood of the ACR guideline being adequately informed by patient values. Similar to the ACR's current efforts to obtain a pool of voting members with relevant clinical and methodological expertise, the College may also consider investing in training a pool of patient representatives, who would then be able to participate in multiple CPG development projects. Determining the “best” panel composition, however, will require further efforts. Other possible approaches include ensuring that a sufficient number of patients participate (e.g. compose 50% or more of the panel) in order to be able to influence the direction or strength of the vote and expanding the pool of experts to include other prescribers (e.g. physician assistants) and/or health professionals (e.g. physical therapists) depending on the PICO questions under consideration.
It must be emphasized that the recommendations described in this manuscript are presented to illustrate a novel approach of including patients in CPG development process and should not be interpreted as ACR recommendations for the management of RA. The patients found the experience to be informative, meaningful, and of significant importance. While the majority of recommendations were concordant across both panels, patients valued outcomes differently in some scenarios. Moreover, the requirement of moderate to high quality evidence limited the number of questions to less than 15% of those initially considered by the physician-dominated panel. Additional experiences are necessary to explain the differences that do exist between the two groups, evaluate their differential impact, and advance the evidence necessary to determine what panel composition is optimal to produce the best guidelines.
Supplementary Material
Significance and Innovation.
There is widespread agreement that patients should be involved in the development of clinical practice guidelines (CPGs).
How best to involve patients in the development of CPG recommendations, however, is not known.
In this pilot study, we demonstrated the feasibility of developing CPG recommendations based on a voting panel composed entirely of patients.
Patient and physician-dominated panels developed the same recommendations for most questions for which there was evidence warranting moderate to high confidence.
ACKNOWLEDGEMENTS
We are extremely indebted to the AR-PoWER Patient Powered Research Network (Arthritis Power) for its support and involvement in this project. Special thanks to Jon Aumann, Bryan Leopold, and Jonathan Hunter for their time, effort and dedication.
Financial support for this study was provided in part by the American College of Rheumatology, the Global Healthy Living Foundation and the National Institute of Arthritis and Musculoskeletal and Skin Diseases, part of the National Institutes of Health, under Award Number AR060231-01 (Fraenkel). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report. Jasvinder A. Singh has received research and travel grants from Takeda and Savient; and consultant fees from Savient, Takeda, Allergan and Regeneron.
Contributor Information
Liana Fraenkel, Yale University School of Medicine, New Haven, CT, 06520.
Amy Miller, American College of Rheumatology, Atlanta, GA, 30319.
Kelly Clayton, CreakyJoints® (part of the Global Healthy Living Foundation), Upper Nyack, NY 10960.
Rachelle Crow-Hercher, CreakyJoints® (part of the Global Healthy Living Foundation), Upper Nyack, NY 10960.
Shantana Hazel, CreakyJoints® (part of the Global Healthy Living Foundation), Upper Nyack, NY 10960.
Britt Johnson, CreakyJoints® (part of the Global Healthy Living Foundation), Upper Nyack, NY 10960.
Leslie Rott, CreakyJoints® (part of the Global Healthy Living Foundation), Upper Nyack, NY 10960.
Whitney White, CreakyJoints® (part of the Global Healthy Living Foundation), Upper Nyack, NY 10960.
Carole Wiedmeyer, CreakyJoints® (part of the Global Healthy Living Foundation), Upper Nyack, NY 10960.
Victor M. Montori, Knowledge and Evaluation Research Unit, Mayo Clinic, Rochester, MN, 55905
Jasvinder A. Singh, University of Alabama, Birmingham, AL, 35233
W. Benjamin Nowell, Global Healthy Living Foundation, Upper Nyack, NY, 10960.
References
- 1. [May 22, 2015]; http://www.iom.edu/Reports/2011/Clinical-Practice-Guidelines-We-Can-Trust/Standards.aspx.
- 2. [May 22, 2015]; http://www.g-i-n.net/working-groups/gin-public/toolkit.
- 3.Agree Collaboration [June 3, 2015];Appraisal of guidelines research and evaluation. http://www.agreetrust.org/resource-centre/agree-ii-grs-instrument.
- 4.Krahn M, Naglie G. The next step in guideline development: Incorporating patient preferences. JAMA. 2008;300:436–8. doi: 10.1001/jama.300.4.436. [DOI] [PubMed] [Google Scholar]
- 5.Fraenkel L, Bogardus S, Wittink DR. Understanding patient preferences for the treatment of lupus nephritis with adaptive conjoint analysis. Med Care. 2001;39:1203–16. doi: 10.1097/00005650-200111000-00007. [DOI] [PubMed] [Google Scholar]
- 6.Fraenkel L, Bogardus ST, Concato J, Felson DT, Wittink DR. Patient preferences for treatment of rheumatoid arthritis. Ann Rheum Dis. 2004;63:1372–8. doi: 10.1136/ard.2003.019422. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Hauber BA, Fairchild AO, Johnson FR. Quantifying benefit-risk preferences for medical interventions: An overview of a growing empirical literature. Appl Health Econ Health Policy. 2013;11:319–29. doi: 10.1007/s40258-013-0028-y. [DOI] [PubMed] [Google Scholar]
- 8.Ryan M, Farrar S. Using conjoint analysis to elicit preferences for health care. BMJ. 2000;320:1530–3. doi: 10.1136/bmj.320.7248.1530. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Fraenkel L, Chodkowski D, Lim J, Garcia-Tsao G. Patients' preferences for treatment of hepatitis C. Med Decis Making. 2010;30:45–57. doi: 10.1177/0272989X09341588. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Nilsen ES, Myrhaug HT, Johansen M, Oliver S, Oxman AD. Methods of consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material. Cochrane Database Syst Rev. 2006:CD004563. doi: 10.1002/14651858.CD004563.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Guyatt G, Oxman AD, Akl EA, et al. GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64:383–94. doi: 10.1016/j.jclinepi.2010.04.026. [DOI] [PubMed] [Google Scholar]
- 12.Brożek JL, Akl EA, Alonso-Coello P, et al. Grading quality of evidence and strength of recommendations in clinical practice guidelines. Allergy. 2009;64:669–77. doi: 10.1111/j.1398-9995.2009.01973.x. [DOI] [PubMed] [Google Scholar]
- 13.Mustafa RA, Santesso N, Brozek J, et al. The GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses. J Clin Epidemiol. 2013;66:736–42. doi: 10.1016/j.jclinepi.2013.02.004. [DOI] [PubMed] [Google Scholar]
- 14.Balshem H, Helfand M, Schunemann HJ, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol. 2011;64:401–6. doi: 10.1016/j.jclinepi.2010.07.015. [DOI] [PubMed] [Google Scholar]
- 15.Andrews JC, Schunemann HJ, Oxman AD, et al. GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation's direction and strength. J Clin Epidemiol. 2013;66:726–35. doi: 10.1016/j.jclinepi.2013.02.003. [DOI] [PubMed] [Google Scholar]
- 16.Jaeschke R, Guyatt GH, Dellinger P, et al. Use of GRADE grid to reach decisions on clinical practice guidelines when consensus is elusive. BMJ. 2008;337:a744. doi: 10.1136/bmj.a744. [DOI] [PubMed] [Google Scholar]
- 17.Guyatt GH, Oxman AD, Vist GE, et al. GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336:924–6. doi: 10.1136/bmj.39489.470347.AD. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Fraenkel L, Bogardus ST, Felson DT, Wittink DR. Treatment options in knee osteoarthritis: The patient's perspective. Arch Intern Med. 2004;164:1299–304. doi: 10.1001/archinte.164.12.1299. [DOI] [PubMed] [Google Scholar]
- 19.Fraenkel L, Cunningham M. High disease activity may not be sufficient to escalate care. Arthritis Care Res (Hoboken) 2014;66:197–203. doi: 10.1002/acr.22098. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Muhlbacher AC, Juhnke C. Patient preferences versus physicians' judgement: Does it make a difference in healthcare decision making? Appl Health Econ Health Policy. 2013;11:163–80. doi: 10.1007/s40258-013-0023-3. [DOI] [PubMed] [Google Scholar]
- 21.Singh JA, Saag KG, Bridges SLJ. American College of Rheumatology Recommendations for the treatment of rheumatoid arthritis. Arthritis Care Res (Hoboken) 2015;xx:xxxx–xx. 20xx. [Google Scholar]
- 22.Sanderson T, Morris M, Calnan M, Richards P, Hewlett S. Patient perspective of measuring treatment efficacy: The rheumatoid arthritis patient priorities for pharmacologic interventions outcomes. Arthritis Care Res (Hoboken) 2010;62:647–56. doi: 10.1002/acr.20151. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Sanderson T, Morris M, Calnan M, Richards P, Hewlett S. What outcomes from pharmacologic treatments are important to people with rheumatoid arthritis? Creating the basis of a patient core set. Arthritis Care Res (Hoboken) 2010;62:640–6. doi: 10.1002/acr.20034. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.van Tuyl LH, Smolen JS, Wells GA, Scholte-Voshaar M, Hoogland W, Boers M. Patient perspective on remission in rheumatoid arthritis. J Rheumatol. 2011;38:1735–8. doi: 10.3899/jrheum.110394. [DOI] [PubMed] [Google Scholar]
- 25.Hellmann DB, Imboden JB. Update in Rheumatology: Evidence published in 2014: Update in Rheumatology. Ann Intern Med 2015. 162:W122–6. doi: 10.7326/M15-0273. [DOI] [PubMed] [Google Scholar]
- 26.Lee EB, Fleischmann R, Hall S, et al. Tofacitinib versus methotrexate in rheumatoid arthritis. N Engl J Med. 2014;370:2377–86. doi: 10.1056/NEJMoa1310476. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.