Engaging Stakeholders to Inform Clinical Practice Guidelines That Address Multiple Chronic Conditions

Abstract

Background

Having more than one chronic condition is common and is associated with greater health care utilization, higher medication burden and complexity of treatment. However, clinical practice guidelines (CPGs) do not routinely address the balance between harms and benefits of treatments for people with multiple chronic conditions (MCCs).

Objective

To partner with the Kaiser Permanente Integrated Cardiovascular Health (ICVH) program to engage multiple stakeholders in a mixed-methods approach in order to: 1) identify two high-priority clinical questions related to MCCs, and 2) understand patients’ and family caregivers’ perceptions of meaningful outcomes to inform benefit/harm assessments for these two high-priority questions. These clinical questions and outcomes will be used to inform CPG recommendations for people with MCCs.

Design and Participants

The ICVH program provided 130 topics rank-ordered by the potential for finding evidence that would change clinical recommendations regarding the topic. We used a modified Delphi method to identify and reword topics into questions relevant to people with MCCs. We used two sets of focus groups (n = 27) to elicit patient and caregiver perspectives on two important research questions and relevant patient-important outcomes on benefit/harm balance for people with MCCs.

Key Results

Co-investigators, patients and caregivers identified “optimal blood pressure goals” and “diabetes medication management” as important clinical topics for CPGs related to people with MCCs. Stakeholders identified a list of relevant outcomes to be addressed in future CPG development including 1) physical function and energy, 2) emotional health and well-being, 3) avoidance of treatment burden, side effects and risks, 4) interaction with providers and health care system, and 5) prevention of adverse long-term health outcomes.

Conclusions

Through the application of a mixed-methods process, we identified the questions regarding optimal blood pressure goals and diabetes medication management, along with related patient-centered outcomes, to inform novel evidence syntheses for those with MCCs. This study provides the lessons learned and a generalizable process for CPG developers to engage patient and caregivers in priority-setting for the translation of evidence into future CPGs. Ultimately, engaging patient and stakeholders around MCCs could improve the relevance of CPGs for the care of people with MCCs.

INTRODUCTION

Forty-eight percent of people older than age 65 have three or more chronic health conditions.^{1 , 2} Having multiple chronic conditions (MCCs) increases costs, risk of polypharmacy and the complexity of treatment regimens.³ Understanding the potential overlap between various chronic health conditions is especially important as each condition may influence the care of the other condition(s) through limitations of life expectancy, interactions between therapies and/or direct contraindications to therapy.^{4 , 5} Since most clinical practice guidelines (CPGs) focus on a single disease, people with MCCs may experience disease-focused—and thus, by definition, not patient-centered—care.^{3 , 6 – 10} Ideally, the balance between treatment benefits and harms or side effects of any intervention among people with MCCs would and should be reflected in CPGs.^{3 , 6 – 10}

Because CPGs have the potential to influence clinical practice, improve quality of care and reduce the inappropriate use of tests and therapies,¹¹ the Institute of Medicine (IOM) has suggested standards to enhance their validity, trustworthiness and rigor.¹² As part of these standards, the IOM recognized the importance of engaging patients and other stakeholders in the development of CPGs, through the formulation of clinical questions and external review of the final documents, and encouraged strategies to foster increased participation throughout the entire process.¹² Guideline development organizations need effective and efficient methods for engaging patients in this process.¹³ The Patient-Centered Outcomes Research Institute (PCORI) has recognized both the necessity and associated challenges with engaging patients as stakeholders and co-investigators in research and CPG development.^{14 , 15} This study describes the first aim of a larger project involving a partnership with the Kaiser Permanente National Guideline Program to set priorities for clinical topics and outcomes relevant to people with MCCs. In 2010, Kaiser Permanente’s National Guideline Program reorganized its guideline work to bring recommendations for diabetes, hypertension, hyperlipidemia, coronary artery disease and heart failure under the scope of one guideline program (thereafter called the Integrated Cardiovascular Health [ICVH] program). Given this reorganization to address concordant conditions, the project team determined this ICVH guideline program to be best suited to serve as a platform for a methods project addressing the broader landscape of MCCs beyond cardiovascular health.¹⁶ We sought to build on best practices for successful and effective stakeholder engagement in research and to apply these lessons learned to engage multiple stakeholders in the process of evidence synthesis to inform CPG development for people with MCCs. Findings from this effort are needed to directly inform the subsequent aims of this project in carrying out a quantitative benefit/harm assessment to inform Kaiser Permanente’s CPG development. Ultimately, the overarching goal of this project is to inform recommendations and person-centered care for people with MCCs.

We used a multi-step mixed-methods process to engage multiple stakeholders in order to accomplish the following study objectives: 1) to identify two high-priority questions related to MCCs, and 2) to understand patients’ perceptions of meaningful outcomes to inform guideline-driven treatment decisions for these two high-priority questions.

METHODS

The institutional review boards of Johns Hopkins University and Kaiser Permanente Colorado approved this study.

Identification of Guideline Topics for Addressing MCCs

The Kaiser Permanente National Guideline Program provides evidence-based recommendations to its organizations that support the delivery of care and optimize the health of its patients. Each year, the ICVH clinical leads review and prioritize the existing recommendations in need of updating based on 1) the importance of maintaining a recommendation on a clinical topic, and 2) the likelihood of needing to update the literature search and recommendation because of new evidence. The ICVH clinical program provided the research team with their list of high-priority topics for 2014. This topic list had been generated by clinicians and guideline developers and served as the starting point for the identification of high-priority topics related to the presence of these and other chronic conditions.

Overview of Steps for Topic Prioritization and Opportunities for Stakeholder Engagement

We used a six-step process for multi-stakeholder prioritization of cardiovascular health-related topics and outcomes important to people living with MCCs. At each stage of our prioritization, we engaged the following four stakeholder groups: 1) patient and caregiver co- investigators, 2) co-investigators who were leaders from the Kaiser Permanente National Guideline Program, 3) co-investigators who also served as clinical leads for the ICVH Guideline Program, and 4) patients and caregivers with MCCs from Kaiser Permanente Colorado. Topic prioritization in steps 1 through 4 involved patient and caregiver co-investigators and Kaiser Permanente National Guideline Program co-investigators. Topic prioritization and outcome generation in steps 5 and 6 involved focus groups with patients and caregivers with MCCs. We describe each of the steps below and in Figure 1.

Figure 1.

Steps for topic and outcome prioritization with stakeholder engagement.

Steps 1 and 2: Identification of Topics of Clinical Importance for Adults with MCCs

In the first step, the research team eliminated topics not relevant for MCCs, regarding screening, pediatric patients/conditions and pregnant women. We then excluded the bottom tertile of topics, because they were deemed lowest priority and not likely to be updated (Fig. 1). Using two Delphi rounds,¹⁷ our entire team, including CPG developers and patient and caregiver co-investigators, reviewed the guideline topics and responded to the queries: “Is the question relevant to people with multiple chronic conditions, or can it be modified to be relevant?” (Yes, No, or Yes if modified). We ranked the “overall relevance to people with MCCs after modification” (scale: 1 = least relevant, 9 = most relevant). Questions with scores below the calculated mean were eliminated.

Steps 3 and 4: Creation of Broader Clinical Categories of Questions, Patient-Friendly Language

For steps 3 and 4, the research team organized the list of questions into broader categories of questions (e.g. hypertension, diabetes mellitus, hypercholesterolemia, coronary artery disease) and rephrased the remaining 12 questions into patient-friendly language with feedback from our patient and caregiver co-investigators. The principal investigator (PI) (CB) presented these questions to a meeting of the ICVH clinical leads to get their feedback about the relevance of these questions to their needs and to patients with MCCs.

Steps 5 and 6: Focus Groups with Patients and Caregivers

In step 5 we conducted two focus groups with patients and caregivers to elicit patients’ perspectives and to select two high-priority questions relevant to living with or caring for loved ones with MCCs. We had determined a priori the need to choose two questions to ensure that the next steps of this project were feasible and practical. Following step 5, in step 6 we conducted two more focus group with patients and caregivers, focused on identifying outcomes most important to patients with MCCs related to the two questions identified in step 5.

Conduct of Focus Groups in Steps 5 and 6 in the Mixed-Methods Process (Fig. 1)

Our team developed a semi-structured moderator guide for each set of focus groups (see Supplement), pilot-tested the guides with our patient and family caregiver co-investigators, and incorporated their feedback. In step 5, focus group participants received large printed cards with the health condition question and definitions. These cards were used to facilitate further discussion and the selection of high-priority topics. The moderator guide for step 6 utilized more open-ended questions and prompts related broadly to patient-important outcomes and treatment goals. Moderators from our clinical team used the nominal group technique in focus groups to enable in-depth discussions in both steps 5 and 6.¹⁷

Participants and Setting for Focus Groups in Steps 5 and 6

We identified focus group participants as eligible if they were members of Kaiser Permanente Colorado and had 1) at least one cardiovascular condition (coronary artery disease, hypertension, heart failure, hyperlipidemia, diabetes mellitus, stroke, transient ischemic attack and peripheral arterial disease/peripheral vascular disease), 2) one or more comorbidities from those listed in the Quan adaptation of the Elixhauser comorbidity index,¹⁸ and 3) a score of 3 or more based on the Quan adaptation of the Elixhauser comorbidity index used as a marker of disease severity¹⁸ to identify focus group participants likely to have MCCs. We also enriched our sample by including patients with dementia and invited family members or caregivers of participants. Focus groups lasted approximately 90 min and were held in convenient settings at Kaiser Permanente facilities in Denver, CO, in July and September 2015.

Qualitative Analysis of Focus Groups in Steps 5 and 6

The four focus groups were audiotaped and transcribed verbatim, and references to personal health information were removed. To develop the coding templates, three investigators (RW, HT, WB) independently read the transcript from the first group of each set, applying an editing style analysis to identify meaningful segments and create codes.¹⁹ Two investigators then independently coded the remaining transcripts, applying the coding template, which was iteratively modified as the analysis proceeded, and grouped codes into general themes. The team, including the patient-caregiver co-investigators, collectively selected the themes and representative quotes we present in this paper. To facilitate data management and coding, we used ATLAS.ti 6.2 software (ATLAS.ti Scientific Development GmbH, Berlin, Germany).

RESULTS

Flow Through the 6-Step Multiple Stakeholder Engagement Process (Fig. 1)

Figure 1 shows the flow through the steps in the stakeholder engagement and prioritization process. In step 1, the guideline program provided the research team with a list of 130 topics in rank order. Examples of high-priority topics were “anti-platelet therapy following stent placement…,” “medication management for type 2 diabetes” and “treatment for heart failure with preserved ejection fraction.” We eliminated the four topics addressing screening or primary prevention, pediatric populations and pregnancy, and the 42 topics in the bottom tertile, yielding 84 topics. Step 2 involved two Delphi rounds where 72 of the 84 questions were modified by investigators to make them more relevant to people with MCCs, and 52 questions were eliminated as part of the Delphi process. In step 3, the remaining 32 topics were organized into broader categories by disease condition, and were narrowed and reworded into 12 patient-friendly questions in step 4 (Fig. 1).

Steps 5 and 6 informed the question and outcomes selection by engaging patients with MCCs and their caregivers in four focus groups. Step 5 involved a total of 11 participants selecting from questions important for patients and caregivers with MCCs. Step 6 involved a total of 16 participants discussing high-priority outcomes for people with MCCs.

Results of Step 5: Focus Groups Selected Two High-Priority Questions Relevant to People with MCCs (Fig. 1, Table 1)

Table 1.

Topics and Prioritization Ranking During First Steps of Focus Groups

Topic	Illustrative quotations
Blood pressure management
Choosing of blood pressure treatments	I think it’s nuts…. I did have a medication once that caused me to cough…That’s the only time I’ve ever gone to somebody and said, “Let’s change this.”
	Certain medications you add differently to the individual.
Targeting blood pressure levels	I was on the smallest dose of blood pressure medicine, but I started working out several times per week, and my blood pressure was dropping lower. [I]t wasn’t within the standard guidelines goals, but that small dose was impacting me. So my doctor finally took me off blood pressure medication.
Diabetes management
Choice of diabetes treatments	If you are borderline [diabetic, you might not take medication, but] some people take medicine and some people take shots, so there must be a reason why you can take a pill and somebody else have to take insulin.
	I am a borderline diabetic. I don’t take medication. I exercise and, you know, they told me to do these things to keep it down. But I shouldn’t be on the same as he would be. Or his brother. And I shouldn’t be taking their type medication.
Goals for blood sugar level	And you wonder about the A1c. Everybody is saying it has to be below 7. But what about 7.2 being fine? But these pharmaceutical companies want the money.
Cholesterol management
Figuring out who needs cholesterol medicine	My cholesterol was always high …my doctor always says it’s ok ‘cause it’s my good cholesterol that is pushing the ratio up…I didn’t need any medication…After I saw her…I got a letter …saying your cholesterol is high…and you better learn how to control it and all this stuff.
	If you have high blood pressure and high cholesterol—if you are treating one of them, is the benefit from, like, one of them going to lower …going to give you added benefit from the other one without actually taking another medication? Do I have to take both in tandem? Or is there one that is going to treat both?
Treatments for cholesterol/goals of treatment	You know what I’d like to see? I’d like to see this percentage of the people who take this medication and suffer from muscle aches or fatigue or blah, blah, blah, instead of the long list on that little piece of paper you get…
Coronary artery disease
Blood thinners	They treat you like guinea pigs to see what works. That’s what they do…this doesn’t work? Try this one. They just keep switching.
	With the blood thinner, I think it would be very important with regards to if a person was going to have surgery, to be very clear and find out if the person was taking something.
Other treatments	…for me, the beta blocker, it really was like somebody sitting on my chest 24 h a day. But it was helping my heart at the same time. But I just could not function. So we changed. He said, ok, we’ll work in another way…but I said no more of them, because …once they found out I had COPD and that’s why I was going around sounding like a horse, you know?

Table 1 shows representative quotations for the 12 questions on the health conditions discussed in the first two focus groups (step 5): hypertension management, diabetes treatment, cholesterol treatment and management of coronary artery disease. Focus group participants shared their personal challenges in making decisions about medication choices and whether to take medications, given their other diseases and treatments. The two most commonly discussed questions as areas of great uncertainty and high relevance for people with MCCs were: “Of people with diabetes, are there groups of people for whom the balance between benefits and harms would affect the choice of treatment?” and “Should target blood pressure be different for some groups of people?”. One participant summarized his rationale for choosing diabetes as an important topic for people with MCCs: “I picked [the question on diabetes medications] as my first choice because diabetes is such a horrible disease, and I don’t think that the same treatment works for everybody…I think that is very important.” The two questions on blood pressure targets and diabetes medications were then promoted to step 6, for two more focus groups to identify high-priority outcomes.

Results of Step 6: Focus Groups Identified Outcomes Relevant to People with MCCs (Fig. 1, Table 2)

Table 2.

Identification of High-Priority Outcomes on Diabetes and Hypertension Management in People with MCCs, Using Focus Groups

Outcomes	Examples
Physical function and energy	Physical function (activity, mobility)
	Being able to do the things you enjoy or want to do
	Comfort and overall feeling good
	Having energy
Emotional health and well-being	Mood
	Faith
	Stress
	Anger and guilt
	Optimism
	Thinking clearly
Interaction with providers and health care system	Communication between patient and providers
	Care coordination between providers
	Advocating for self and own needs
	Navigation of insurance and coverage benefits
Avoidance of treatment burden, risks and side effects	Medication burden
	Drug interactions
	Desire for relief without risks
	Side effects of medications
	Avoiding taking more medications
	Being able to try alternative and holistic medicines
	Using diet as a treatment
	Utility of over-the-counter or alternative meds
	Costs (co-pays, premiums) of medications and treatment
	Role of genetics in choosing treatments
Prevention of adverse long-term health outcomes	Prevention of pain
	Healthy aging
	Kidney disease
	Stroke
	Death
	Blood pressure level
	Medication dependence/addiction
	Surgery Avoidance
	Eye complications: glaucoma/blindness

In step 6, we conducted two additional focus groups with patients with MCCs and their caregivers to discuss outcomes related to the questions on optimal blood pressure goals and diabetes medication management. Participants highlighted five broad outcome categories, as listed in Table 2 and described below, with examples of quotations: 1) physical function and energy, 2) emotional health and well-being, 3) avoidance of treatment burden, side effects and risks, 4) interaction with providers and the health care system, and 5) prevention of adverse long-term health outcomes.

Physical Function, Energy, Emotional Well-Being

Patients and caregivers valued outcomes of physical function, energy, quality of life and emotional well-being. They described chronic pain as a major barrier to day-to-day physical functioning and wellness, as exemplified by one participant who stated, “All these chronic pains that you have and then life knocks on the door, you have to be able to function.” Participants described their own experiences with aging, setting new health goals and addressing emotional health and well-being. One participant described his struggle to continue to function and maintain high quality of life:

My philosophy—and my wife and I fight about this all the time—I work out 6 days a week. … I take my dog out for walks even though I hurt so bad I can’t even begin to tell you how much pain I’m in when I’m walking. But I do it because I’m not going to let this thing stop me from living my life. I would rather only live for another 2 years and enjoy it, than to start watching everything I eat, and worry about this and worry about that, to live another 20 years.

Avoidance of Treatment Burden, Side Effects and Risks

Patients and caregivers highlighted outcomes of treatment burden, side effects (e.g. feeling “dizzy, falling out on the floor”) and the potential for drug interactions, especially when on multiple medications (“how [new medicines] affect everything else…[i]ncluding their health conditions and medications”). One participant with MCCs described her experience, weighing the potential benefits of taking multiple medications against her concerns for potential side effects and interactions that could lead to more health problems:

[T]o me, taking all the medications that are recommended for all of these chronic illnesses we all have—the side effects outweigh the possibility of recovery. Because the side effects are 16 pages long for this one little thing you are on…so you open the door to a whole bunch of other things you didn’t even have.

Participants commonly articulated the important outcome of assessing the burden of taking many different medications for different conditions: “I take nine or ten medications—different ones—twice a day. And I have anxiety, dementia and diabetes…just to name a few, and I take meds for all of that.”

Interaction with Providers and Health Care System

Participants valued communication and care coordination with health care providers as an outcome of their care. They expressed the need for providers to understand their unique situations and to “match” treatments to individual patient needs, prior history, having MCCs, race/ethnicity and personal/home situation. For example, two participants described their related questions and experiences:

Do you [have different recommendations] for different ethnic groups, like Native Americans, and different races? ‘Cause I’m American Indian, and diabetes is really rampant. [My father] never lost anything—toes or anything—but it shut everything down all at once…lungs, kidneys, heart…Do you [have different] targets for blood sugars for different races instead of just [one goal] for all?

My [cholesterol] prescription didn’t match me. So I don’t take it. I don’t take any. And it is high. It’s too high.

Participants also discussed their own strategies for how they go about making treatment decisions while weighing “risks and benefits” and “incorporating their individual needs” into their treatment choices. Some participants relied on their providers to know what was best for them:

I believe in my doctor and I do what he says, because he is the one who is taking care of me and he seems to know… And I can tell him, "This isn’t working. Do you have another remedy?”

Other participants were concerned that providers did not have the tools to make treatment decisions and desired greater autonomy. One participant described his process for weighing risks and benefits:

You weigh [the options]. You think about them. And you figure, is it good for me to do it at this particular time? or later? And the result is I am always going to do what I think is best for me. Regardless of what the doctor says. Because he doesn’t have my body. He assumes I have this type of pain, but does he know what I am feeling? … So you choose according to what you figure is best for you at any given time.

Participants described conversations they have had with their providers to make difficult treatment decisions, individualized to their own needs and other health conditions:

But I just could not function. So we changed. [My doctor] said, ok, we’ll work it another way…but I said no more of them, because the lungs was the hardest part, once they found out I had COPD and that was why I was going around sounding like a horse all day long. And to treat [my breathing] was more important than the little [benefit] I got [from the beta blocker for treating my] large heart with fatty tissue growing around it….[T]he beta blocker just made me feel worse.

Prevention of Adverse Long-Term Health Outcomes

Focus group participants considered the balance between potential short-term benefits, such as pain control, and the longer-term benefits, such as the prevention of kidney disease and heart attacks (“I don’t want the effects—the horrible effects—that diabetes brings. ‘Cause it brings them if you don’t watch it.”) versus the risks of side effects and treatment burden. One participant described her experience with arthritis, use of anti-inflammatory medications and her need for surgery, which had not completely resolved her arthritic pain:

[T]hey gave me an anti-inflammatory. But that anti-inflammatory… created some other problems. Subsequently, knee replacement was the answer to quiet the arthritis, but the arthritis…is still in the body and is not going to go away.

DISCUSSION

Through a six-step process using multiple stakeholder engagement, including patients and guideline developers, we identified two important treatment questions and the related patient-important outcomes relevant for people with MCCs, in order to inform the process of developing recommendations for CPGs. Patients with MCCs summarized overarching themes related to their care that should be addressed in CPGs. These themes included their desire to understand and discuss with providers the potential benefits and harms of treatments, wanting individualized treatments that addressed their unique health and life situation and valued function, energy and quality of life. Although our process was focused on choosing topics for evidence syntheses that can ultimately inform CPGs relevant for people with MCCs, we believe our process has the potential to inform other guideline development organizations on a strategy to include and engage patient stakeholders early in the process of guideline development.

Our findings show that people with MCCs desire patient-centered care that prolongs their lives, but also—most importantly—addresses the benefits and risks of treating all of their conditions as a part of the whole person to maximize quality of life. To summarize, one of our patient co-investigators stated that “I need doctors who are going to be able to treat me as a whole person, not look at me as different parts of this same body.” Despite the specific selection of MCC patients/caregivers with diabetes and hypertension, the list of outcomes that they identified were broadly applicable to multiple chronic health conditions. These outcomes are consistent with prior studies assessing the effect of treatment burden on people with MCCs.^{20 – 22} For example, Ridgeway and colleagues interviewed and conducted focus groups with people with MCCs, and these study participants also described the importance of emotional health and maintaining a positive attitude, and effective communication with providers about the risks and benefits of treatment.²⁰ Notably, the majority of outcomes that were identified by the patients and caregiver focus groups are not currently addressed in CPGs. A strength of our process was that CPG developers had the opportunity to hear and reflect on the perspectives of patients and caregivers at each stage of the process, thus aiding in translating the patient voice to future CPGs.

Next Steps to Inform CPGs Related to MCCs

As our team begins the next step of reviewing evidence related to diabetes and hypertension in evidence syntheses, we will incorporate the outcomes identified in benefit/harm analyses to be subsequently addressed in CPGs. In addition, there could be a downstream effect on future guidelines not related to these topics. As CPG developers listen to and reflect the patient perspective in their guidelines, they will identify gaps in the literature around the effectiveness of treatments for many conditions in people with MCCs. CPG developers will be able to include and address patient-important outcomes, which could lead to additional proposals, funding and evidence generation to close these evidence gaps.

We identified several limitations of our study. First, qualitative research is not generalizable, but rather aims to generate hypotheses. The focus group participants were all patients of Kaiser Permanente Colorado, and their perspectives may differ from those of patients with MCCs in other health care delivery settings. In addition, individuals who participate in focus groups may be different from and potentially more activated and engaged in care than those who choose not to participate or do not have access to do so. Second, our work was limited to the Kaiser Permanente National Guideline Program, and other guideline-producing professional or health care organizations may have different processes. Kaiser Permanente’s National Guideline Program uses a process consistent with the standards recommended by the IOM in its high-quality systematic reviews and clinical practice guidelines. However, given the IOM reports, it is important for all guideline developers to consider how to integrate patients and other stakeholders into their recommendation process. Third, we started with a list of high-priority topics that had been previously generated by the ICVH guideline program without patient stakeholder input, and we limited the study a priori to select two high-priority topics to inform the upcoming quantitative analysis. Although the selected ICVH topics and outcomes resonated with our focus group participants and patient stakeholder co-investigators, our final list is not meant to be all-inclusive, as other health conditions, topics and outcomes may also be important to address. Conducting this project in partnership with a high-quality CPG program with rigorous methods was viewed as essential for these results to be both practical and applicable for other CPG developers, but necessitated working within their existing CPG process.

Our results have implications for systematic reviewers, CPG developers, researchers and policymakers working to improve patient-centered care for the large population of people living with MCCs. Our process provides a model for other CPG-developing organizations to meet the standards established by the 2010 IOM report, Clinical Practice Guidelines We Can Trust,¹² and the Guidelines International Network.²³ Several studies have proposed methods for eliciting patient preferences for care and for engaging stakeholders in the research process.^{24 – 26} However, we are not aware of studies that have connected patient stakeholders with CPG stakeholders in the same process to hear each other’s perspectives to directly inform the next step of evidence syntheses. As part of this process, we encountered several challenges with the stakeholder engagement process, including adding patient and caregiver stakeholders to the institutional review board protocol, involving them in large conference calls (vs. more personal meetings) and communicating the technical research methods. These challenges were addressed by facilitating the necessary human subjects protection training by the study team for patient and caregiver stakeholders, having a combination of in-person meetings with study co-investigators and virtual meetings with the entire team, and focusing on describing research methods at each patient and caregiver co-investigator meeting. The value of patient and caregiver participation in the process was solicited from the larger study team and communicated back to the patient and caregiver team members.

In conclusion, we have described a feasible, potentially replicable mixed-methods process involving multiple stakeholders for identifying two high-priority questions and relevant clinical outcomes for people with MCCs. These questions related to diabetes and hypertension will be the focus of future CPGs that address people with MCCs and incorporate the benefit/harm balance into the outcomes identified by patients and caregivers.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they do not have a conflict of interest.