Developing an instrument to assess patient preferences for benefits and risks of treating acute myeloid leukemia to promote patient-focused drug development

Abstract

Objective

Acute myeloid leukemia (AML) is a progressive blood cancer with few effective treatment options. As part of a patient-focused drug development (PFDD) initiative led by the Leukemia and Lymphoma Society (LLS), this study sought to use a community-centered approach to develop and pilot an instrument to measure patient preferences for the benefits and risks of treating AML.

Methods

Instrument development was informed by a literature review, engagement with expert stakeholders (n = 12), engagement with community stakeholders, and pre-testing. A discrete-choice experiment (DCE), in which participants made choices between 16 pairs of hypothetical treatments, was developed with five attributes: event-free survival, complete remission, time in hospital, short-term side-effects, and long-term side-effects. A pilot test was conducted and analyzed using conditional logistic regression. Results are presented using relative attribute importance (RAI) scores.

Results

Patients with AML and caregivers were engaged in developing (n = 15), pre-testing (n = 13), and pilot testing (n = 26) the instrument. The pilot included patients with AML (n = 18) and caregivers of living or deceased patients with AML (n = 8). Participants had a mean age of 50 years (range =24–81), and were mostly college educated (n = 22), privately insured (n = 21), and employed (n = 13). Based on the DCE, complete remission was identified as the most important attribute (RAI =10), followed by event-free survival (3.7), time in hospital (2.8), long-term side-effects (2.3), and short-term side-effects (2.1).

Conclusion

The mixed-methods approach to PFDD was welcomed by all stakeholders and there was strong endorsement to implement this DCE as part of a national survey.

Introduction

Acute myeloid leukemia (AML) is a cancer of blood-forming myeloid cells that results in bone marrow failure¹. AML has an incidence of ~3–4 per 100,000 people, and tends to effect older adults, with a median age of diagnosis of 67 years². A rapidly progressing disease, AML can be deadly within weeks or months³. Those who are younger than 60 years of age achieve 5-year survival 35–40% of the time, while individuals older than 60 achieve 5-year survival 5–15% of the time^4,5.

Although many agents are currently under development, the last drug approval specifically for the treatment of AML was in 1990, and newer agents have failed to meet the US Food and Drug Administration’s (FDA) threshold for approval. Combination chemotherapy remains the standard approach to treating AML^6–11. This strategy has not substantially changed, despite it carrying a substantial risk of adverse drug events that can result in serious and fatal complications¹². While aggressive treatment increases the likelihood of remission, a substantial portion of patients are considered unfit for intensive chemotherapy or clinical trials because of age or performance status^13–17. Specifically, the likelihood of an adverse event increases with age, and older patients often have to forgo treatment and clinical trials altogether¹⁸. Untreated, older individuals with AML only have a median survival of 2 months¹⁹. Understanding the risk tolerance of diverse patients could inform treatment decision-making and might be used to inform drug development²⁰.

The value of incorporating patients’ knowledge and experiences about living with a health condition is increasingly recognized by healthcare decision agencies^21,22. After a congressional mandate, the US Food and Drug Administration (FDA) started to engage patients through the Patient-Focused Drug Development (PFDD) initiative^17,23. While most PFDD meetings have focused on qualitative testimony and engagement, quantitative methods can also be used to describe the patient experience and their preferences. Stated-preference methods such as discrete-choice experiments (DCE) can be used to estimate the relative importance of specific treatment attributes to patients and can be used to calculate the benefit–risk tradeoffs patients are willing to make²⁴. There is a relative paucity of information about the preferences of patients with blood cancers^25,26, and research has tended to focus on quality-of-life²⁷. We sought to develop and pilot a DCE to assess patient and caregiver preferences for AML treatments, with the goal of informing a PFDD meeting at FDA^17,22,28.

Methods

This study adopted a five-step framework to instrument development: (1) evidence synthesis, (2) diverse stakeholder engagement, (3) patient and caregiver engagement, (4) pre-test interviews, and (5) pilot testing and presentation^29–33. All steps of the research process were guided by existing good research practices^24,34,35. This study was reviewed and approved as a non-human subjects research activity by the institutional review board at the Johns Hopkins Bloomberg School of Public Health (#00006858). This enabled us to have repeated interactions with all committee members and to engage them as key-informants rather than research participants.

Engagement process

Figure 1 illustrates the study governance structure and engagement process surrounding three committees^29–32. An executive team consisting of staff from the Leukemia and Lymphoma Society (LLS) and stated-preference researchers from Johns Hopkins University oversaw and conducted the study. The LLS then created an expert stakeholder committee and a community stakeholder committee, which facilitated continuous engagement with diverse community members throughout the duration of the study.

Figure 1.

Governance structure and stakeholder engagement process.

The expert stakeholder committee primarily included clinicians and industry representatives. The expert stakeholder committee provided input on the research questions and the scope of the study, consulted on various aspects of the study as it progressed, and provided guidance on the clinical accuracy and relevance of the survey. The community stakeholder committee was comprised of AML patients and caregivers. Members of the community stakeholder committee contributed to key design features of the DCE, and piloted the survey. These patients and caregivers were able to ensure that each step of the study process fully considered the patient experience. To ensure collaboration across stakeholder groups, two patients from the community stakeholder committee also participated on the expert stakeholder committee. Most study decisions were made by consensus, but, when consensus could not be reached, the final study decisions were made by the executive team. The engagement process was intensive and spanned 6 months, during which many interactions between the executive team, the community stakeholder committee, and the expert stakeholder committee occurred, as represented in Figure 2.

Figure 2.

Timeline of the study.

Evidence synthesis and attribute selection

We conducted an extensive review of the scientific literature and pharmaceutical product information to identify key attributes that are relevant to currently available chemotherapy treatments for AML. We searched Micromedex treatments approved for AML and details on benefits and possible adverse drug events.

Expert stakeholder engagement

The expert stakeholder committee determined the scope of the study and advised on the clinical accuracy of described attributes and levels in the survey instrument. They also discussed whether the included attributes and levels were relevant to clinical practice. Members were invited to engage in individual interviews (n = 7), and three 1-hour group calls from November 2015 to March 2016. All interviews and discussions were recorded and transcribed. Committee members relied upon their knowledge and experience to help identify the most important attributes.

Patient and caregiver engagement

LLS team members recruited patient and caregiver participants for the community stakeholder committee. A list of potential participants was generated from an LLS database of volunteers who had agreed to collaborate with the LLS on a variety of activities and topics. Potential committee members were contacted by LLS staff and provided information about the project and its goals. Informed consent was obtained from all individual participants included in the study.

Semi-structured qualitative interviews were conducted with the committee members. These interviews focused on the personal experience of patients and caregivers with AML. Then group phone calls were conducted to discuss the selected attributes and to develop the instrument. All interviews and group calls were recorded and transcribed. Results from individual interviews and group calls were analyzed by members of the executive committee to identify common themes and extract the treatment attributes that were most relevant to these stakeholders.

Pre-test interviewing

Pre-test interviews were conducted in order to identify ways to improve the survey, identify any gaps in the survey, and to assess the burden of the survey on potential respondents. Participants were encouraged but not required to complete the survey instrument before the pre-test interview. Participants received the survey instrument via email in advance of the phone interview. During the phone call, participants were asked to verbalize their reactions, thoughts, concerns, and questions regarding the content, wording, and formatting of different survey designs. All interviews were recorded and transcribed. Based on feedback from the pre-test participants, the survey was updated on an iterative basis.

Pilot testing

After the pre-test interviews, the pilot survey was developed. This survey included five attributes at three levels each: Event-free survival (6, 9, 12 months), Complete remission (20%, 40%, 70% chance), Time in hospital (none, 1 month, 3 months), Short-term side-effects (mild, moderate, severe), and Long-term side-effects (none, mild, moderate) (Figure 3). The categorical attribute level descriptions were based on the descriptions of grade 1, 2, and 3 of the Common Terminology Criteria for Adverse Events (CTCAE). The pilot survey instrument was pilot tested to confirm participants could make trade-offs between the different attributes and to make sure the instrument could successfully be administered online.

Figure 3.

Preference weights for acute myeloid leukemia (AML) treatment attributes (A) and relative attribute importance (B). (A) Preference weights for AML treatment attributes (n = 26). Preference weights are rescaled to set the lowest level of each attribute to zero. The y-axis represents how much participants value the attribute: a positive preference weight indicates that participants preferred the attribute, a negative preference weight indicates that participants avoided the attribute. (B) Relative attribute importance is calculated by taking the difference in the preference estimate of the most preferred level of an attribute and the least preferred level of an attribute. The RAIs were rescaled so that the largest RAI (for complete remission) was equal to 10. The vertical bars surrounding each RAI denote the 95% confidence interval about the point estimate.

In the pilot, survey participants were presented with 16 choice tasks containing different attribute-level combinations of two hypothetical treatments for AML and asked to select the one they preferred. Participants were advised that the AML treatments presented were not currently available, but could be available in the future.

A D-efficient design with zero priors was generated using Ngene (ChoiceMetrics, Melbourne, Australia) with 16 choice tasks³⁵. Pilot survey results were assessed using a conditional logit model with effects coding in Stata14 (StataCorp, College Station, TX). The dependent variables indicated whether the participants chose the treatment²⁴. Coefficients were shifted such that the neutral level was centered at zero. We then calculated the relative attribute importance (RAI) by taking the difference in the preference estimate of the most preferred level of an attribute and the least preferred level of an attribute. The RAIs were rescaled so that the RAI for the most important attribute was equal to 10. Differences between attribute levels and RAIs were assessed using independent t-xtests.

Results

Evidence synthesis and attribute selection

From identified treatments, we extracted a list of benefits and risks of different AML chemotherapy treatments. The CTCAE version 3 was used to describe different levels of severity of adverse events³⁶. The executive team selected 10 attributes to be presented to the two stakeholder committees. These attributes included event-free survival (EFS); complete remission (CR); time in hospital; need for blood transfusions; mental health; fever and infection; nausea, vomiting, and decreased appetite; organ toxicity; chemobrain; and fatigue.

CR and EFS were chosen as measures of treatment benefit. CR has been the gold standard for assessing the early success of new agents in AML over the past 30 years³⁷. EFS is also considered a surrogate end-point for survival, an event refers to relapse from CR or death⁸. Overall survival was not included because it is difficult to measure in clinical trials, and participants might not be willing to trade overall survival in a stated-preference setting.

The remaining eight attributes describe possible harms of chemotherapy treatments. Time in hospital for inpatient treatment was selected because over 60% of patients with AML spent time in hospital as inpatients, receiving either potentially curative or supportive treatment³⁸. The need for blood transfusion was selected because essentially all patients with AML receive blood transfusions as a supportive therapy^39–43. Fever, infection, nausea, vomiting, decreased appetite, and fatigue were identified as common adverse drug reactions with AML treatments^44–51, and organ toxicity was included as a severe adverse drug reaction^44,48,52,53. Mental health and chemobrain were included to capture overall patient well-being^53–55.

Expert stakeholder engagement

The committee endorsed using both EFS and CR as measures of benefit and weighed in on how the levels of the harm attributes should be described. For example, they suggested that a “severe” level should be included for organ toxicity. This committee also discussed and endorsed a suggestion from the community stakeholder committee, to consolidate the different long- and short-term side-effects and create compound attributes simply described as short- and long-term side-effects. Feedback from the expert stakeholder committee ensured that the attribute descriptions and levels were appropriate and would resonate with the medical community.

Community stakeholder engagement

Fifteen 1-h individual interviews and four 1-h group discussions were conducted over the telephone with patients and caregivers between January and March 2016. These engagement activities were crucial to our understanding of symptoms, treatments, side-effects, clinical trials, and the overall disease experience of AML^56–58. For example, we learned that 10 participants focused on survival, rather than quality-of-life during their treatment for AML, whereas five individuals focused on “going back to normal”. Other common themes included the need for more treatment options, higher participation in clinical trials, availability of individualized treatment, and expedited drug approval.

Pre-test interviews

Thirteen 1-h individual pre-test calls were conducted between February and March 2016. Based on the pre-test interviews, the survey was iteratively updated, resulting in four versions of the survey. A select group of pre-test participants’ reactions to the attributes and attribute descriptions in the different survey versions are presented in Table 1. As an example, a significant group of patients suggested that blood transfusion should be removed as an attribute, because it was relatively unimportant to their overall disease experience, and others expressed that blood transfusions were a benefit because they made them feel better.

Table 1.

Patient and caregiver responses to attributes and levels of different survey iterations throughout the survey development process.

	Sample quotation
	Survey version 1 (pre-tested)	Survey version 2 (pre-tested)	Survey version 3 (pre-tested)	Survey version 4 (pre-tested and pilot tested)
Event-free survival	“I always think of event-free, like, you don’t get any of the symptoms, right, of the disease, like, you don’t get unusual infections or any-thing …” (Participant 7)	“I like the fact that you use the word event-free because I don’t like when people ask me why aren’t you cured?” (Participant 17)	“Why would anybody want 12 months? What is the point of this is what I’m asking? If I’m in remission, and then I relapse six months later” (Participant 5)	“Those all resonate. I think it is extremely important” (Participant 10)
Complete remission	“Make that a little bit clearer like; I am cancer-free. That is it” (Participant 12)	““If they are cancer-free, wouldn’t it be an anomaly if they got cancer again?” (Participant 17)	“But out of all of these, for me, complete remission is much more important than any of the other factors” (Participant 5)	“I had trouble with this one, only because remission as a goal didn’t seem that important to me if you didn’t know it was gonna last” (Participant 31)
Time in hospital	“I like the length of hospital stay because I really picture that, I spent a month after diagnosis in the hospital, and it felt like forever” (Participant 1)	“I never had outpatient chemo, ever. I was always inpatient. Moreover, so I feel like it made sense” (Participant 16)	“It makes sense to me” (Participant 14)	“Very, very clear” (Participant 12)
Blood transfusion	“I do not know how many transfusions other people need, but actually I think for my husband the experience was not onerous” (Participant 7)	“The two factors I would get rid of would be the blood trans-fusion, and that was because blood transfusions, they vary from person and the person’s circumstance” (Participant 16)	Attribute removed	Attribute removed
Nausea and vomiting	“I took out the words severe amounts of, I worded it you may lose fluids that need medical intervention” (Participant 4)	“The level, none, doesn’t make any sense. You are going to experience it” (Participant 11)	Attribute removed	Attribute removed
Fever and infection	“Maybe [remove] infection and fever or nausea, vomiting can be combined into one like a short-term during treatment side-effect” (Participant 1)	“I think you may want to dropthe ‘no fever’ possibly” (Participant 17)	Attribute removed	Attribute removed
Short-term side-effects	Not applicable	Not applicable	“That is clear” (Participant 14)	“Well done. And showing mild that says, you might have some, but you’ll manage it, all the way down to severe where they could be life-threatening and potentially fatal” (Participant 10)
Long-term side-effects	“You do not have specific levels, you just have a definition of what you are looking for, but there are no levels here particularly” (Participant 7)	“I think it is inconsistent with the other way you’ve done it, and because you’ve used a sort of like qualitative— this does not make sense because it is very specific” (Participant 17)	“If you asked me what one of the most important side-effects that would be chemobrain” (Participant 5)	“What you’ll be dealing with long-term is how mild do you consider mild? And what are the potential areas of impact? The thing with moderate, what are the potential areas of impact?” (Participant 10)

These qualitative pre-test interviews ensured that the attribute descriptions accurately conveyed the perspectives of the patients and caregivers⁵⁷. Overall, participants understood the objectives of the survey and believed our study would effectively inform the FDA about the needs of the AML community. The majority of pre-test participants recommended we use a question stem that utilized the first-person perspective.

Pilot test

A total of 26 participants completed the pilot survey online. Pilot participants were not representative of the general population with AML, but were patients and caregivers from the community stakeholder committee, as well as a convenience sample of 11 newly-recruited individuals. Table 2 summarizes key clinical and demographic information. On average, participants were diagnosed 6 years ago. Most participants were Caucasian (n = 19), college educated (n = 22), married (n = 15), and privately insured (n = 21), and half of them were employed (n = 13). After being diagnosed with AML, most patients began their first treatment within a week (n = 23). Although many participants were not given any choice regarding their treatment options (n = 14), most were involved in the decision-making process regarding whether to get treatment and (if applicable) which treatments (n = 21). Nearly all participants underwent some kind of chemotherapy (n = 25), and slightly less than half received a stem cell transplant (n = 16).

Table 2.

Characteristics of pilot participants.

Characteristics	Patients (n) (n = 18)	Caregivers (n) (n = 8)	Total (n = 26)
Demographic characteristics
Age (years)	48	55	50
Gender (female)	9	8	17
Race/ethnicity (White/Caucasian)	12	7	19
Education (college or graduate/professional school)	15	7	22
Marital status
Single, never married	5	2	7
Married or domestic partnership	11	4	15
Employment
Employed for wages	10	3	13
Receiving disability	3	0	3
Retired	3	3	6
Annual household income ($US)
More than $75,000	13	4	17
Between $25,000 and $74,999	4	3	7
Less than $25,000	1	1	2
Insurance
Medicare or Medicaid	3	1	4
Private insurance	15	6	21
Treatment characteristics
Years since diagnosis	6	6	6
Duration from diagnosis to first treatment
The same day	4	2	6
Within a week	12	5	17
Any treatment choice given (yes)	4	1	5
Patient’s role in treatment decision-making process
The decision was made together with doctors	9	2	11
Doctors made the decision after considering patient’s opinion	2	0	2
Patients made the decision after considering doctor’s opinion	3	5	8
The decision was made with little or no input from patients	4	1	5
Treatment
Chemotherapy	18	7	25
Ra1diation	5	4	9
Targeted therapy	2	1	3
Allogeneic transplantation	11	5	16
Clinical trial (participated)	8	5	13
Molecular/chromosomal test
Has abnormality (FLT 3 mutation)	6 (3)	4 (3)	10 (6)

As patients and caregivers exhibited the same trend in preferences when stratified, pooled results are presented throughout. Figure 3(A) represents the preference weights and 95% confidence intervals (CI) for each attribute level and indicates that preferences moved in the direction expected: participants preferred higher benefits and wanted to avoid higher risks⁵⁹.

Higher preference weights were associated with better outcomes, and lower preference weights were associated with worse outcomes. Regarding benefits, participants considered a 70% chance of CR as the most preferable outcome (coefficient =3.3; 95% CI =2.8–3.7), approximately twice as preferable as a 40% chance of CR (1.5; 95% CI =1.3–1.7). For EFS, participants preferred 9 months (1.1; 95% CI =0.9–1.4) and 12 months (1.2; 95% CI =0.9–1.5) over 6 months of EFS (0.0; 95% CI = −0.3–0.3), but the preference weights for 9 and 12 months of EFS were not significantly different (p = 0.7). When evaluating the risk attributes, 3 months in hospital was the least preferable outcome (−0.9; 95% CI = −1.2 to −0.6), preferred approximately six times less than 1 month in hospital (−0.2, 95% CI = −0.4–0.1). The preference weights of no time and 1 month in hospital were not significantly different (p = 0.3). As hypothesized, participants wanted to avoid severe (−0.7; 95% CI = −1.0 to −0.4) followed by moderate (−0.2; 95% CI = −0.3 to −0.1) and mild (0.0; 95% = CI −0.3–0.3) short-term side effects. Participants wanted to avoid mild (−0.5; 95% CI = −0.7 to −0.2) and moderate (−0.8; 95% CI = −1.1 to −0.4) followed by no (0.0; 95% CI = −0.3–0.3) long-term side-effects. The preference weights for mild and moderate long-term side-effects were not significantly different (p = 0.2).

Figure 3(B) illustrates the RAI scores. An improvement in CR was most important (RAI =10.0; 95% CI =8.2–11.8), followed by EFS (3.7; 95% CI =2.4–5.1), time in hospital (2.8; 95% CI =1.6–4.0), long-term side-effects (2.3; 95% CI =1.0–3.7), and short-term side-effects (2.1; 95% CI = 0.7–3.4). The pilot test results were consistent with the findings from the qualitative interviews: participants were willing to trade time in the hospital and treatment side-effects for survival benefits.

Discussion

Incorporating the patient perspective into the assessment of treatment benefits and risks has become increasingly important to regulatory authorities^60,61. Specifically, the FDA has attempted to capture the patient perspective and advance the science of community engagement through a patient-focused drug development initiative called The Voice of the Patient⁶². This study showed that patient preferences can be captured in a scientific, reproducible, and quantitative manner to inform a PFDD discussion.

This study also demonstrated that diverse stakeholders can be actively engaged in the process of developing a stated-preference instrument. Notably, the diverse stakeholders not only played a unique and important role as individuals, but also worked together effectively in the development of the survey instrument. Through stakeholder engagement, attributes and levels were selected that were understandable to patients, but also relevant to clinical practice and clinical trial design. For example, easily understood categorical levels (none, mild, moderate, severe) for short- and long-term side-effects were described in a way consistent with the CTCAE to ensure they were clinically relevant.

While preference studies are not currently required in the regulatory review process, the results of the pilot test were presented to the FDA at an independently organized PFDD meeting. At this meeting, FDA representatives discussed how the presented information could be incorporated into clinical trial design. A national rollout of the study and the potential value of study results from a larger and more diverse AML community was also discussed. Methodologically, this study successfully employed a five-step framework that emphasizes the power of a mixed-methods approach to PFDD activities³³.

The pilot test results showed that the participants were able to understand the DCE tasks and that the participants were willing to make tradeoffs between the included attributes. Specifically, the results of the pilot test indicate that participants considered the chance of CR to be the most important attribute and short-term side-effects to be the least important attribute. If these findings were replicated in a large-scale survey, it could provide evidence about patients’ preferences for treatments that have a high benefit and high-risk profile.

This study has several limitations. First, patient and caregiver participants were not necessarily representative of all people affected by AML. For example, the investigators were not able to directly incorporate the perspectives of AML patients who had passed away or those who were experiencing short-term, life-threatening side-effects, although the investigators attempted to incorporate their preferences through caregivers. The viewpoints and preferences of non-participants may have differed from those of individuals who agreed to participate. Second, the identified attributes indicating clinical efficacy might not perfectly represent treatment benefits. A possible explanation for why CR weighed most heavily in decision-making is that participants were more familiar with the concept of CR as a clinical end-point than EFS because CR has been used by medical professionals for several decades^6–8,10,11. Third, participant positivity and recall bias may also have influenced instrument development. In the pilot survey, 23 participants agreed with the statement “I am always optimistic about my future”. The sample size in this study was not large enough to conduct sub-group analyses; large-scale studies need to be administered to a representative sample of the AML community to show the possible differences in the weighting of the relevant decision criteria by patient characteristics. Fourth, caregivers cannot serve as perfect surrogates for patients with AML. In a large-scale study, a separate version of the survey should be created for caregivers, and researchers should formally evaluate the discordance between patient preferences and caregiver preferences.

Conclusion

We have demonstrated how patients, caregivers, and other stakeholders can work together to develop and pilot an instrument to measure treatment preferences and add to the literature on the use of qualitative and engagement methods in the development of preference instruments. This approach to preference research is necessary in order to develop a deeper understanding of the unmet needs and treatment preferences of patient populations. To further the impact of this work, the presented survey instrument needs to be administered to a large and representative population sof patients and caregivers of AML.

KEY POINTS FOR DECISION MAKERS.

• The Leukemia and Lymphoma Society (LLS) initiated an independent effort to promote patient-focused drug development (PFDD). This study presents the development and piloting of a preference study as a first step in this initiative.

• Results of this pilot study were used to guide a PFDD meeting to discuss the lived experience of patients and caregivers affected by AML.

• Productive engagement by all patients, caregivers, and stakeholders throughout the process resulted in strong endorsement of the project’s approach and recognition of the need to conduct a national study.