Development of a discrete choice experiment (DCE) questionnaire to understand veterans’ preferences for tobacco treatment in primary care

Abstract

Background:

Providers often prescribe counseling and/or medications for tobacco cessation without considering patients’ treatment preferences.

Objective:

The primary aims of this study are: 1) to describe the development of a discrete choice experiment (DCE) questionnaire designed to identify the attributes and levels of tobacco treatment that are most important to veterans, and 2) to describe the decision-making process in choosing between hypothetical tobacco treatments.

Methods:

We recruited current smokers who were already scheduled for a primary care appointment within a single VA healthcare system. Subjects were asked to rate the importance of selected treatment attributes and were interviewed during two rounds of pilot testing of initial DCE instruments. Key attributes and levels of the initial instruments were identified by targeted literature review; the instruments were iteratively revised after each round of pilot testing. Using a “think aloud” approach, subjects were interviewed while completing DCE choice tasks. Constant comparison techniques were used to characterize the issues raised by subjects. Findings from the cognitive interviews were used to revise the initial DCE instruments.

Results:

Most subjects completed the DCE questionnaire without difficulty and considered two or more attributes in choosing between treatments. Two common patterns of decision-making emerged during the cognitive interviews: 1) counting “pros” and “cons” of each treatment alternative, and 2) using a “rule out” strategy to eliminate a given treatment choice if it included an undesirable attribute. Subjects routinely discounted the importance of certain attributes, and in a few cases, focused primarily on a single “must-have” attribute.

Conclusion:

Cognitive interviews provide valuable insights into the comprehension and interpretation of DCE attributes, the decision processes used by veterans during completion of choice tasks, and underlying reasons for noncompensatory decision-making.

1. Introduction

The prevalence of smoking is significantly higher in US veterans than in the general civilian population (adjusted odds ratio in men = 1.19 (95% confidence interval = 1.11–1.26)),^1,2 and approximately $2.7 billion in costs to the US Veterans Health Administration (VHA) were attributable to the health effects of smoking in 2010.³ Despite the evidence that telephone counseling and pharmacotherapy significantly improve cessation rates,^4,5 many veterans are reluctant to talk to a smoking cessation counselor⁶ and many others prefer to quit “cold turkey” without any medications.⁷ In 2013, only 25.6% of veterans who currently used tobacco received guideline-recommended pharmacotherapy for smoking cessation.⁸ Those who attempt to quit with medication frequently misunderstand and misuse pharmacotherapy, and relapse secondary to medication nonadherence.^9,10 Historically, providers have selected treatments for smoking cessation without involving patients in the decision-making process, despite the fact that clinicians may not be very good judges of patients’ treatment preferences.^11,12

Patient-based preference models can inform VHA policy makers and operational partners about the utility of current and hypothetical treatment modalities in their ongoing efforts to develop more patient-centered cessation programs. In this regard, the discrete choice experiment (DCE) is a particularly well suited methodology for exploring the relative importance of healthcare attributes in the context of a real-world decision and demonstrating how patients trade among attributes when choices are constrained by limited resources.^13,14 In a DCE, participants are asked to select their preferred option in a series of hypothetical scenarios (choice sets). The DCE approach assumes that patients act rationally when selecting health services, i.e., they choose a particular option only if its utility is higher than the utility of the other options in a given choice set.¹⁵ Moreover, the use of DCE to assess preferences assumes that participants are engaged in completing the choice task, that they understand the information presented, and that they make their choices based on trade-offs between all attributes that are important to them (i.e., they demonstrate compensatory decision-making).^16,17

In actuality, some patients routinely violate normative principles of rational decision making,¹⁸ construct their preferences “on the fly” during completion of choice tasks,^16,19 or use simplifying heuristics that increase measurement error.²⁰ Others make additional assumptions or consider external information and personal experience in making choices. This study describes our efforts to circumvent some of these problems in the development of DCE questionnaires to assess veterans’ preferences for smoking cessation counseling and pharmacotherapy. In addition, this study provides qualitative insight into the decision-making processes used by US veterans in choosing between treatment alternatives, including the decision to opt out.

2. Methods

We used an exploratory sequential mixed methods design, in which qualitative data were primarily used to develop and improve quantitative instruments of cessation counseling and pharmacotherapy.²¹ Our approach to DCE questionnaire development was informed by prior literature (Figure).^15,22,23 During stage 1, we focused on identifying key attributes and levels pertinent to tobacco treatment in the VHA, constructed DCE instruments, and pretested them to assess comprehension and response burden. During stage 2, we pilot tested the DCE instruments, conducted “think aloud” interviews to explore participants’ decision making processes, and iteratively revised the instruments. In stage 3, updated DCE instruments will be administered to a larger sample of veterans in order to estimate part-worth utilities for treatment attributes/levels (quantitative component).

Figure.

Stages of DCE questionnaire development.*Also assessed during stage 2. **Except for “counselor’s emphasis on autonomy” (2 levels)

2.1. Development of DCE choice sets.

To prepare the initial DCE questionnaire (Stage 1), the first author (DAK) conducted a targeted literature search in 2013 to identify key treatment attributes and levels of each attribute that have been used in similar studies of preferences for smoking cessation treatment,^24,25 or that have been described in US guidelines for smoking cessation counseling and pharmacotherapy.⁵ The following search terms were used to find pertinent studies in the literature: (discrete choice experiment or conjoint analysis or patient preference) AND (smoking cessation or tobacco cessation or tobacco use disorder). The list of candidate attributes and levels was reviewed by a panel of study investigators (KRS, MP, MVW, GG) and the final list was determined by consensus. We labeled these attributes and levels using lay language that captures the meaning of each concept.²³ Treatment attributes that met the following criteria were eligible for inclusion in the DCE questionnaire: 1) reflect potentially modifiable features of VA tobacco treatment, 2) are capable of being traded off, 3) are at least somewhat familiar to patients, 4) have minimal overlap in their meaning, and 5) do not perfectly predict individuals’ choice in a deterministic manner.^26,27 Levels were chosen to represent a realistic range over which participants were expected to make tradeoffs; the levels of each attribute were mutually exclusive and were limited to 2–4 per attribute.^22, In the case of continuous attributes, such as quit rates for pharmacotherapy, we used the results of pertinent meta-analyses to inform the selection of plausible levels.^5,29 To promote compensatory decision making, it was also important to select levels with a wide enough range to induce subjects to trade between attributes.²⁰

From this list of attributes/levels, we used the experimental-design macros in JMP (SAS Institute, Version 9) to select an efficient subset of all possible treatment scenarios to develop separate DCE choice sets for cessation counseling and pharmacotherapy (using a fractional factorial design).³⁰ To estimate the independent effect of each attribute with a relatively high degree of precision (i.e., high statistical efficiency), we designed choice tasks that generally satisfied the criteria of orthogonality and level balance.³¹ Specifically, we checked for orthogonality using factor analysis, eliminated any dominated alternatives, and checked for level balance by verifying that each level of each attribute occurred with equal (or near equal) frequency. To reduce task complexity, we allowed some attribute levels to be the same in the profiles being compared (i.e., attribute overlap).³² All choice tasks and survey questions were written at an 8^th grade reading level to maximize comprehension.

As absence of decision-relevant attributes can bias the results of a DCE study,^15,20,33we used semi-structured interviews to explore issues and concepts related to tobacco treatment in VA primary care to identify attributes/levels that may have been missed in our literature review (Appendix 1). Compared to focus groups, in-depth interviews enabled study interviewers to spend more time with subjects and allowed each subject to discuss his/her experiences in greater depth.²³

2.2. Recruitment and sampling strategy.

We recruited active primary care patients in the Iowa City VA Healthcare System who smoked at least 1 cigarette per day (on average) and planned to quit smoking within the next 6 months (18 and 30 patients during Stages 1 and 2, respectively). Eligible patients were randomized in a 1:1 ratio to either the cessation counseling or pharmacotherapy questionnaire. Veterans with altered mental status, acute medical decompensation, history of cognitive impairment (e.g., dementia), communication barrier (e.g., aphasic), and terminal illness were excluded. Current smokers were identified by chart review in advance of a scheduled clinic appointment, and were sent a letter of invitation describing the study and the elements of informed consent. A research assistant (RA) followed up by telephone, confirmed eligibility, and arranged a time for completion of the study questionnaire in the clinic around the time of an already scheduled primary care appointment.

2.3. Data collection procedures.

Completing DCE choice tasks can be cognitively demanding.²² Because of this and the characteristics of our patient population, we administered the study questionnaire using a pencil-and-paper format (rather than by mail, telephone, or internet survey) as it allowed the interviewer to intervene when more explanation was needed, to answer questions, and to document any difficulties the subject had in completing the choice sets. The number of choice sets was limited to 15, as respondents can generally manage up to 16 choice tasks before fatigue and boredom supervene.^22,34 Interviews were conducted in a private exam room by a PhD anthropologist and an observer experienced in qualitative methodologies. In preparation for the DCE choice sets, subjects were asked to rate the overall importance of selected tobacco treatment attributes on a 5-point Likert scale.³⁵ In addition, a warm-up exercise was employed to familiarize subjects with the DCE procedure (e.g., choosing a cell phone plan)(Appendix 2). During stage 1, we included a single choice set to test within-set monotonicity (i.e., to assess whether the subject chose a dominant treatment that included superior levels of all attributes). Non-monotonic choices may indicate that the subject misunderstood the treatment attributes. To assess the test-retest stability of treatment choices (also known as completeness), we included a duplicate choice set at the end of the DCE questionnaire. During Stage 2, we assessed whether subjects’ choices were consistently dominated by a single high priority attribute³⁶ (in violation of the continuity axiom), and confirmed the presence of non-compensatory decision-making in the think aloud data (see below).

During completion of the DCE questionnaire, the interviewer actively encouraged subjects to verbalize everything that went through their minds when completing the choice task (“think aloud” approach).³⁷ If the subject stopped thinking aloud, the interviewer prompted him to reflect back on his choices after every second or third choice task; this approach minimizes unnecessary disruptions of the thinking process.³⁸ This form of cognitive interviewing also provides data on the relevance and clarity of questionnaire items³⁹ and can be used to assess whether subjects use complex decision strategies or simplifying heuristics (e.g., “rules of thumb”).¹⁷ The observer monitored completion time of the DCE questionnaire.

2.4. Qualitative coding and data analysis.

Two qualitative analysts analyzed the think aloud data to evaluate subjects’ overall comprehension of, and any difficulties in completing, the choice tasks. Guided by prior think aloud studies in the literature,^16,40 we constructed a provisional coding structure that was tested using a subset of three transcripts.⁴¹ The coding structure was revised iteratively as new themes emerged. Each set of transcripts was coded by two independent analysts using constant comparison techniques^42,43 and assessed for inter-rater agreement at two intervals (with a preset goal of greater than 80% for all themes).⁴⁴ When agreement was less than 80%, all discrepancies were discussed until a consensus between coders was reached. We attained data saturation after coding the interviews of 30 subjects.⁴⁵ In reviewing the transcripts, analysts attempted to characterize the decision process that subjects used in making a treatment choice, and documented potential instances of non-compensatory decision making. We carefully evaluated “opt out” choices, which may indicate non-compensatory decision-making or implausible treatment options. Although some of these effects are impossible to prevent, our goal was to use the interview data to better understand the decision making process and thereby minimize these problems when revising the DCE questionnaires.

2.5. Quantitative data analysis.

Ratings of attribute importance were summarized by: 1) mean and standard deviation, and 2) the proportion of subjects who rated the attribute as very or extremely important. We calculated intra-rater reliability for duplicate choice sets using the kappa statistic.

3. Results

Characteristics of the study sample are shown in Table 1. Age and gender composition of the study sample were comparable to that of VA tobacco users nationally,⁴⁶ but study subjects were predominantly white. Compared to stage 1 subjects, those interviewed during stage 2 had worse self-reported health and a lower proportion had made a prior quit attempt. Overall, seventy-one percent of subjects had tried to quit previously with medication, and 23% had used cessation counseling or coaching.

Table 1.

Characteristics of study participants^a

Variable	Stage 1 (N=18)	Stage 2 (N=30)
Age, mean (sd)	57.9 (8.0)	56.4 (11)
Gender, % male	94	97
Race, % white	100	90
Married, %	67	50
Education in years, median (IQR)	13.6 (2.2)	13 (12-14)
Income < 200% federal poverty level, %b	33	47
Self-rated health, % fair-poor	22	47
Belief that you currently have smoking-related  medical problem, %	39	33
Belief that quitting smoking would improve your  health, % quite a bit or extremely so	72	67
Cigarettes per day, median (IQR)	20 (10-26)	16 (15-20)
Contemplation ladder, mean (sd)c	7.4 (2.2)	6.6 (2.7)
Prior quit attempt with medication, %	83	63
Prior quit attempt with counseling, %	33	17

^aIQR=interquartile range, sd=standard deviation

^bThe federal poverty level is a measure of income issued every year by the US Department of Health and Human Services to determine eligibility for government programs and benefits. For a family size of one and four in 2013, 200% of the federal poverty level was $22,980 and $47,100, respectively.

^cThe contemplation ladder assesses a smoker’s readiness to quit on a continuum ranging from having no thoughts about quitting to being actively engaged in quitting (range 0–10).

3.1. Attribute development

3.1.1. Rating survey.

We calculated the average rating for each attribute. Overall, patients valued the quality of the smoking cessation counselor’s communication skills and a focus on problem-solving strategies most highly (Table 2a). An emphasis on the patient’s choice on when and how to quit was also highly rated (mean = 3.9 on 1–5 scale, N=48). With regard to pharmacotherapy, patients highly rated medications that effectively relieve cravings for tobacco (mean = 4.2), that have a low risk of behavioral and physical side effects (mean = 4.2 and 4.1, respectively), and that significantly increase the odds of quitting (mean = 4.0)(Table 2b).

Table 2a.

Key attributes of cessation counseling – Rating survey

	Stage 1 (N=18)		Stage 2 (N=30)
	Mean (sd)	Very/extremely important (%)	Mean (sd)	Very/extremely important (%)
1. You should be able to get printed self-help materials on smoking cessation.	3.2 (1.0)	39	2.9 (1.1)	27
2. Cessation counseling should be provided without having to leave your home.	2.9 (1.2)	28	2.5 (1.0)	20
3. Cessation counseling should be provided by a healthcare professional who is familiar with your history.	3.4 (1.1)	50	3.2 (1.2)	45
4. Cessation counseling should be provided by a clinician (doctor or nurse) instead of a non-clinician counselor.	2.7 (1.3)	33	2.3 (1.2)	13
5. Multiple sessions (≥3) of cessation counseling should be provided to improve your chances of quitting.	3.3 (1.1)	44	3.1 (1.1)	40
6. Your cessation counselor should listen very carefully to you.	4.1 (0.7)	83	4.0 (1.1)	87
7. Your cessation counselor should provide clear instructions on quitting.	4.3 (0.7)	89	3.7 (1.2)	77
8. Your cessation counselor should send text or email messages to build your motivation to quit.	3.4 (1.1)	39	2.4 (1.4)	23
9. Your cessation counselor should discuss how to avoid smoking in high risk situations.	4.3 (0.7)	89	3.7 (1.2)	80
10. Your cessation counselor should explain the risks of smoking.	4.1 (0.8)	77	3.2 (1.4)	50
11. Your cessation counselor should explain the nature of nicotine addiction.	4.2 (0.9)	72	3.2 (1.3)	47
12. Your cessation counselor should discuss how to overcome barriers to quitting.	4.3 (0.8)	83	3.8 (1.2)	60
13. Your cessation counselor should help you identify reasons for quitting.	4.3 (0.8)	78	3.2 (1.4)	47
14. Your cessation counselor should emphasize that it is your choice on when and how you quit.	4.1 (0.7)	78	3.8 (1.1)	63

Table 2b.

Key attributes of medication therapy – Rating survey

	Stage 1 (N=18)		Stage 2 (N=30)
	Mean (sd)	Very/extremely important (%)	Mean (sd)	Very/extremely important (%)
1. You should not have to take more than 1 or 2 doses of smoking cessation medication daily.	3.3 (1.1)	61	3.1 (1.4)	47
2. Smoking cessation medication should be provided without any out-of-pocket cost to you.	3.2 (1.2)	50	4.2 (0.9)	80
3. Drugs for smoking cessation should at least double your odds of quitting.	3.8 (0.9)	83	4.1 (1.0)	80
4. Drugs for smoking cessation should have a low risk of physical side effects (e.g., skin rash).	4.2 (0.8)	78	4.0 (1.1)	83
5. Drugs for smoking cessation should have a low risk of psychological side effects (e.g., insomnia)	4.4 (0.7)	89	4.1 (0.9)	83
6. Drugs for smoking cessation should not lead to weight gain more than 5 pounds.	4.1 (0.8)	72	3.4 (1.3)	53
7. You should receive monthly telephone monitoring while you are taking smoking cessation medication.	3.4 (1.3)	39	2.7 (1.4)	30
8. Drugs for smoking cessation should be taken for the minimum length of time needed once you stop smoking.	4.0 (0.8)	72	3.3 (1.2)	47
9. Drugs for smoking cessation should not be highly addictive in their own right.	3.9 (0.9)	67	3.9 (1.2)	70
10. Drugs for smoking cessation should totally relieve your cravings for tobacco.	4.4 (0.7)	89	4.0 (1.0)	77
11. Drugs for smoking cessation should have a quick onset of action.	4.1 (0.6)	83	3.7 (1.0)	60
12. Drugs for smoking cessation should be inconspicuous when you are in public.	3.4 (1.4)	56	2.8 (1.4)	33

3.1.2. Changes between Stages 1 and 2 DCE.

Each attribute was identified as “most important” by at least one subject in Stage 1. In preparation for Stage 2, we made several minor revisions in item phrasing and shortened the description of certain attributes/levels to reduce cognitive burden (Tables 3a and 3b). In the counseling DCE, for example, we sharpened the contrast between levels of “communication skills” and “counselor’s thoroughness” by presenting subjects with a mutually exclusive choice (“always” versus “does not always”). In the medication DCE, we reworded some of the attributes to improve clarity and specificity. For example, “cost for 3 months” was changed to “copayment for 3-month course of drug therapy.” Interview procedures remained the same in both stages.

Table 3a.

Cessation counseling DCE – Item changes over three stages of instrument development

Stages 1 and 2 – Cognitive interviews (item refinement)	Stage 3 – Final instrument	Description of changes (and illustrative quotes)
Convenience	Convenience
▪ Counseling by phone in your own home	▪ Unscheduled counseling on request (“call-back”)	▪ Added unscheduled (“call-back”) counseling to reflect another option in current use
▪ Face-to-face counseling in VA clinic	▪ Scheduled counseling by phone ▪ Scheduled face-to-face counseling in VA clinic
	▪ Scheduled face-to-face counseling in VA clinic
Familiarity of counselor	Familiarity of counselor
▪ Someone whom you usually see	▪ Someone whom you see more than half of the time	▪ Added quantitative labels of frequency to improve clarity
▪ Someone whom you do not usually see	▪ Someone whom you see less than half of the time ▪ Someone who you have not seen before
	▪ Someone who you have not seen before
Type of counselor	Type of counselor
▪ Clinical counselor (e.g., nurse)	▪ Clinical counselor (e.g., nurse)	▪ Added peer counseling as separate level
▪ Non-clinical counselor (e.g., health coach, peer coach)	▪ Non-clinical counselor (e.g., health coach)	▪ Subjects recognized peer counseling as an alternative to counseling by a health professional:
	▪ Peer counselor (coaching by another veteran)	S: I think I’d prefer…a nurse or psychologist over a non-clinic counselor,…[such as] a health coach or peer coach, like a trainer at the gym. They can tell you all the good things, but a nurse or psychologist perhaps would be more knowledgeable and able to help you more. (Subject 123)
Communication skills	Communication skills
▪ Always listens carefully and explains things in a way that is easy to understand	▪ Always listens carefully and explains things in a way that is easy to understand	▪ Added third level to consider a greater range of options
▪ Does not always listen carefully and explain things in a way that is easy to understand	▪ Often listens carefully and explains things in a way that is easy to understand ▪ Sometimes listens carefully and explain things in a way that is easy to understand
	▪ Sometimes listens carefully and explain things in a way that is easy to understand
Counselor’s thoroughness	Attribute dropped
▪ Always considers smoking cessation in the context of your other medical conditions		▪ Dropped on account of non-orthogonality; some subjects also had difficulty with scenarios with discordant thoroughness and communication skills, as exemplified by the following:
▪ Does not always considers smoking cessation in the context of your other medical conditions		S: Um, I couldn’t, I can’t, seem to separate between someone that considers your medical condition [and someone who listens carefully], um, which I consider a big item. I would even consider someone considering your medical condition over the success rate. Um, but if they don’t listen, then, how can they consider your medical condition? Um, it, it, seems to contradict itself, uh, with someone that does not listen to you versus someone that always considers your medical condition in Group A, or Option A, um, that, that’s a conflict in my book. (Subject 113)
Follow-up	Follow-up
▪ 1–2 sessions	▪ 1–2 sessions	▪ Added third level to consider a range of options that are consistent with current practice
▪ 3 or more sessions	▪ 3–4 sessions
	▪ 5 or more sessions
Quit rate at 1 year	Quit rate at 1 year
▪ 14%	▪ 10%	▪ Expanded range of quit rates, consistent with current evidence
▪ 18%	▪ 16%
	▪ 22%
Format of counseling
▪ Group counseling with other veterans		▪ Dropped - not a plausible option in combination with other attributes (i.e., convenience”)
▪ Individual counseling
	Counselor’s emphasis on autonomy
	▪ Emphasizes that it is your choice on when and how to quit	▪ Added new attribute based on results of rating survey and feedback from subjects, as illustrated by the following:
	▪ Does not emphasize that it is your choice on when and how to quit	I: Um, is there anything else you would like the research team to know as far as patient preferences for smoking, or anything you think might be helpful? S: (long pause) (exhales heavily) Can’t really think of much. Um, I know drugs work for some guys and others, you know, cold turkey like I like to do it. I mean--, but there’s gotta be some flexibility built into that program to where the smoker can choose his own way, gets supported however he happens to choose to try. (Subject 128)
	Type of counseling aid between sessions
	▪ Print materials (e.g., brochure on quitting)	▪ Added new attribute based on feedback from operational partner and input from patients. Some subjects felt that a combination of face-to-face counseling and follow-up checks (using various modes of communication) would be optimal:
	▪ Interactive website (e.g., internet based information on quitting)
	▪ Mobile phone messaging (e.g., text messages on quitting)	I: Um, would you prefer face-to-face or over the phone? S: Uh, actually a combination of both. I: OK. So, what would that look like? S: Well it would probably have to start with a face-to-face, um, and in between face-to-face visits, uh, you know, phone calls, uh, emails, texts, whatever. (Subject 113)

Table 3b.

Medication DCE – Item changes over three stages of instrument development

Stages 1 and 2 – Cognitive interviews (item refinement)	Stage 3 – Final instrument	Description of changes (and illustrative quotes)
Dosing schedule ▪ Once daily ▪ Twice daily ▪ Eight daily	Treatment schedule ▪ Fixed dose (once or twice daily) ▪ Symptom-triggered (i.e., multiple times daily as needed) ▪ Both fixed dose and symptom-triggered (i.e., combination therapy)	▪ Revised attribute to reflect choice between fixed dose or symptom-triggered dosing (de-emphasized dose frequency) ▪ Several subjects had difficulty with the idea of taking a medication eight times daily, as exemplified by the following:  “That’s the only deciding factor in that one, cause they both got the same dosing schedule, ain’t no way I’d take eight pills, eight medications a day, once a day is enough.” (Subject 104)
Risk of medication-related side effects (e.g., skin rash, nausea) ▪ 10% ▪ 20% ▪ 30%	Risk of physical side effects due to medication (e.g., skin rash, nausea) ▪ 8% ▪ 16% ▪ 24%	▪ Divided side effects into two separate attributes ▪ Some subjects highlighted the importance of physical side effects, as in the following example:  I: Right. Um, so with the patches, you said that they bother you. Can you describe a little bit more, like, what-  S: Uh, [they] give me a skin irritation...make my skin hurt. (Subject 120)
	Risk of behavioral side effects due to medication (e.g., sleep problems) ▪ 4% ▪ 6% ▪ 8%	▪ Divided side effects into two separate attributes ▪ Some subjects highlighted the importance of behavioral side effects, as in the following example:  I: Did you notice any negative side effects?  S: Well, that’s just it. Of course, something along with my other medications and I, you know, I was gettin’ cranky and “rrrnnng” (makes a sort of snarling sound). And um, so I just, I weaned myself off of the stuff I could and that was one of ‘em. (Subject 118)
Weight gain during process of quitting ▪ 5 pounds ▪ 10 pounds ▪ 15 pounds	Weight gain during process of quitting ▪ 5 pounds ▪ 10 pounds ▪ 15 pounds	▪ No change
Telephone monitoring ▪ None ▪ Single check-in after starting medication ▪ Monthly check-in	Telephone monitoring ▪ None ▪ Single check-in after starting medication ▪ Monthly check-in	▪ No change
Copayment for 3 months of drug therapy ▪ $0 ▪ $12 ▪ $24	Copayment for 3 months of drug therapy ▪ $0 ▪ $24 ▪ $48	▪ Expanded range of copayments for drug therapy ▪ For several patients, the difference between the initial levels of copayment was considered to be unimportant, as in the following example: “I’m not real concerned with the copayments on this [$0 vs. 24]…I mean, if anybody wants to quit, I don’t think they’d be concerned either.” (Subject 128)
Quit rate at 1 year ▪ 12% ▪ 18% ▪ 24%	Quit rate at 1 year ▪ 10% ▪ 16% ▪ 22%	▪ Expanded range of quit rates, consistent with current evidence

3.1.3. Changes between Stages 2 and 3 DCE.

Based on lessons learned during this stage, we included three levels for all counseling attributes (except for the autonomy item, which had two levels) in order to consider a greater range of options (e.g., peer counseling) and a wider range of levels for certain continuous attributes (e.g., quit rate)(Table 3a). In cognitive interviews, for example, several subjects felt that the difference between a 14 and 18% quit rate was not clinically important; thus, the final instrument included three levels with a 12% spread. Because some descriptors of frequency (e.g., usually) are potentially vague, we also added quantitative labels to improve clarity (e.g., more than half of the time). We also added two new attributes based on the results of the rating survey (e.g., emphasis on autonomy) and in response to feedback from our operational partner (Director of Tobacco and Health at the VHA Office of Public Health). We dropped one attribute (counselor’s thoroughness) because of non-orthogonality with another attribute (counselor’s communication skills)(Table 3a).

Based on cognitive interviews, we also revised the medication DCE (Table 3b). For example, several subjects indicated that taking a medication eight times daily was not an acceptable option. Instead of assessing the utility of dose frequency, we revised this attribute to reflect the underlying concept, i.e., the choice between fixed dose or symptom-triggered dosing (or their combination). This attribute also tapped into subjects’ stated desire for rapid relief of nicotine withdrawal symptoms. Moreover, we divided the side effects domain into two separate attributes: physical and behavioral. In addition, we estimated the excess risk of side effects relative to placebo based on data from the EAGLES trial, which examined the safety of various smoking cessation medications in patients with and without psychiatric disorders.⁴⁷ We also expanded the range of copayments for drug therapy to reflect the cost of possible combination therapy and because several subjects had indicated that the difference between levels in Stage 2 was relatively unimportant. The final attributes and levels to be assessed in Stage 3 are shown in Tables 3a and 3b, and sample choices sets are shown in Appendix 3.

3.1.4. Findings from Stages 1 and 2.

The first 2 iterations of the DCE questionnaire identified several attributes of tobacco treatment that were found to be important to subjects on the rating survey. With regard to counseling, it seemed clear that “always listening” was critically important and emphasized the importance of building rapport; the type of counselor (clinician vs non-clinician) and his/her familiarity with the patient did not seem to be important. Subjects preferred telephone counseling mostly because of convenience, but some patients felt that “it would be better in person because then that way the person knows if you’re B.S.-in’ ‘em” (Subject 110).

With regard to medication, subjects tended to focus on quit rate and side effects; other factors, such as copayment, were regarded as secondary concerns.

3.2. Insights related to DCE administration and decision processes

The first part of this section focuses on issues related to comprehension and intra-rater reliability. In the remainder of this section, we have grouped subjects’ quotes from the think aloud interviews into two categories: 1) general issues related to completion of the DCE questionnaire, and 2) decision-making processes.

3.2.1. Monotonicity, completeness, and non-compensatory decision making.

We assessed within-set monotonicity in Stage 1: 100% of subjects selected the dominant choice set in both DCE questionnaires. For the counseling DCE, intra-rater agreement for duplicate choice sets was 56 and 60% in Stages 1 and 2; the corresponding values of kappa were 0.23 and 0.21, respectively. For the medication DCE, intra-rater agreement was 78 and 80%, but kappa improved significantly (from 0.50 to 0.70, respectively), suggesting that preferences for medication treatment were more stable during Stage 2. During Stage 2, five subjects demonstrated single-attribute non-compensatory decision-making (17%, confirmed by the think aloud data).

3.2.2. General issues.

Mean completion times for the counseling and medication DCE questionnaires during Stage 1 were 21 and 18 minutes; the corresponding times during Stage 2 were 26 and 17 minutes, respectively. Most subjects (S) completed the DCE questionnaire without difficulty; however, not all subjects were fully engaged, and in some cases, the interviewer (I) documented signs of cognitive overload or exhaustion. In a few cases, subjects struggled a bit with the idea of choosing between two hypothetical medications, as shown in the following example:

S: Mark what, now?

I1: So, basically you’re weighing Option A and B and you’re looking at all the attributes for--

S: (Overlapping) Now this has nothing to do with reality ‘cause me and him’s already talked about it.

I2: Yeah, mm-hmm…These aren’t actual treatment plans that we’re gonna ask you to pick from now. (Subject 111)

In other cases, the subject questioned the veracity of the values (levels) used in the DCE questionnaire, or asked the interviewer for clarification of the meaning of attributes or levels used in the experiment. For example, the following subject appeared to misread the numerical information pertaining to side effects in the medication DCE:

I: What’s your question about the side effects?

S: Okay, does that mean that your odds is twenty percent of NOT having any effects?

I: Of having.

S: Of having. Okay, that’s kinda what I thought... (Subject 126)

Some patients recalled the attributes of a preferred treatment profile in a prior choice set and opted out when presented with choice sets that did not include the preferred treatment option (suggestive of an anchoring effect):

“I: But if you’re weighing what attributes are most important to you, where would you end up?

S: Well I can’t say all of A or all of B cause they, crisscross. So I have to prefer, I’d have to take neither of the options. How do I do that?

I: Well you would check neither. (laughter)

S: Because these, these here, you know, on the other page they was over here.

I: So--

S: You can’t-- go ahead.

I: Ok. So, imagine that the other page just didn’t even exist. Each page is taken in isolation but it’s the same scenario. (Subject 119)

3.2.3. Decision-making processes.

Most subjects showed good comprehension of the choice tasks and clearly considered two or more attributes in choosing between medications. Two common patterns of decision-making emerged during the cognitive interviews: 1) counting “pros” and “cons” of each treatment alternative, and 2) using a “rule out” strategy to eliminate a given treatment choice if it included an undesirable attribute. Use of the former strategy compared treatment options by simply counting the number of preferred attributes within each alternative, as exemplified by the following:

I: Why Option B?

S: Face-to-face counseling, clinical counselor, individual counseling. And always considers the smoking cessation with your medical conditions - four. Three over here (pointing to Option A). (Subject 102)

In another case, the subject explicitly described weighing the attributes’ relative importance in addition to counting “pros” and “cons”:

I: Ok, why A.

S: More options that I like. I like these three. But these have more options that I like and the options that I like are more important. So I would have to go with A on this one.” (Subject 119)

Subjects routinely discounted the importance of certain attributes, as in the following example: “I don’t think telephone monitoring [of medication] would affect me one way or another, so it--,…uh, I would probably end up goin’ with Plan A on this one.” (Subject 122) Despite use of a counting heuristic (in which the desired attributes in a given treatment profile are simply tallied up, with equal weighting), some subjects were indifferent between the two treatment alternatives, resulting in a “toss-up.” In a few cases, subjects zeroed in on a single attribute and were not willing to trade, regardless of the values of other attributes. For example, the following subject seemed to focus exclusively on the quit rate: “Oh, on this one, it’s still A. The, the quit rate is higher whether--, to me, it doesn’t matter. If you really want to quit, that’s, [I: Yeah.] that’s your GOAL.” (Subject 124) For the same subject, out-of-pocket cost became a key determinant of choice when quit rate was held constant across treatments: “And, on this one... I’d have to go with A. There’s no cost. Quit rate’s the same... Just want to quit smokin’, but I want to be cheap, too.” (Subject 124)

Several patients used a “rule out” strategy to eliminate a given treatment choice if it included an undesirable attribute (“deal breaker”). For example, one subject felt strongly that the counselor’s communication skills were an essential attribute of good cessation counseling and had already determined his desired level of this attribute. In this scenario, the subject had clear preferences as he read through the attributes until he reached the point where options A and B offered the same attribute for communication skills:

“S: For follow-up, Option A. They’re both equal. I would choose Option A. And the quit rate, Option A, ‘cause [of] the greater rate, but (long pause) they both don’t listen to me so (laughter)…mmm…neither one of them listen to me… If I had to choose, I would prefer neither. I would prefer neither (long pause). Even though that one gotta a better success rate, he still ain’t gonna listen to me.” (Subject 106)

There was also evidence that subjects’ choices were influenced by prior experience with specific treatments or broader perceptions of the treatment context (see example in Table 4).

Table 4.

Examples of decision processes during completion of DCE choice tasks

Pertinent code (attribute)	Selected quotes
Counting	S: Convenience - face-to-face versus counselling by phone. I like counseling by phone. Familiarity with the counselor - doesn’t make a difference. Type - clinical versus non-clinical. Doesn’t make a difference. Format - would rather do group. Thoroughness - always considers smoking cessation. It’s more preferred than doesn’t consider. Follow-up - prefer two to three but that’s not an option so I’ll say three. Quit rate - higher percentage. Communication skills - doesn’t always listen versus always listens. I prefer always listens. I had more boxes checked in Option B. (Subject 105)
Multiple attributes (medication)	S: Choice set 3. Dosing schedule - both are eight daily. Co-payment - $0 and 24. Um, even though there’s a 24 dollar difference over the period of three months, I’m gonna be quitting smoking [so] basically I can afford it. Uh, quit rate at one year - both are 24%. Uh, risk of …medication related side effects - 20% and 10%. My weight gain same, telephone monitoring a single check-in [versus] none. So those two are about equal again. So once again, even though option B is $24 [and] the other one is $0, the risk of medication related side effects is 10% [versus 20%]... Option B. (Subject 101)
Single attribute (counseling-communication)	S: …You always want someone that’s gonna listen carefully and explain things in a way that’s easy to understand, (Long pause) you know. And so, you, you, you, you’ve gotta go with that, you know, you gotta make that the number one priority. (Subject 103)
Single attribute (medication-quit rate)	S: Eh, copayment for three months, uh, twelve bucks. One’s free. Quit rate’s twenty-four percent. That’s looking better. Risk of medication side effects, ten percent for that one. Umm, I would go with Plan A. I: And why Plan A? S: Nahh, just, uh, the quit rate. (Subject 124)
Main factor (counseling-type of counselor)	S: …Hey, you know I think I’d rather speak to a nurse. Um or a counselor versus a doctor. Cause to me doctors have a tendency to be uppity. And that’s specially from my experience here. Yeah, (mimicking voice) “I’m the doctor. I know.” “You don’t know shit, jack.” (Subject 108)
Main factor (medication-side effects)	Choice set 1…Risk of medication related side effects - 10%, 20%. Not a big fan of any type of side effects so that’s a big... downer for me for option B. Amount of weight gain, um, whether I gain 5 pounds or 10 pounds for me that’s no big deal. Uh, telephone monitoring option A has none. Option B has a single check-in... over a three month period so. (long pause) I’m not all that impressed with that. So out of these two, I would definitely take option A mainly [because] the risk of medication related side effects is only 10%. With everything else being fairly equal. (Subject 101)
Deal breaker (counseling-convenience)	I: And then would you prefer telephone or, uh, face to face? S: (overlapping) No, I don’t like, I don’t like telephone. I like looking people in the eye and tell ‘em, “This is my belief,” and hopefully they don’t take, you know, an attitude towards the way I’m telling things... (Subject 119)
Deal breaker (counseling-follow-up)	S: …Single check-in, monthly check-in. Well, if you didn’t have the telephone monitoring question in there I would probably answer more [i.e., select choice A or B], but [my choice is] neither. Because I’m on the free phone plan and I got limited minutes and I can’t, I don’t want to waste, uh, my minutes, being called about smoking. So, if you didn’t have that in there, you’d have a different result. (Subject 116)
Deal breaker (medication-weight gain)	I: All right. Up to [choice set] seven. S: (pause) Neither, none of ‘em. I: And why neither on this one? S: Weight. Y’know. I mean, sure, that’s not--, I mean, these are good odds, but I just don’t want it because I just, I don’t want it because of the weight. (Subject 120)
Unimportant factor (medication-weight gain)	S: …Go with option A ‘cause I’m not worried about the weight gain. ‘Cause if I can quit smoking, hopefully I’ll start exercising more (laughs). (Subject 101)
Toss-up (medication)	S: Dosage schedule - both are 8 daily. Co-payment - 12 or 24 doesn’t really matter. Quit rate 12 and 18 - I definitely like the 18% over the 12%. Uh, side effects are the same at 30%. 15 or 5 pounds. Monitoring, monthly check-in or none. Um… wow. These are about even if you’d ask me, just because I ain’t worried about the co-pay. Uh… the quit rate…I lean toward option B because it’s higher, but I also like the telephone monitoring - at least you get monthly check-in. So I’m kind of tied between those two, because I’m not worried about the weight gain. So I’m gonna go ahead and mark neither of these options. I: So neither because they’re both-- S: --equal. (Subject 101)
Prior experience (medication-side effects)	S: (long pause) Take Plan A. Um, quit rate, you know, better. I mean, Option A, I guess, you know, twenty percent... it’s a big percentage. I mean, I guess, like I said, I’ve had some side effects from medicine that I don’t care for and, and it’s, uh you know, it’s--, I just don’t want it. I mean, I--, my brother was on Chantix and he said it was just--, I mean, he quit, you know, for a long time, but then he said the side effects were just crazy, so. (Subject 120)

4. Discussion

Challenges faced by the VA population with regard to quitting tobacco include socioeconomic deprivation, a high burden of medical and psychiatric comorbidities,⁴⁸ and a military culture that historically has normalized tobacco use.^49,50 This study demonstrates the value of using qualitative methods to develop a DCE questionnaire to explore the preferences of veterans regarding smoking cessation treatment. Our findings also illustrate the importance of piloting treatment attributes and giving clear instructions on completion of choice tasks.^23,51 Cognitive interviewing provided valuable insights into the comprehension and interpretation of DCE attributes, the decision processes used by subjects during completion of choice tasks, and the underlying reasons for noncompensatory decision-making and “opt out” choices. Regarding the latter, we found that qualitative analysis revealed strong preferences for (or aversion to) particular treatment attributes. These findings can facilitate interpretation of DCE results, as it may not always be clear from the choice data what subjects assume about the opt out alternative.²⁰ Although this study focused on smoking cessation treatment, our approach can be used to assess face and content validity of DCE choice sets more generally.⁵²

This study highlights some of the practical issues related to the use of DCE to assess patient preferences. Although comprehension of the DCE task was generally good, subjects may misinterpret quantitative risk information, which may be aggravated by lack of numeracy skills.^53,54 A few subjects were observed to rush through the questionnaire without carefully considering the treatment options or avoided difficult choices by opting out. Others struggled to engage because choice tasks were hypothetical and they may have had difficulty conceptualizing treatments with which they had little prior experience. This may explain why intra-rater reliability for cessation counseling was relatively low during pilot testing. Furthermore, the responses used to assess stability can be noisy, as subjects attempt to understand the choice task (first choice set) and as fatigue sets in (last choice set).⁵⁵ Indeed, we observed that subjects tended to learn which attributes/levels were most important to them during the process of completing the initial choice sets.

There was clear evidence that most subjects weighed the trade-offs between multiple attributes; however, a few individuals made choices based on a single attribute, as reported by others,^56,57 or used other simplifying heuristics. This form of noncompensatory decision-making may reflect a truly dominant preference for a single attribute (in violation of the continuity axiom) or it may reflect how some patients deal with a complex choice task. With regard to the latter, some individuals tend to be “maximizers” who are inclined to invest substantial cognitive effort into determining the best treatment option, whereas others are “satisficers” who tend to select the treatment option that is “good enough” with the least possible effort.⁵⁸ Time pressure, task complexity, motivation, mode of administration, and prior experience with the treatment or service in question have also been shown to influence the use of heuristics to simplify choice tests.^57,59,60 Pretesting of the DCE questionnaire is recommended to ensure comprehension of the choice tasks, to assess whether the response burden is acceptable, and ultimately to reduce the use of simplifying heuristics. We also believe that having a researcher present during administration of the DCE increases the quality of the data collected.

This study has several limitations that warrant discussion. First, even though subjects were prompted to think aloud during completion of the DCE questionnaire, they showed varying levels of engagement and did not always articulate their reasons for making a particular treatment choice. Indeed, some subjects verbalized only considering a single attribute for a particular choice set, but individual utility models suggest that these same subjects in fact considered multiple attributes during the process of selecting treatment (data not shown). Second, we were unable to determine from the think aloud data the extent to which specific biases (e.g., halo effects, memory effects) may have influenced treatment selection. Third, we did not assess health literacy (or numeracy), which may affect subjects’ comprehension of the DCE attributes and their likelihood of trading off between multiple attributes.¹⁷ Fourth, the test of within-set monotonicity that we used in Stage 1 can be relatively easily passed by chance alone.⁵⁵ In addition, the use of a single duplicate choice set may not have accurately captured the global test-retest stability of patient preferences in our study sample.⁵² Cognitive interviews provided unique insight into the participant’s reasoning and complemented these quantitative tests of choice validity and reliability. Fifth, we selected subjects who were contemplating cessation or actively preparing to quit; level of engagement in the DCE choice tasks would likely be lower in subjects who are less ready to quit. Sixth, for logistical reasons, we recruited a convenience sample of study subjects who were already scheduled for a primary care visit. Use of maximum variation sampling may have allowed us to further explore how treatment selection varied across specific patient characteristics or prior experience. Finally, we recruited a convenience sample of subjects from a single VA healthcare system in the Midwest; subjects’ use of language and comprehension of the study instruments may differ in other parts of the country.

Conclusions.

This study demonstrates the feasibility of using DCE methods to elicit the preferences of veterans for smoking cessation treatment, and provides guidance for other investigators about the role of qualitative methods in developing DCE instruments. Qualitative work at the outset of this project provided a deeper understanding of the treatment context and confirmed the relevance of attributes that were derived from the literature and clinical experience, while later work helped to refine attribute/level descriptors to ensure that their intended meaning was conveyed.²³ In particular, cognitive interviews enabled the study team to assess comprehension and interpretation of DCE attributes, the decision processes used by veterans during completion of choice tasks, and underlying reasons for noncompensatory decisionmaking. Future research should explore the use of DCEs to provide real-time feedback on the relative importance of treatment attributes to individual patients and their clinicians, to facilitate shared decision-making in the clinic, and to develop new tobacco treatment programs that are informed by veterans’ preferences.

Key Points for Decision Makers.

• 1)Discrete choice experiments can be used to elicit the preferences of veterans in order to develop more patient-centered smoking cessation treatment programs.
• 2)Cognitive interviews can help to refine the attribute and level descriptors, as well as to ensure task comprehension, in a discrete choice experiment.
• 3)Analysis of think aloud data provides insight into the cognitive strategies used by patients when comparing treatment alternatives and can help to explain noncompensatory decision making and opt-out choices.