Automated Computer Interviews to Elicit Utilities

Abstract

Objective: To assess the practicality of an automated computer interview as a method to assess preferences for use in decision making. To assess preferences for outcomes of deep vein thrombosis (DVT) and its treatment.

Study Design: A multimedia program was developed to train subjects in the use of different preference assessment methods, presented descriptions of mild post-thrombotic syndrome (PTS), severe PTS and stroke and elicited subject preferences for these health states. This instrument was used to measure preferences in 30 community volunteers and 30 internal medicine physicians. We then assessed the validity of subject responses and calculated the number of quality-adjusted life years (QALYs) for each individual for each alternative.

Results: All subjects completed the computerized survey instrument without assistance. Subjects generally responded positively to the program, with volunteers and physicians reporting similar preferences. Approximately 26.5% of volunteers and physicians had preferences that would be consistent with the use of thrombolysis. Individualization of therapy would lead to the most QALYs.

Conclusions: Utilization of computerized survey instruments to elicit patient preferences appears to be a practical and valid approach to individualize therapy. Application of this method suggests that there may be many patients with DVT for whom treatment with a thrombolytic drug would be optimal.

The treatment of deep venous thrombosis (DVT) is controversial.^1,2,3,4 Use of a thrombolytic drug in combination with intravenous heparin can reduce the risks of post-thrombotic syndrome (PTS), a long-term complication of DVT³ that is associated with significant morbidity.^5,6,7 However, the addition of a thrombolytic drug increases the risk of bleeding,⁸ which can lead to a stroke or even death.⁴ Thus, the decision to treat patients with DVT with thrombolysis requires physicians to balance short-term risks from iatrogenic causes against long-term benefits. In 1980, a National Institutes of Health consensus panel stated that the risk-to-benefit ratio was generally favorable and recommended broad use of thrombolytic drugs to treat DVT.⁹ However, must DVT patients continue to be treated with intravenous heparin alone.

The decision to use a thrombolytic drug to treat DVT fits well into the framework of decision analysis. In 1994, O'Meara et al.¹⁰ developed a decision model for DVT therapy. They found that treatment recommendations depended heavily on patient preferences and proposed the use of these preferences to determine who should receive treatment with streptokinase plus heparin. The degree to which patients believed their quality of life would be lowered by the development of PTS was found to be the key factor in identifying patients for whom the benefits of thrombolytic therapy outweighed its risks. O'Meara et al. applied utility theory¹¹ in order to measure preferences and determine if the benefits from prevention of PTS were great enough to warrant the risks of adding streptokinase. The population in which O'Meara studied preferences was a small group (n = 36) of general medicine patients. About 44% of these patients had previously been treated for DVT.

The specific method that O'Meara et al. used is called the Standard Gamble (SG).¹¹ In this procedure, subjects are asked to evaluate a series of gambles with p of normal life coupled with a complementary probability (1 - p) chance of death (▶). The utility, or quality of life weight, assigned to a health state such as PTS occurs when the patient cannot decide which he or she prefers: life with the health condition or p of normal life coupled with (1 - p) probability of death (▶). The SG utility assessment for a health state is generally perceived to be a difficult¹² and time-consuming task to perform. Few health care providers have the skills and/or ready access to the appropriate visual aids to utilize this method. Further, in order to use this method for decision making, the physician must show the patient how the health conditions affect longevity and quality of life. For example, to apply utility assessment in patient decision making for DVT, patients would need to be shown what PTS is and its effect on quality of life. These logistical problems have largely precluded the use of preference elicitation to support decision making in clinical environments.

Figure 1.

Standard Gamble preference assessment method. The Standard Gamble is the index method to measure preferences for health conditions. To measure a patient's values for a health condition using this method, the assessor assists the patient with determining the probability P where the patient is indifferent between two alternatives: (1) living the remainder of his or her life with some health condition; and (2) a gamble with P probability of normal life coupled with 1 - P probability of immediate death.

The computer offers a potential solution for describing potential health outcomes as well as for measuring preferences. We combined multimedia methodology for describing potential health states and computer surveys to elicit preferences into a decision-support program. The decision-support program would allow preference elicitation at the bedside, enabling widespread application of decision analyses to direct therapy. This paper describes our experiences with developing and testing a pilot program to direct therapy for DVT based on patient preferences.

Methods

We developed an interactive multimedia computer program that described PTS or stroke¹⁰ using video and audio.¹³ Our health state descriptions were adapted from medical books.¹⁴ The health states are described as follows:

• Mild post-thrombotic syndrome: In mild post-thrombotic syndrome, chronic or lifelong mild leg swelling and skin changes occur. The chronic swelling and skin changes can sometimes be complicated by infection. The mild leg swelling and skin changes may cause some discomfort or cosmetic problems. With this condition, you should be able to walk normally. You should also expect a normal life span.

• Severe post-thrombotic syndrome: In severe post-thrombotic syndrome, chronic leg swelling and intermittent leg ulceration occur. The leg swelling will be lifelong and may cause some limitations in your ability to walk or stand for prolonged amounts of time. The ulcer that occurs intermittently is often large and may take months to heal. In some patients, this ulcer causes mild to moderate pain. In others, this ulcer does not cause any pain. This ulcer will require daily care (such as a bandage or a dressing) and may require regular visits to your doctor's office. The above pictures show an example of the chronic leg swelling and the recurring leg ulcers that are typical of this health problem. Although severe chronic venous insufficiency should not affect your life expectancy, it would certainly lower your quality of life because of pain, leg swelling and the other problems that it causes.

• Stroke: We would like to compare your opinions about the quality of life with post-thrombotic syndrome with your opinions about the quality of life in a more severe disease. The disease that we will compare it to will be a stroke. There are many different ways that both young people and old people can experience severe damage to the brain from a stroke. To keep things simple, we would like you to consider only one type of stroke, even though there are many different types and degrees of severity of stroke. Keep this particular type of stroke in mind when you answer the next set of questions. Imagine a stroke that causes paralysis of the right side of the body and causes the patient to be unable to understand spoken and written language. This means that the patient will not be able to walk. The patient will not be able to communicate by spoken or written language. He or she will require much assistance with daily activities.

Pictures illustrating each description were obtained from a medical textbook,^14,15 faculty slide collections, or role playing. Pictures were processed using Photoshop 4.0 (Adobe, Mountain View, CA) for Macintosh computers (Apple Computer, Cupertino, CA) coupled with the text descriptions of each health state.^13,16 The program was developed using IMPACT, an authoring system designed to allow health professionals to construct multimedia utility survey instruments.¹⁷ All instructions were displayed on the screen and read aloud using PlainTalk software (Apple Computer, 1994).

Subjects used a computer mouse to respond to instructions. They were then taught to use two preference assessment methods: a three-dimensional, animated visual analog scale (VAS) similar to the Torrance Health Thermometer,¹⁸ and an SG measurement for utilities. The subjects received descriptions of mild and severe PTS and stroke sequentially; they were asked to rate their imagined quality of life in each of these health states.

Using a VAS, subjects imagined and rated their own quality of life in the given health state (▶). After this series, subjects repeated the process with the SG method¹¹ illustrated in ▶. In addition to collecting preference ratings, the survey collected data on each subject's age, sex, educational level, and number of children. Lastly, the program asked subjects to rate their overall experience using the survey instrument on a five-point Likert scale (highly positive, positive, positive and negative, negative, and highly negative). They also rated the clarity of the PlainTalk synthetic voice using a five-point Likert scale (clear and pleasant, mostly clear and generally pleasant, often unclear and sometimes distracting, often unclear, and always unclear).

Figure 2.

Multimedia computer-based implementation of the Visual Analog Scale. The scale uses a double anchored scale with death at the bottom and perfect health at the top. An illustration of severe post-thrombotic syndrome is also shown. Subjects can click anywhere on the scale to provide their ratings for quality of life. They can adjust the level by dragging the level up and down using the mouse pointer.

Figure 3.

Multimedia computer-based implementation of the Standard Gamble (SG). To make the SG preference elicitation task as similar as possible to the types of decisions that patients ordinarily face, we used real-life scenarios and a visual representation of the levels of risk patients may encounter through treatment. The SG utility for a health condition is approximately one minus this maximum acceptable risk of death. This picture shows a point early in the assessment process at which the computer is determining if a patient would take a small risk of death to avoid life with post-thrombotic syndrome.

We recruited 30 healthy women between the ages of 20 and 40 from the communities surrounding our institution by placing flyers in shopping malls and other public areas. We also administered this survey to a group of 30 physicians, 15 of whom were chosen randomly from the second-year residents in the Stanford Internal Medicine Program. The other 15 were a complete sample of faculty members from the Departments of Family Practice Medicine at Stanford University and General Internal Medicine at the Livermore Division of the Palo Alto Veterans Affairs Health Care System. None of the subjects had formal training in decision analysis.

The study was approved by the Institutional Review Board for Human Subjects at Stanford University Medical Center. Volunteers and resident physicians were reimbursed $10 for participating. Data were collected between September and December 1994. The decision model in the O'Meara paper was re-implemented using Excel 4.0 (Microsoft Corp., Redmond, WA), and statistical analyses were performed using Systat 5.2 (Systat Inc., Evanston, IL).

Results

The distribution of demographic variables is shown in ▶. Volunteer subjects who participated in the study were highly educated; all except one had college or graduate degrees. About half of the subjects either were married or had been married, and a minority (27%) had children. Resident physicians were predominantly male. Attending physicians were older than other subjects and were more likely to have children.

Table 1.

Demographic Data of Participants Included in Study

		Medical Doctors
Characteristic	Volunteers	Resident	Attending	All Subjects
No. of subjects	30	15	15	60
Age (yr)
20 to 30	18	10	1	29
31 to 40	12	5	7	24
Older than 41	0	0	7	7
Median	25	25	35	35
Gender: female/male	30/0	5/10	9/6	44/16
Education
High-school	1	0	0	1
College and beyond	29	15	15	59
Marital status
Single	15	12	1	28
Married/previously married	15	3	14	32
No. of subjects with children
No child	22	14	3	39
One child	2	1	2	5
More than two children	6	0	10	16

All subjects successfully completed the preference assessment survey and were able to provide utilities for all three health states. After being shown how to work the trackball interface, all subjects went through the entire program, including training in preference elicitation techniques, without assistance. They typically required between 20 to 40 minutes to complete the program. Subjects reported that using the program was either a generally positive (93%) or very positive (85%) experience (▶). Most (79%) subjects rated the PlainTalk synthetic voice, used to describe health states and to provide instructions on use of preference assessment methods, as either “clear and pleasant” or “mostly clear and generally pleasant.” There were no statistical differences between physicians and volunteer subjects in their subjective ratings of the program or in their ratings of the acceptability of synthesized speech.

Figure 4.

User acceptance of the program. The SG preference assessment requires patients to contemplate their own mortality; this can be an unpleasant experience for some people. At the end of the computer interview, subjects were asked to evalute the experience using this program on a five-point Likert-type scale (Very Positive, Generally Positive, Positive and Negative, Generally Negative, and Very Negative). More than 90% of subjects described the experience of using the program as generally positive or very positive despite the potentially unpleasant nature of the tasks required during the computer interview.

If subjects were able to understand our questions and express their preferences coherently, then preference data should have rational properties. To examine the validity of preferences elicited, we considered logical consistency and consistency of responses across methods of assessment.¹⁹

Logical consistency indicates that if one health state has all the limitations and pain of another health state as well as additional impairments, then preference or utility values for that condition should be less than or equal to the utilities for the condition with fewer impairments. To assess logical consistency, we compared utility values for mild and severe PTS. In all subjects, utilities for severe PTS were less than or equal to utilities for mild PTS as measured by both VAS and SG methods. All subjects also rated the utility of stroke to be lower than the utility of severe PTS.

Internal consistency indicates that the rank order of preferences would be the same across preference assessment methods (e.g., if utility of mild PTS > severe PTS > stroke by SG, then the VAS rank should be the same). Consistency of ratings across methods was observed for all subjects.

Preferences for each of the health states were similar between subject groups. There were no differences in visual analogue scale rating for mild PTS, severe PTS, or stroke between volunteers and physicians (mean overall ratings ± SD were 0.77 ± 0.14, 0.61 ± 0.17, and 0.25 ± 0.22). The SG utilities of volunteers and physicians were also similar and are shown in ▶. Many subjects had utilities for mild post-thrombotic syndrome that were less than 1.0 (mean ± S.D was 0.98 ± 0.04). Thirty-seven percent of subjects were willing to accept a 1% risk of death or more to avoid mild PTS. Seventy-seven percent of subjects were willing to assume a 1% or greater risk of death to avoid life with severe post-thrombotic syndrome. The median utility for the severe post-thrombotic syndrome was 0.95 (e.g., half of the subjects were willing to take more than a 5% risk of death to avoid life with the syndrome). The mean utility for severe PTS (0.93 ± 0.07) was significantly lower than the utility of mild PTS. Using multiple regression analysis, we did not detect any significant relationships among the utility values for mild PTS, severe PTS and stroke, and the age, gender, marital status, number of children, and education of participants.

Table 2.

Utilities Values Obtained by Standard Gamble for Selected Outcomes According to Group

			Medical Doctors
Health State	All Subjects (n = 60)	Volunteers (n = 30)	All (n = 30)	Attending (n = 15)	Resident (n = 15)
Mild Post-thrombotic syndrome*†	1.00 (0.91-1.00)	1.00 (0.91-1.00)	1.00 (0.91-1.00)	1.00 (0.88-1.00)	1.00 (0.97-1.00)
Severe post-thrombotic syndrome*†	0.95 (0.79-1.00)	0.93 (0.76-1.00)	0.95 (0.83-1.00)	0.97 (0.82-1.00)	0.92 (0.84-1.00)
Central nervous system bleeding*†	0.60 (0.02-1.00)	0.66 (0.08-1.00)	0.50 (0-1.00)	0.70 (0.25-1.00)	0.45 (0-0.88)

^*Median values are given, with 95% confidence interval in parentheses.

^†Mean utility value of a health state within the same group differs (P < 0.001) as compared with the other two states.

We used the decision model described by O'Meara et al. to estimate the change in quality-adjusted life expectancy with each treatment for each subject, based on that subject's preferences and his or her life expectancy estimated from age- and sex-specific lifetables (▶). A significant minority of subjects (30% of volunteers and 23% of physicians) had preferences resulting in combination therapy using streptokinase plus heparin as being the optimal choice (the proportion of volunteers was not statistically different from the proportion of physicians—Pearson Chi-square 0.34, P > 0.56). However, the gains in QALYs predicted for this minority were larger than the losses predicted for the majority. Thus, combination therapy was the superior strategy, resulting in gains of 0.13 QALYs in volunteer subjects and 0.06 QALYs in physicians, compared with therapy using heparin alone.

Table 3.

Comparison of QALYs After DVT According to Therapeutic Method

	Quality-Adjusted Life Years
Study and Subgroup	Heparin alone	Heparin plus streptokinase	Individualization of therapy
Our study
All subjects	46.16	46.29	46.42†*
Volunteers subjects	50.10	50.32	50.44†*
Physicians	42.19	42.25	42.40*
O'Meara et al. study	19.71	19.64	NA

^†P < 0.01 for the comparison with QALYs using heparin alone.

^*P < 0.01 for the comparison with QALYs using a combination of heparin plus streptokinase.

The gains realized in individualizing therapy based on preferences were substantial. Individualized treatment based on preferences resulted in an average gain of 0.26 QALYs compared with treatment with heparin alone, and an average gain of 0.13 QALYs versus treatment of all patients with heparin plus streptokinase.

Discussion

PTS can be associated with physical, emotional, and financial consequences.^5,6,7 Lindholm et al.⁵ and Phillips et al.⁶ have examined the effects of chronic leg ulcers on quality of life. Both studies found that leg ulcers frequently caused moderate to severe pain, decreased mobility, and increased loss of income from disability. Many patients also have significantly negative changes in emotional health.^5,6 A prospective study⁷ involving 24 patients with DVT found that pain, swelling, and discoloration in the previously affected leg were common in patients who developed PTS and that these changes lowered overall health-related quality of life.⁷ These studies support the argument that preventing severe PTS is worth assuming some risk.

We found that in 26.5% of subjects, treatment with streptokinase and heparin was the optimal therapy.

However, the gain in QALYs in this subgroup was substantial (as much as 1.3 QALYs). In fact, the expected gains in this subgroup dominated the expected losses in the majority of patients who favor treatment with heparin alone, resulting in treatment with streptokinase being the optimal choice.

The results of this study differed from the results reported by O'Meara et al.¹⁰ They found that few subjects were willing to assume even small levels of risk to avoid life with severe PTS—no one in the study had preferences consistent with the use of streptokinase. Two factors may explain the observed differences. We used a multimedia methodology to describe the health states, allowing subjects to receive video and audio descriptions of the health states associated with DVT and its treatment. These extensive descriptions (reviewed by vascular and internal medicine specialists) are potentially more realistic than written descriptions, thus increasing the subjects' understanding of the implications of the different health states. Subjects who fully understand the different health states and the assessment questionnaire would be more likely to express their preferences rationally. In this study, all subjects expressed preferences with rational properties. Another factor contributing to the differences in study results was the age of our subjects. Younger subjects, such as those in our study, may be more willing to assume risks to gain health benefits than older subjects are.²⁰ However, the results of preference elicitation in physicians were identical to those seen in our normal subjects. Physicians, who presumably had independent knowledge of the effects of PTS on quality of life from the information presented in our survey, also frequently had preferences consistent with the use of streptokinase.

Our results indicate the need to individualize therapy based on patient preferences. Additionally, in order to make the right recommendation in this circumstance, a physician needs precise knowledge of patient preferences. This level of precision can only be achieved through formal measurement techniques. However, is it practical for a physician to assess patient preferences before deciding whether or not to use streptokinase in the treatment of DVT? While utilities assessment has rarely been used to guide therapy, treatment of DVT is a clinical situation in which formal preference measurement could easily be incorporated into the diagnostic process. The preference assessment instrument we have developed runs on portable computers that can be brought to the patient's bedside. Most patients with acute DVT have normal mental status and could provide insight into their preferences at the time of clinical decision making. Although assessment of preferences might briefly delay the start of heparin therapy, this delay poses minimal risk.

The use of multimedia methods to describe the potential outcomes of treatment has been pioneered by the Foundation for Informed Decision Making.^21,22 This organization has developed a series of video discs aimed at informing and assisting patients with critical medical decisions. The Foundation's programs differ from ours in several ways. For example, there are differences in production costs. A single video disc can cost more than $1,000,000;²² our software was developed at the cost of a few thousand dollars. Also, the style of decision support offered by these programs is markedly different. Rather than eliciting preferences from patients, the video discs describe the probabilities of different therapeutic outcomes. In addition, they attempt to help patients make decisions by presenting testimonials from model patients. While this approach may educate and assist patients with decision making, it does not ensure that decisions are optimal. Decisions made by patients using these discs on hinge on the patients' perceptions of quality of life in one or two key health states,²³ the significance of which might be better ascertained using formal preference assessment.

Use of bedside computer interviews to measure preferences will potentially solve many of the logistical problems that have prevented the use of decision analysis to direct therapy. Using the software described in this paper or other systems for utility elicitation²⁴ at the bedside may improve both patients' participation in therapeutic decisions and the overall quality of medical decision making. This study demonstrates that use of utility elicitation software is feasible at the bedside. All subjects completed the interview and all had utilities values that were logically consistent, suggesting a 100% success rate for the utility elicitation process. Utility elicitations required about 30 minutes to complete and were performed using portable computers that could literally be brought to a patient's bedside. While we did not test the acceptability of using utility elicitation in decision support in this study, we have examined this issue in another study. Subjects, using similar software, felt the tasks required in utility elicitation interviews were reasonable (78%), and about 52% said they would feel comfortable using this approach to direct medical decisions.¹⁷

Our study also exemplifies the need to include quantitative measures of patient preferences in therapeutic decisions. A substantial minority of patients (26.5%) had preferences consistent with the use of streptokinase in addition to heparin to treat DVT. Gains from treatment with streptokinase in these individuals were significant, increasing their quality-adjusted life expectancy by up to 1.3 QALYs. Further study of the clinical usefulness of computer interviews to direct therapy is warranted.