Condition-Specific Performance Information: Assessing Salience, Comprehension, and Approaches for Communicating Quality

Abstract

This study assesses how consumers view condition-specific performance measures and builds on an earlier study to test an approach for communicating quality information. The study uses three separate designs: a small experiment, a cross-sectional analysis of survey data, and focus groups. We test whether providing information on the health care context affects consumer understanding of indicators. Focus groups were used to explore how consumers view performance measures. The cross-sectional survey analysis used survey data from the experiment and the focus groups to look at comprehension and the salience of condition-specific performance measures. Findings show that a general consumer population does view condition-specific performance measures as salient. Further, the findings provide evidence that information on the health care context makes a difference in how consumers understand performance measures.

Introduction

Current efforts to inform consumers about the relative quality of health plans are rapidly expanding. However, empirical evidence about the efficacy of various approaches to informing consumers is not yet available. Further, we are just beginning to learn what kinds of quality information consumers want. Consumer interests or preferences have not been the primary drivers for the development of quality indicators (QIs), but rather expert perspective and large purchaser interests. Condition-specific performance measures are among the newer QIs being adopted. Almost nothing is known about how a general consumer population will view and understand such measures. The purpose of this study is twofold: to assess how consumers view condition-specific performance measures (breast cancer) and to build on the findings of an earlier study to test an approach for communicating quality information.

Background

The health sector is changing rapidly; these changes have profound implications for consumers. With an increasing percent of the population entering managed care plans that limit provider choice, plan choice may be more consequential than ever before. New incentives encourage providers to limit care and do less for patients. Consumer satisfaction information and informed consumer choice are key counter-balances to incentives that could result in underserving plan members.

Most report card efforts include some or all of the Health Plan Employer Data and Information Set (HEDIS) QIs. HEDIS, sponsored by the National Committee for Quality Assurance (NCQA), is a collaborative effort to develop and disseminate comparative information on the quality of health plans. HEDIS includes major areas of plan performance measurement, including rates of preventive measures among plan members, appropriateness of care measures, and patient satisfaction. In addition, HEDIS provides information on the efficacy of the management of chronic diseases such as asthma and diabetes. The original HEDIS indicator set was developed specifically to inform the decisions of large employers and the quality improvement efforts of health plans. With the development of HEDIS 3.01, however, NCQA made a commitment to incorporating measures that would be useful to consumers choosing among health plans.

A new organization, FACCT, has recently emerged to establish standardized performance measures relevant for both consumers and purchasers. FACCT is just beginning its work and will develop both population-based and condition-specific measures of performance. Condition-specific QIs have the distinct advantage of providing information on outcomes.

Little is known about how consumers will use clinical performance measures or how condition-specific measures will be understood and used. Focus group studies have reported that consumers want to know about health plan performance from “people like me” (National Committee for Quality Assurance, 1995). This has implications for how consumers may view condition-specific measures. Most of the members of health plans are healthy. Just looking at plan performance in relation to care of its healthy population is not a very good test of its performance. It is useful also to assess how well the plan performs when it has the greater challenge of caring for sick patients. Ideally, one would want to know how the plan would take care of “me” if I should become seriously ill. Thus, an important question in the development and dissemination of performance measures is to assess the degree to which consumers, and particularly well consumers, will understand and use information about how well the plan treats members when they are seriously ill. Both because these measures are potentially useful to consumers and because quality assessment is moving in this direction, the question of how a general consumer audience will understand and use condition-specific QIs is important.

Consumers' Use of Quality Information

Several researchers have identified factors that increase the likelihood consumers will want and use comparative health plan information. Among the elderly, Davidson (1988) found that being female, being white, and having a relatively high income were all associated with a higher level of interest in comparative benefit information on plans. Juba, Lave, and Shaddy (1980) having a positive attitude toward innovation and change, feeling vulnerable economically, and being risk averse as factors that influence a consumer's likelihood of comparing health plan benefits and making a change. McGee and Hunter (1992) interviewed State of Minnesota employees after the first publication of comparative consumer satisfaction data on health plans and identified factors that predisposed certain employees to be more interested in this information. Those considering a change of health plan or those forced by circumstances to make a change were most interested in the comparative information. Hibbard and Jewett (1996) found that while consumers preferred quality information about prevention and consumer satisfaction, when it came to making a choice, these indicators were less important than indicators that reflected adverse events (and therefore a perceived threat in a situation with low personal control).

Barriers to Consumers' Comprehension of Quality Information

A number of factors may affect consumers' ability to comprehend QIs. Quality assessments reflect the complexity of the medical care process and the many dimensions of medical practice. For example, several studies have found confusion regarding the safety and effectiveness of breast and cervical screenings among the general population and minority populations (Fulton et al., 1991; Cockburn et al., 1992). This presents a barrier to viewing performance indicators based on rates of these screening tests as a measure of quality (Jewett and Hibbard, 1996). Furthermore, many QIs require some comprehension of probabilities and risk assessment; concepts that are difficult both to understand and communicate. Jewett and Hibbard (1996) found that understanding quantitative concepts and aggregations was a significant barrier for consumers' comprehension of quality information. This was particularly true for Medicaid enrollees and the uninsured.

The rapid evolution and growing complexity of delivery system options also present barriers to consumer comprehension. To understand what a plan is and how it works is no small feat. The multiplicity of health plan structures, payment systems, and benefit designs makes it difficult for consumers to comprehend plan options. In addition, competing plans often share overlapping panels of physicians, making them difficult to differentiate. A recent national Harris Poll showed that 63 percent of adults said they do not have a good understanding of the difference between fee-for-service (FFS) health insurance and managed care (Louis Harris and Associates/Towers Perrin, 1995). Fifty-five percent said they had never heard the term “managed care.” Most did not know about the major features of managed care (for example, the gatekeeper role of primary care physician, or the emphasis on prevention).

Earlier Study

In an earlier investigation, Hibbard and Jewett investigated consumer perspective on current QIs (Hibbard and Jewett, 1995; Jewett and Hibbard, 1996). Consumer comprehension and perceived salience were assessed utilizing content analysis based on focus group data and survey data. The study population included three groups: a privately insured, a publicly insured (Medicaid and expanded Medicaid), and an uninsured population. The findings show that while consumers are very interested in quality information, much of the information is not well understood. Not surprisingly, those indicators that are not well understood are also viewed as not very useful for choosing a plan (Hibbard and Jewett, 1995). The findings also showed that one of the largest barriers to understanding QIs was not understanding the health care context. That is, quality measurement is largely based in the assumptions and the processes of the new health care context of managed care (Hibbard and Jewett, 1995; Jewett and Hibbard, 1996) These assumptions include:

• Plans are responsible for the health care of a defined population, and quality can be measured with a population-based approach.

• Plans are accountable for improving and maintaining the health of their populations.

• Plans can influence the quality of care delivered by the providers and hospitals in their networks.

These assumptions are vastly different than the assumptions operational under a FFS environment. Most consumers are more familiar with a FFS environment and do not understand these new assumptions. When consumers did not see that QIs told them about quality, it was often because they did not agree with or understand one of the aforementioned assumptions.

In capitated or managed care systems, a valid assessment of performance is to evaluate how well a plan cares for and maintains the health of a defined population. Even though these concepts are relatively new to consumers, current efforts to disseminate quality information (health care report cards) do not provide information on either of these two concepts (systems of care or a population-based approach to quality assessment), or even on how managed care plans work.

A basic assumption, not universally held by consumers, is that quality differences exist in plans and hospitals Jewett and Hibbard, 1996). The entire approach of disseminating information to influence the health care market rests on this assumption. Both from a system-performance perspective and from a consumer-protection perspective, it is crucial that consumers understand that these differences do exist.

It is not surprising that consumers don't see the plan's role in influencing the quality of care. The linkages between the plan, the hospital, and the physicians are not visible to consumers. Consumers are unaware of how incentives, policies, and practices shape physician-referral patterns, rates of preventive care, use of expensive procedures, and management of chronic conditions. Understanding that the plan has a role in and can influence the quality of care delivered by the providers and hospitals in the plan is an essential prerequisite to viewing QIs as meaningful and plan choice as relevant.

Thus, it appears that consumers do not understand the new health care context. This is a barrier for viewing quality information as relevant. Further, it is a barrier to truly informed choice.

This study builds on the findings of this previous work; its goals are twofold: to gain consumer perspective on the usefulness and understandability of one set of condition-specific performance measures, and to determine whether providing consumers with information about the health care context will improve understanding of performance measures. The condition-specific indicator set used in this investigation focuses on plan performance with regard to breast cancer.

Methods

Research Questions

• Are breast cancer performance measures salient to a general consumer population (e.g., those without the condition)?

• Which breast cancer measures are most salient?

• What are the meanings that consumers ascribe to the indicators?

• Are there differences between those who received information on the health care context (“context group”) and those who did not (“no context group”) in how they understand the measures?

• How well are the breast cancer performance measures understood?

Study Design

The study includes three separate designs: a small experiment, a cross-sectional analysis of survey data, and focus groups. The small experiment was conducted to test whether providing information on the health care context affected consumer understanding of the indicators. Respondents were randomly assigned to receive performance measure definitions and additional written health care context information or to receive the performance measure definitions only. Both groups responded to the same surveys (N=17 per group). The surveys are designed to assess the salience of the indicators, to explore what meanings consumers ascribe to the indicators, and to assess comprehension.

Also, focus groups were used to explore how consumers view the performance measures. Focus group members discussed how they perceived the condition-specific measures and afterward responded to the same surveys used in the experiment. Thus, the focus group discussion provides insight into how consumers view the indicators and data from these discussions add richness to the interpretation of the survey data. Four focus groups were conducted with a total of 38 men and women. Focus groups lasted about 2 hours and were video- and audio-taped.

The cross-sectional analysis used the survey data from both the experimental groups and the focus groups to look at comprehension and salience of condition-specific performance measures (N=72).

No comparisons are made between the survey responses of the focus group members and the survey responses of the experimental group. Because the focus group members had a chance to discuss both the context information and the QIs before they responded to the surveys, they essentially had a different “intervention” than the experimental group. It would be difficult to interpret the meaning of any differences that might be observed between the groups.

Experimental Intervention

The written, two-page health care context information explained:

• What a health plan or a system of care is.

• What a good health plan should be doing for members.

• That there are differences in the quality of care in different health plans.

• The ways that plans can influence quality.

• Why it might be important to know how well the plan does on caring for seriously ill patients (even if you yourself are well).

• How the choice of a plan is a consequential decision.

Both the “context group” and “no context group” received definitions of the performance measures.

Study Population

Persons studied were recruited through calls to randomly chosen names off voter registration lists for the Eugene/Springfield, Oregon area, and were paid for their participation. To increase the homogeneity of the focus groups (thus facilitating discussion), participation was limited to those between the ages of 35-55 who had 14 years or less of education. Breast cancer patients were not included. Health care workers were excluded, as were those with no health insurance. Participants for the experiment were then randomly assigned to either the “context group” or the “no context group.”

Selecting participants from names off voter registration lists likely introduces some bias into the sample. People who vote may be more likely to see that their choices can make a difference. Thus, if there is a bias, it would suggest that the respondents are more likely to be more interested in the performance information than the general public.

Table 1 shows the demographic distribution of the study population by group (focus groups, the “context group,” and the “no context group”). Although there are some differences in the demographics, none are statistically significant. The study population, by definition, is limited in at least two demographic factors, age and education. For the total study population (N=72), 56 percent are female. The mean age is 44 years. Sixty-nine percent of the population is employed. Sixty-three percent report being enrolled in a health maintenance organization.

Table 1. Demographic Characteristics, by Group.

Characteristic	Total (N=72)	Focus Group (N=38)	Experiment Group (N=17)	Control Group (N=17)
Percent Female	56	55	56	56
Average Age (Years)	44	42	45	43
Percent With at Least One Child at Home	58	58	56	62
Percent With High School Education or More	75	82	67	69
Percent Employed	69	74	72	56
Percent in HMO	63	67	69	46
Percent With Oregon Health Plan (Expanded Medicaid)	22	18	28	25
Percent Who Reported Health Good or Excellent	87	84	89	94
Percent With at Least One Chronic Condition	46	39	50	56

NOTE: HMO is health maintenance organization.

SOURCE: Hibbard, J.H., Jewett, J.J., University of Oregon, Department of Planning, Public Policy and Management, and Sofaer, S., George Washington University Medical Center, 1996.

Surveys

In addition to demographic information, the surveys focused on three areas: consumers' ratings of the importance of each performance indicator in helping to select a health plan; consumers' perception of the type and amount of information each indicator provides; and comprehension of the indicators. In the first questionnaire, respondents were asked to rate the importance of each of seven indicators in helping select a plan. Ratings ranged from one (not important) to five (very important). In the second questionnaire, respondents were asked to rate how much information each indicator gave them about six aspects of care (0=no information; 5=a great deal of information). The six aspects of care were derived, in part, from the framework developed and used by the Office of Technology Assessment (1988) in its study on quality information for consumers.

Finally, respondents were asked to write, in their own words, what they thought three selected indicators were intended to tell about. The three indicators selected were those that appeared to be the least understood: “lumpectomy rates,” “5-year survival,” and “quality of life for breast cancer patients.” The written responses were coded for level of comprehension.

Responses were coded on a 3-point scale: low comprehension (1), some comprehension (2), and adequate comprehension (3). Scores of (1) were given when the answers did not contain any part of an acceptable definition or when the respondent said he or she had “no idea.” Scores of (2) were given where there was some partial acceptable definition or glimmer of understanding. Scores of (3) were given when the answers were mostly correct or had the general idea of at least one important component of the definition. Consensus coding by two raters of the scored data were carried out.

Focus Groups

The focus group guide was developed and refined using two pilot groups conducted prior to the study. To control moderator bias, all explanations, instructions, questions, and prompts were verbatim from scripts in the guide. Questions from participants were directed to other group members or delayed until the conclusion of the session. First, report card efforts, quality-of-care measures, and managed care systems were explained to consumers (including the context information). Then they were shown sample report cards and definitions of the breast cancer indicators. Before discussion started, focus group members responded to written ballots, choosing which three indicators were most useful for selecting a plan. Next, in a group interview, each participant was asked to discuss the choices they made on their ballot and the reason for their choices. Open discussion followed. For the ballot portion, the responses are uncontaminated by group discussion. Surveys were completed at the end of the focus groups. It is recognized that the participants' responses were likely influenced by the group discussion.

Breast Cancer Performance Measures

The breast cancer measures are one set of condition-specific measures being proposed by the FACCT. This study is a first step in assessing how a general consumer audience will understand and use such performance measures. The measures are intended to include indicators of clinical outcomes, quality of life, patient satisfaction, burden of disease, early detection, and essential medical practices (e.g., breast-conserving surgery or lumpectomy rates). The specific measures are:

• Breast cancer patient satisfaction measures (information, timeliness of tests, access, interpersonal communication).

• Five-year survival rates (by stage of diagnosis).

• Quality of life for breast cancer patients.

• Days unable to work (burden of disease).

• Lumpectomy rate.

• Percent of patients diagnosed in early stage.

• Mammography rate for older female plan members.

Findings

The findings are organized around the major research questions.

Are Breast Cancer Performance Measures Salient to a General Consumer Population? Which Breast Cancer Performance Measures Are Most Salient?

Salience of the breast cancer performance measures is assessed using both survey data and written ballot responses from the focus groups. Survey respondents were asked how important it would be to know about each indicator, for example, “how well the plan and their doctors do on early detection of breast cancer” in choosing a health plan or doctor group. Respondents could respond from 1 (not at all important) to 5 (very important).

Most of the breast cancer performance measures are rated very high in importance (Table 2). Interestingly, the second lowest rated indicator is mammogram rate, the only one that is directly applicable to a well population. The most salient indicators are “early-stage diagnosis” and “patient satisfaction.” “Five-year survival” is the third highest rated indicator.

Table 2. Indicators Rated Most Important for Choosing a Plan (Survey Data).

Indicator (N=72)	Mean Score	SD	Rank
Breast Cancer Patient Satisfaction	4.37	0.98	2
Breast Cancer Patients' Quality of Life	3.82	1.25	5
Days Breast Cancer Patients Are Too Sick to Work	3.06	1.35	7
Percent of Breast Cancer Patients Still Alive After 5 Years	4.17	1.24	3
Percent of Breast Cancer Patients Diagnosed in Early Stage	4.61	0.90	1
Percent of Breast Cancer Patients Who Had a Lumpectomy	4.03	1.14	4
Percent of Older Women Who Had a Recent Mammogram	3.36	1.14	6

NOTES: SD is standard deviation. Score:1=not important; 5=very important.

SOURCE: Hibbard, J.H., Jewett, J.J., University of Oregon, Department of Planning, Public Policy and Management, and Sofaer, S., George Washington University Medical Center, 1996.

The data from Table 2 are based on all respondents and each indicator is rated for importance. A slightly different picture emerges from the focus group ballot data (Table 3). Focus group members were asked to select the top three breast cancer QIs that would be most useful to them in selecting a plan or doctor group. The indicators chosen as useful show that mammogram rates are more important than what emerged in the survey data. Mammograms are third most often chosen in the top selections in the focus groups. Most often selected are “early-stage diagnosis” and “patient satisfaction.” “Five-year survival” was chosen by only 44 percent of participants in their top selections of indicators (Table 3).

Table 3. Most Useful Indicators Chosen From Focus Group Ballots: Percent of Respondents Who Included Indicator in Top Three Choices.

Indicator (N=38)	Percent Choosing Indicator in Top 3 Choices	Rank
Whether Breast Cancer Patients Are Satisfied With Their Care	55.3	2
How Breast Cancer Patients Feel About the Quality of Their Lives	42.1	5
How Many Days Breast Cancer Patients Are Too Sick to Work	2.6	7
How Many Breast Cancer Patients Are Still Alive After 5 Years	44.7	4
What Percent of Breast Cancer Patients Were Diagnosed in Early Stage	78.9	1
What Percent of Breast Cancer Patients Had a Lumpectomy	26.3	6
What Percent of Women Had a Recent Mammogram	50.0	3

SOURCE: Hibbard, J.H., Jewett, J.J., University of Oregon, Department of Planning, Public Policy and Management, and Sofaer, S., George Washington University Medical Center, 1996.

Some gender differences are found in the selection of the most useful indicators. Women were more likely to include “quality of life” in their top three selections (52 percent) than were men (29 percent, p=.05). Men more often included “early-stage diagnosis” (94 percent) in their top three choices than did women (66.7 percent, p=.03).

Surprisingly “5-Year Survival” did not receive a higher rating. Some of the comments from the focus groups reveal why this might be:

• “I wouldn't want to know how long I had to live. I would not want that kind of information.”

• “There are too many factors that come into play with surviving to tell if it has anything to do with the care.”

• “Five years is way down the road; if I were in that situation I could not think that far ahead. Just coping with the immediate things would be important.”

Focus Group Discussions

Focus group members also told us the reasons behind their indicator selections. The reasons given illuminate what participants think the measures mean. The overwhelming reason given for selecting “early-stage diagnosis” was that it told about early detection and prevention. Some comments indicated that they saw this measure as key. That is, if the rate of early stage diagnosis was high, this told them a lot about other aspects of care for breast cancer care (including whether patients were likely to survive). This emphasis on early detection and prevention is not surprising. It makes sense that it is at this end of care that “well patients” would be most interested.

Those who chose “patient satisfaction” as a useful indicator said the reason was that it told them about whether the breast cancer patients were confident and feeling good about their care. They saw this as linked with a quicker recovery. Some said that patient satisfaction told them about how well patients were coping with their disease. Many said that patient satisfaction information tells them almost everything they need to know about the quality of care.

Finally, the survey data provides an additional piece of information about the salience of condition-specific performance measures. A survey item asked whether they want performance information on how well the plan does with all its members, how well the plan does on taking care of seriously ill patients, both types of information, or neither. The findings show that 43 percent want information on all patients, while 56 percent want both types of information (information on all plan members and information on seriously ill). Thus, there does seem to be some interest in condition-specific measures among a general consumer audience.

What are the Meanings That Consumers Ascribe to the Indicators?

Survey instruments were used to determine what meanings consumers ascribe to the indicators. Survey questions explore what aspects of care consumers think the indicators tell them about, including overall quality of breast cancer care. Also assessed is whether the indicators reveal any information about the overall quality of the care for the general plan population.

Survey questions asked if each indicator tells anything about:

• Whether the plan educates and informs its breast cancer patients.

• Whether the plan doctors give breast cancer patients the referrals they need.

• Whether breast cancer patients are treated with respect and caring.

• Whether the medical staff has good technical skills for treating breast cancer.

• Whether breast cancer patients are coping and functioning well.

• Whether the quality of care overall is high for breast cancer.

• Whether the quality of care overall is high for the plan patients, in general.

Respondents rated how much information each indicator provided (0=no information; 5=a great deal of information).

Table 4 shows the seven breast cancer performance measures and the top three aspects of care that respondents felt the indicator told about.

Table 4. Patient Satisfaction With Amount of Information Indicator Provides About Different Aspects of Care: Top Three Meanings Ascribed to Indicators (N=72).

Indicator	Average Rating
Patient Satisfaction with Breast Cancer Care
Overall Quality of Breast Cancer Care	3.45
Respect and Caring for Patients	3.38
Patients Are Informed and Educated	3.24
Quality of Life for Breast Cancer Patients
Patients Are Coping and Functioning	3.33
Respect and Caring for Patients	3.10
Overall Quality of Breast Cancer Care	3.03
Days Too Sick to Work
Patients Are Coping and Functioning	2.06
Overall Quality of Breast Cancer Care	1.53
Technical Skill of Medical Staff	1.23
Five-Year Survival Rates
Technical Skills of Medical Staff	3.43
Overall Quality of Breast Cancer Care	3.21
Referrals to Needed Services	3.10
Early-Stage Diagnosis
Technical Skills of Medical Staff	3.74
Referrals to Needed Services	3.63
Respect and Caring for Patients	3.17
Lumpectomy Rates
Patients Are Informed and Educated	2.89
Referrals to Needed Services	2.48
Overall Quality of Breast Cancer Care	2.24
Mammogram Rates
Patients Are Informed and Educated	3.28
Overall Quality of Breast Cancer Care	2.44
Referrals to Needed Services	2.41

NOTE: Rating 0=no information; 5=a great deal of information.

SOURCE: Hibbard, J.H., Jewett, J.J., University of Oregon, Department of Planning, Public Policy and Management, and Sofaer, S., George Washington University Medical Center, 1996.

Respondents seem to understand, in general, what the indicators were designed to tell about. For example, respondents thought that the indicators “5-year survival” and “early-stage diagnosis” told them the most about the technical skills of the medical staff.

It is clear that consumers thought that some of the indicators gave a good deal of information and some indicators gave very little information about any aspect of care. This is consistent with the salience findings. Those indicators rated as most important are also the ones that consumers think are giving them the most information about different aspects of care.

Respondents thought that “patient satisfaction,” “5-year survival rate,” and “early-stage diagnosis” gave the most information about the overall quality of breast cancer care. While respondents didn't think that the breast cancer QIs told them much about the overall quality of care in the plan for all plan members, the indicators that gave them the most information on overall quality were: “patient satisfaction,” “quality of life for breast cancer patients,” and “early-stage diagnosis” (all receiving average ratings of about 2.0). Thus, while consumers found the information salient, they did not see that it told them much about the overall quality of the plan.

Are There Differences Between the “Context” and “No Context” Groups in Understanding the Performance Indicators?

Almost no differences on the salience of the individual breast cancer indicators are found between the two groups. However, differences between the “context” and “no context” groups on the overall salience of condition-specific information was observed. Eighty-nine percent of the “context” group rated “knowing how well seriously ill patients are treated,” as very important, while only 50 percent of controls rated this information as very important (p=.05). This difference was also true for “the importance of knowing how well breast cancer patients were treated.”

Similarly, definite differences emerged when consumers rated how much information indicators gave about the different aspects of care. The “context” group consistently rated the indicators as giving more information about the different aspects of care (Figure 1). For example, the “context” group thought that the indicator “quality of life for breast cancer patients” gave them a great deal of information on how well the plan educates and informs patients (4.2 on a scale of 0-5), while the “no context” group thought it gave only a moderate amount of information (1.9). Similarly, the “context” group thought that the indicator “lumpectomy rates” gave them much information on the technical skills of the medical staff (3.3), while the “no context” group thought it gave less information on the medical staff technical skills (1.8).

Figure 1. Differences Between “Context” and “No Context” Groups on Meaning Ascribed to Indicators: Top Three Meanings for Each Indicator.

Looking at the data another way (not shown in Figure 1), 38 percent of the “no context” group said that knowing about the lumpectomy rates gave no information about how well the plan doctors “educated and informed” patients. Only 11 percent of the “context” respondents thought this gave no information about how well the plan informed and educated. Respondents rated how much information each of 7 indicators gave them about 6 different aspects of care, or 42 ratings. Ratings from the respondents in the the “context” group indicated that the measures gave them more information on 40 of the 42 possible responses. Fourteen (35 percent) of these differences were great enough to reach statistical significance.

Thus, the respondents who received the context information thought that the indicators gave more information about the different aspects of care. Because the context material laid out what a good health plan should be doing and how the plan can work to achieve better quality, it is not surprising that the “context” group had a better understanding of what the indicators told about.

How Well Are the Breast Cancer Performance Measures Understood?

All respondents were asked to write, in their own words, what they thought the three selected indicators were intended to tell about. The three indicators were those judged as the least understood: “lumpectomy rates,” “5-year survival,” and “the quality of life for breast cancer patients.” The written responses were coded for level of comprehension on a 3-point scale: low comprehension (1), some comprehension (2), and adequate comprehension (3).

Table 5 shows the comprehension scores for the three selected indicators. “Lumpectomy rate” was the poorest understood, followed by “quality of life,” and then “5-year survival.” Twenty-seven percent of all the respondents scored low on all three indicators. Women tended to score higher on the comprehension of the indicators than men (some differences reach statistical significance). Those with some college generally had higher comprehension scores than did those with only a high school education or less. However, there are no real differences between the “context” and “no context” groups on the comprehension measure. This is likely because the context information does not give information about the specific indicators.

Table 5. Comprehension Scores on Three Selected Breast Cancer Indicators (N=72).

Indicator	Mean Comprehension Score	Percent Scoring Low on Comprehension
Lumpectomy	1.56	54.2
Quality of Life	1.63	50.0
5-Year Survival	1.95	27.8

NOTE: 1 = low comprehension; 3=adequate comprehension.

SOURCE: Hibbard, J.H., Jewett, J.J., University of Oregon, Department of Planning, Public Policy and Management, and Sofaer, S., George Washington University Medical Center, 1996.

Content analysis of the comprehension comments found two major areas of misunderstanding. Coverage was a major element in the interpretation of the indicators. Consumers tended to equate coverage with quality. High coverage meant you got good care. (“Lumpectomy rates tell about the coverage people have. If their insurance covers lumpectomy, then this is what they will have;” “Quality of life tells about how good the patient's coverage is. High quality of life means that the patients have good coverage; they don't have to anguish about high medical care bills;” and “Five-year survival tells about how good the coverage is, they have coverage for all the needed services.”)

Coverage has been the primary concern under indemnity plans. Respondent comments suggest that this is still a central part of their concept of plans. Consumers haven't made the transition from concerns about coverage to understanding that in this new environment coverage is less of a concern than is access (and referrals) to the covered services. Thus, understanding the health care context was still a problem.

A further analysis looked at who had made these coverage comments. It made no difference, whether the respondent had an HMO or if they were in the “context” or “no context” group-each group was as likely to mention coverage in their interpretation of the indicators. We did not specifically discuss this issue in our context information.

The other problem area of comprehension was that consumers often did not see the linkages between the “black box of care” and the outcomes measured by the indicators. For example, respondents were unsure what kind of care might result in a better quality of life for breast cancer patients. The missing linkages often were assumed to be the insurance coverage.

Summary and Conclusions

The findings show that a general consumer population does view the condition-specific breast cancer performance measures as salient. However, other condition-specific indicators can not be assumed to be equally salient. Breast cancer is, perhaps, a special case in that many women feel vulnerable to this disease. Its prevalence also means that many consumers have a close tie to someone with breast cancer or are concerned about breast cancer.

Findings show that providing information on the health care context does make a difference in consumers' perceptions of the measures. Those receiving the context information felt that the indicators told them more about the different aspects of care than did those with no context information. The findings support the notion that education about the new health care context makes the QIs more meaningful. However, the experiment was small and focused on only one set of condition-specific indicators. More work is needed to test this approach with a wider range of indicators, both population-based and condition-specific, and with larger and more diverse study populations.

While the meanings that consumers ascribe the indicators are largely consistent with their intended meanings, the findings also show that some of the indicators are not well understood. Some of the comprehension problems are still related to confusion about the health care context. That is, more work on how to communicate about the health care context is needed.

Implications

The findings provide some support for the dissemination of condition-specific QIs. However, this may only be true for conditions where there are both high prevalence and a general perception of risk. Giving consumers a rationale about why this information may be relevant to them, even if they themselves do not have the condition, also generally increases the salience of condition-specific performance information.

The context information provided to the respondents in this study was designed to address the knowledge deficits observed in the previous study (Hibbard and Jewett, 1995; Jewett and Hibbard, 1996). That is, the materials were designed to address the pieces of information necessary to make the quality information understandable. While it appeared to make a difference in how the measures were understood, it was also insufficient in that the individual indicators were often misinterpreted. Consumers commonly interpreted the indicators as telling about how good one's coverage is.

After analyzing the data and listening to the focus group discussions, one conclusion is that consumers possess a mental map of the key issues. Consumers have an idea of what issues they need to be alert to and to watch for in making health care choices. However, this mental map appears to be for a FFS environment. The consumer issues under managed care and capitated systems of care are very different. A new mental map is needed as more consumers move into this environment-one that helps consumers see what the issues, the risks, and the benefits are under managed care. This mental map will help consumers make informed choices, navigate within a managed care system, and serve a consumer protection function. The context material provided in this study did not include a complete mental map.

Providing consumers with this new mental map for a managed care environment is no small task. Intermediaries, such as employee benefit managers, purchasing alliances, and consumer advocate groups, may be in the best position to communicate and educate on these issues.

Once consumers see what the issues are, the indicators (if linked with these issues), will be more salient. A next step might test the effect of integrating this expanded version of context information into the presentation of the data. That is, first lay out what the issues are for consumers under managed care (e.g., how the features of managed care can result in both good and poor care), then lead consumers to the issues that they are most concerned about. Providing quality information on these identified issues should help make the information both more salient and more comprehensible.

There are many areas of communicating about choice and quality that need further development. Health behavior researchers and practitioners have successfully used behavioral and social science theories for guiding research and shaping effective strategies. Social and behavioral science theories will likely also be useful in exploring how consumers learn and use information in decision making about health care. A greater reliance on theory and empirical work from related disciplines is needed to enrich research and enhance the effectiveness of consumer information interventions.

This investigation focused on an employed-age population. The findings may not be applicable to Medicaid or Medicare populations. Research focusing specifically on these populations and the use of condition-specific indicators is needed. Confusion about the new health care environment is likely greater among the Medicare population, who are less likely to have experienced managed care. Similarly, the Medicaid population is facing more choices and different kinds of choices as more States move to implement managed care programs for their Medicaid population. Future work needs to focus on ways to assist these vulnerable populations in decisions regarding both the nature of their available choices and how to assess what is best for their needs. While a wide range of formal and informal intermediary organizations now exist to assist Medicare and Medicaid enrollees, there is a need for research on how these intermediaries can be more effective in assisting consumer decision making.