Abstract
This study explores consumers' comprehension of quality indicators appearing in health care report cards. Content analyses of focus group transcripts show differences in understanding individual quality indicators and among three populations: privately insured; Medicaid; and uninsured. Several rounds of coding and analysis assess: the degree of comprehension; what important ideas are not understood; and what exactly is not understood about the indicator (inter-rater reliability exceeded 94 percent). Thus, this study is an educational diagnosis of the comprehension of currently disseminated quality indicators. Fifteen focus groups (5 per insurance type) were conducted with a total of 104 participants. Findings show that consumers with differing access to and experiences with care have different levels of comprehension. Indicators are not well understood and are interpreted in unintended ways. Implications and strategies for communicating and disseminating quality information are discussed.
Introduction
Dissemination of health plan report cards is expanding rapidly through the efforts of States; national organizations and agencies; private organizations; and health plans. In theory, consumers who receive these report cards make better choices based on the data provided. This informed choice of health plans then exerts market forces on the health care system to contain costs and ensure quality. The informed component of this theory, however, has not been verified. Simply providing comparative data to consumers may be insufficient to attain an informed status for most consumers. Consider the barriers to the comprehension of quality information: a number of multifaceted concepts and constructs underlie the understanding, interpretation, and application of Quality Care Indicators (QCIs); some QCIs may contain medical concepts which are not understood by consumers; the assumptions and beliefs held by policymakers and large purchasers in deriving these indicators may not be held by consumers; some consumers may be disadvantaged in their understanding of quality information due to a lack of experience or access to the system.
The purpose of this study is to assess what consumers understand about the QCIs currently being disseminated and to determine if there are differences in comprehension based on access to and experience with the health care system insurance status.
State-level report card efforts currently exist or are being developed in California, Florida, Minnesota, New York, Pennsylvania, Arizona, and Oregon. National-level organizations and agencies with existing or planned efforts include the National Committee on Quality Assurance, the Joint Commission on Accreditation of Health Care, and several States. In addition, numerous managed care organizations (MCOs), large purchasers, and coalitions are producing or disseminating report cards, including Kaiser Permanente of Northern California, CalPers of California, The Pacific Business Group on Health, The Massachusetts Healthcare Purchaser Group, Michigan PRO, U.S. Healthcare Corp. of Minneapolis, and the Dartmouth-Hitchcock health care system. Most report cards use data derived from the Health Plan Employer Data and Information Set (HEDIS), a system designed for use by large purchasers. HEDIS provides comparison data on cost and quality to help employers differentiate among health plans. HEDIS measures include plan performance indicators, including rates of preventive measures among plan members (e.g., childhood immunization, cervical screening, mammography, etc.), appropriateness of care measures, as well as patient satisfaction. HEDIS also includes indicators of the efficacy of chronic disease management in the plan (e.g., asthma and diabetes). Currently, HEDIS provides the only uniform formulas and criteria for the collection of quality information, making it the most feasible comparative data for dissemination to consumers. The question, however, is whether consumers understand these quality indicators, which were originally developed for use by large employers.
The logic behind these indicators incorporates fairly sophisticated understanding of managed care, the medical care process, and the many dimensions of medical practice. The structure, mechanisms, and incentives of managed care are not well understood by consumers. Even members of the plan, especially those from disadvantaged groups, may not understand their choices (Lyons, 1995). Rates of preventive measures (e.g., breast and cervical screenings) are used as quality indicators on most report cards. A number of studies, however, show that miscommunication, fear, and fatalism lead to confusion about the safety and efficacy of these procedures among the general and minority populations. (Kottke et al., 1995; Fulton et al., 1991; Cockburn et al., 1992). Consumer attitudes and perceptions may create barriers to the comprehension of quality information. Hibbard and Jewett (1996) examined consumer perceptions of HEDIS QCIs. Findings revealed widespread consumer attitudes that were counter to the intended interpretation and use of the QCI data. For example, one attitude common to almost one-half of all QCIs was that the indicator was not valid because factors outside the plan accounted for the findings. That is, the QCI data had more to do with patient choice, patient responsibility, or other factors not related to plan performance.
Some consumers may be disadvantaged in their understanding of quality information due to a lack of experience or access to the system. This includes the uninsured as well as minority groups who have encountered special cultural and economic barriers to care. The fact that approximately 42 million Americans are uninsured and another 29 million are underinsured has received wide coverage in the health policy literature. But questions of QCI comprehension and whether report card dissemination can “inform” consumer choice raise issues of another American tragedy. The most recent report on progress toward the National Educational Goals reported by the Department of Education (National Center for Education Statistics, 1993) found that 90 million of America's 191 million adults (47 percent) either are illiterate or can perform only simple literacy tasks. This trend appears to be worsening (e.g., the literacy of young adults has decreased significantly during the past 10 years). Both illiteracy and inadequate functional health literacy are recognized as barriers to understanding health information (Williams et al., 1995). Of the 44 million Americans who have the poorest literacy skills, a full 20 percent received their high school diplomas (Baker et al., 1994; National Center for Educational Statistics, 1993). Literacy levels will not change in the foreseeable future. The most recent assessments of the school-age population are bleak (National Assessment of Educational Progress, 1992). Perhaps in no other area are the interaction effects of demography and learning more apparent than in the area of literacy. These demographic characteristics are poverty, limited English proficiency, and minority status, the same as those limiting access to care and experience with the health care system. The changing demographics in the United States indicate large increases in these populations. For any information dissemination, attention must be paid to the diverse learning requirements of these low literacy groups (Baker et al., 1994).
Methods and Data
Conceptual Framework
If informed consumer choice is crucial to the integrity of the system, the onus of educating consumers must be addressed within the policy approach. Educational research has identified the characteristics or features of high quality educational efforts effective with diverse learners. These efforts feature the presentation of “big ideas” rather than a myriad of smaller facts (Carnine, 1994). “Big ideas” are the important themes and concepts key to understanding a content area and that facilitate the greatest amount of learning (Stein, Dixon, and Isaacson, 1994). The National Research Council of the National Committee on Science Education Standards and Assessment (1993) suggests that to provide a foundation for understanding and applying information, “[a] limited number of important concepts, principles…and theories” should be taught. Consumers must be able to connect ideas among and within the QCIs. Without such connections, consumers will view QCIs as too many isolated facts, rather than recognizing the principles relevant to the quality of their care and the relevance of their plan choice. The “big ideas” presented should be carefully selected to develop foundational knowledge, dispel myths, and to illuminate the deeper understandings of quality of care and how that relates to their own plan choice. By focusing on consumer comprehension and analyzing the content of their misunderstanding, we can identify the big ideas which could guide consumer research and serve as an organizational framework for thinking about, designing, and assessing consumer information efforts.
Study Design
The quality measures included in the study were selected from among those already being disseminated to consumers. Eighteen quality indicators that reflect a range of aspects of care are included. The indicators are shown in the Technical Note at the end of this article; they are grouped into the categories of: desirable events; undesirable events; patient ratings of satisfaction and quality; and disciplinary actions.
Research Questions
How well do consumers understand existing quality indicators?
What specifically is not understood?
Are there differences in comprehension based on insurance status?
Study Population
Three groups are included in the study population: a privately insured population; an uninsured population; and Oregon Health Plan members (Medicaid and expanded Medicaid). These three groups were included to assess whether access to care and experience with care would make a difference in the comprehension of quality information. To encourage open discussion, focus groups were constructed to maximize homogeneity. Only those with 14 years of education or less and who were between the ages of 35 and 55 were included in the groups. Participants were paid and were recruited via newspaper advertisements from the Eugene/Springfield Metropolitan area in Oregon. This non-representative sample may restrict the generalizability of the findings. The direction of any bias, however, may be to decrease rather than increase the potential for low comprehension (for example, by recruiting a more literate study population). Seventy-nine percent of the participants in this study have at least 1 year of college. Table 1 shows the characteristics of the study population by insurance status. Education and age do not vary greatly, as these variables were restricted in the selection process. Mean age for the total study population is 41.9 years. Fourteen percent of the study population is other than white. The only significant differences among the three insurance groups are on income, working status, and marital status. Oregon Health Plan members are more likely to have a lower income, are less likely to be married or cohabitating, and are less likely to be employed (working 30 or more hours per week). Health status and use of care are not significantly different among the three insurance groups.
Table 1. Demographic Variables, by Insurance Status.
| Demographic Variables | Privately Insured (N=36) |
Uninsured (N=36) |
Medicaid (N=32) |
|---|---|---|---|
| Percent Female | 77.1 | 52.8 | 52.8 |
| Mean Age | 41.7 (sd=6.1) |
42.3 (Sd=4.7) |
41.8 (sd=6.0) |
| Percent With Children Under 18 Living at Home | 60.0 | 38.9 | 48.6 |
| Percent High School Education or Less | 17.1 | 22.2 | 24.3 |
| Percent Other Than White | 17.1 | 19.4 | 5.4 |
| Percent With Household Income Under $17,500 | 22.9 | 72.2 | **94.6 |
| Percent Married or Cohabitating | 82.9 | 52.8 | **29.7 |
| Percent With at Least One Chronic Disease | 37.1 | 22.2 | 48.6 |
| Percent Who Have Been Hospitalized in Last Year | 11.4 | 11.1 | 8.1 |
| Mean Hospital Days (Among Those Who Have Been Hospitalized) | 5.5 Days | 2.3 Days | 4.8 Days |
| Percent Not Seeing a Doctor in the Last 2 Years | 17.1 | 17.1 | 5.4 |
| Percent Working for Pay | 62.9 | 41.7 | 37.8 |
| Percent Who Work More Than 30 Hours a Week (of Those Working) | 42.9 | 25.0 | *8.1 |
| Percent Who Have One or More Primary Physicians | 65.7 | 34.3 | 83.8 |
| Percent Who Rate Health Status Fair/Poor | 11.4 | 11.1 | 29.7 |
p <.01.
p<.001.
NOTES: Privately insured consumers are the reference group. Medicaid and uninsured consumers are compared with the privately insured. SD is standard deviation.
SOURCE: Jewett, J.J., and Hibbard, J.H., University of Oregon, Department of Planning, Public Policy and Management, 1995.
Focus Groups
One hundred four individuals participated in 15 focus groups, five groups for each type of insurance. All groups lasted approximately 2 hours and were conducted by the same moderator, one of the study investigators. The focus group guide was developed and refined via five pilot groups conducted prior to the study. To control moderator bias, all explanations, instructions, questions, prompts, etc., were verbatim from scripts in the guide. All questions from participants were directed to other group members or delayed until the conclusion of the session. The session format is shown in Figure 1. First, report card efforts, quality-of-care measures, and managed care systems were explained to consumers. Then they were shown sample report cards and definitions of the indicators and managed care plans. Consumers were instructed to read each QCI definition during a 6-8 minute pause. Next, in a group interview, each consumer was asked to respond, individually, to a set of questions. Each consumer selected the QCIs “best” from a list of the 18 presented in the Technical Note at the end of this article. They then stated the reason for their selection (e.g., the perceived strength of the QCI). A group discussion of QCI “weaknesses” followed in which consumers discussed why QCIs did not tell them about quality in a health plan. All 18 indicators were addressed (although not all were perceived to have weaknesses by every group).
Figure 1. Chart of Group Session.
Focus group sessions were audio and video recorded, transcribed, and coded. Codes reflect consumer perceptions and comprehension of the QCIs. Comments were coded by the perceived strength or weakness and comprehension of each indicator. The codes used in this analysis focus on comprehension: misinformation, myths, questions, unfamiliarity, and confusion over interpretations. Specific code definitions are presented with the relevant analyses in this article. A 30 percent random sample of transcript data was evaluated for inter-rater reliability for all coding. Reliability assessments exceeded 99 percent agreement for all topic codes; 98 percent agreement for all strength and weakness codes; and 94 percent for major comprehension codes. The transcripts and codes were entered into The Ethnograph software program for content analysis. The Ethnograph segments and summarizes the data by code categories. The program can sort the code categories and provide frequency distributions for the total population or subsets of the population. A “comment” is used as the unit of analysis for the data. The Ethnograph allows analysis by comment or by the number of lines associated with the topic. Line counts, however, tend to bias data toward the more loquacious participants. Data were also screened so that no code category came primarily from one individual or group. Because of the group processes employed (e.g., group interview), all consumers contributed comments (100 percent participation).
Findings
What Indicators Are Not Understood?
The transcript of each focus group is coded for low comprehension of the QCIs. A total of 1,723 comments regarding the QCIs were recorded. Three hundred thirty-three of these comments reflected low comprehension of the individual indicators (19 percent). However, when low-comprehension comments regarding whole groups of indicators were included (e.g., all preventive measures), 24 percent of all comments reflected low-comprehension.
The two major categories of low comprehension are “misinformation” and acknowledged “lack of information.” Misinformation includes myths and statements about the QCI which are clearly incorrect or distorted. Lack of information is coded when the consumer acknowledges that he or she does not understand the QCI. Lack of information includes: questions about the QCI (“What kind of infections are hospital-acquired infections? Do colds count in this measure?”); statements of confusion (“What does low-birth-weight babies have to do with plan quality?”) or lack of knowledge about the QCI (“I have no clue what this means;” “I need an expert to explain this to me”); and statements expressing difficulty interpreting the QCI (“I don't know whether high c-section rates or low c-section rates show good quality;” “I don't know how to judge this because…”) The Misinformation and Lack of Information code categories are summarized to form an index of low comprehension. Low comprehension is defined by its percentage of all comments (by all participants) made about an indicator during group interviews or discussions. Those QCIs with 20 percent or more of all comments reflecting low comprehension are defined as poorly understood.
Overall, the Undesirable Event QCIs have the lowest comprehension, with rates of pediatric asthma hospitalization and rates of low-birth-weight babies the least understood. Desirable Event indicators also are not well understood. The QCIs in the category Patient Ratings of Care are best understood. Although questions and prompts were identical for each QCI, some QCIs generated little discussion. For these indicators, there was a reduced opportunity to assess comprehension. These differences in the number of comments about each indicator can be seen in the cells in Tables 2 through 4.
Table 2. Does Indicator Comprehension Differ by Insurance Status?1.
| Indicators | Total Percent Low Comprehension | Privately Insured2 (N=36) |
Medicaid3 (N=32) |
Uninsured4 (N=36) |
|---|---|---|---|---|
|
| ||||
| Percent | ||||
| Desirable Event Rates | ||||
| Mammograms | 26.7 (31/116) |
18.9 (7/37) |
*29.8 (17/57) |
**31.8 (7/22) |
| Immunizations for Children | 28.9 (33/114) |
31.3 (15/48) |
**16.7 (5/30) |
36.1 (13/36) |
| Cholesterol Screening | 27.8 (22/79) |
26.7 (8/30) |
23.1 (6/26) |
34.8 (8/23) |
| Diabetic Eye Exams | 13.0 (6/46) |
9.1 (2/22) |
13.3 (2/15) |
***22.2 (2/9) |
| Pap Smears | 15.2 (12/79) |
18.6 (8/43) |
14.3 (3/21) |
**6.7 (1/15) |
| Undesirable Event Rates | ||||
| Hospital-Acquired Infection | 26.0 (27/104) |
25.6 (10/39) |
21.9 (7/32) |
30.3 (10/33) |
| Post-Surgery Complication | 8.1 (7/86) |
10.0 (3/30) |
8.3 (2/24) |
6.3 (2/32) |
| Hospital Death After a Heart Attack | 27.3 (24/88) |
25.0 (10/40) |
22.2 (4/18) |
33.3 (10/30) |
| C-Section Birth | 17.9 (20/112) |
20.8 (10/48) |
17.4 (4/23) |
14.6 (6/41) |
| Pediatric Asthma Hospitalization | 40.8 (31/76) |
25.6 (11/43) |
***63.6 (14/22) |
***54.5 (6/11) |
| Low-Birth-Weight Babies | 44.0 (48/109) |
44.1 (15/34) |
42.9 (18/42) |
45.5 (15/33) |
| Patient Ratings of Care | ||||
| Overall Quality in the Plan | 16.5 (13/79) |
11.4 (4/35) |
***28.0 (7/25) |
10.5 (2/19) |
| Time Spent With Doctor | 11.6 (10/186) |
12.5 (4/32) |
12.5 (4/32) |
9.1 (2/22) |
| Respect Given Patients | 0.01 (1/100) |
0 (0/30) |
2.6 (1/39) |
0 (0/31) |
| Doctor's Communication | 2.7 (3/112) |
2.7 (1/37) |
4.2 (2/48) |
0 (0/27) |
| Disenrollment | 13.6 (12/88) |
20.7 (6/29) |
21.4 (6/28) |
0 (0/31) |
| Disciplinary Actions | ||||
| Malpractice Judgments | 12.5 (17/136) |
19.0 (11/58) |
**7.1 (3/42) |
*8.3 (3/36) |
| Professional Disciplinary Actions | 14.2 (16/113) |
20.0 (7/35) |
16.7 (9/54) |
0 (0/24) |
Percentage of each insurance groups' comments that reflect low comprehension of the Quality Care Indicator (QCI).
132 low-comprehension comments; 670 total comments.
114 low-comprehension comments; 578 total comments.
87 low-comprehension comments; 475 total comments.
p<.05.
p<.01.
p<.001.
NOTES: Bolding indicates which cells have 20 percent or more low-comprehension comments. Privately insured consumers are the reference group for the Chi-square analysis. Medicaid and uninsured consumers are compared with the privately insured. Parentheses show number of low-comprehension comments over total comments made about that indicator.
SOURCE: Jewett, J.J., and Hibbard, J.H., University of Oregon, Department of Planning, Public Policy and Management, 1995.
Table 4. Consumer Beliefs and Their Understanding of Aggregate and Quantitative Concepts.
| Measure | Total1 | Privately Insured2 | Medicaid3 | Uninsured4 |
|---|---|---|---|---|
|
| ||||
| Percent | ||||
| Aggregate or Quantitative Concepts (Round 2) | ||||
| Total | 43 (142/333) |
32.9 | **49.9 | *45.2 |
| Doesn't Understand Rates | 11 (37/333) |
5.1 | ***15.2 | ***13.0 |
| Doesn't Understand the Nature of Comparison or Aggregate Data | 15 (49/333) |
8.8 | ***19.0 | **17.8 |
| Doesn't Understand the Population-Based Definition of Quality of Care | 17 (56/333) |
19.0 | 15.7 | 14.4 |
| Plan-Related Concepts (Round 3) | ||||
| Total | 57 (189/333) |
60.6 | 60.7 | 55.6 |
| Doesn't Believe That Differences in Quality Exist Among Plans or Providers in the Plan | 7 (23/333) |
6.6 | 7.7 | 6.7 |
| Doesn't See What Aspects of Care in the Plan the QCI Tells about | 21 (70/333) |
24.1 | 23.9 | 21.1 |
| Belief That the Plan Has No Role in the QCIs' Performance | 29 (96/333) |
29.9 | 29.1 | 27.8 |
| Deterministic Beliefs (Round 3) | ||||
| Total | 26 (86/333) |
23.4 | 31.6 | 28.9 |
| Believes the Problem Related to the QCI Is Just Not Reducible | 11 (37/333) |
6.6 | ***17.9 | ***15.6 |
| Doesn't See the Value of Preventive Health Measures or Public Health | 15 (49/333) |
16.8 | 13.7 | 13.3 |
333 low-comprehension comments.
132 low-comprehension comments.
114 low-comprehension comments.
87 low-comprehension comments.
p<.05.
p<.01.
p<.001.
NOTES: Privately insured consumers are the reference group for the chi-square analysis. Medicaid and uninsured consumers are compared to the privately insured. Each coding round has a separate set of mutually exclusive codes (with the potential to sum to 100 percent for that round).
SOURCE: Jewett, J.J., and Hibbard, J.H., University of Oregon, Department of Planning, Public Policy and Management, 1995.
Does Comprehension of QCIs Differ by Insurance Status?
Low comprehension is evident in all three insurance status groups. Bolded items on Table 2 indicate those QCIs with 20 percent or more comments coded as low comprehension. For example, for the Medicaid consumers, 64 percent of all comments about the Asthma QCI show low comprehension. The Undesirable Event QCIs are poorly understood in all three insurance groups. Post Surgery Com-plication Rate is the only Undesirable Event QCI which is understood. For privately insured and Medicaid consumers, Desirable Event QCIs are better understood than Undesirable Event QCIs. However, Desirable Event QCIs are less understood by the uninsured. Although not shown on Table 2, these Desirable Event QCIs are the only indicators for which significant differences were found (on four of the five QCIs) between the Medicaid and uninsured consumers. Perhaps the uninsured's lack of access to preventive care negatively affects their understanding of these Desirable Event QCIs. Undesirable Event QCIs appear to be equally misunderstood by all consumers. Table 2 shows other significant differences in comparing privately insured consumers with the other two groups that have less experience or access to the medical care system.
What Are the Characteristics of Low Comprehension?
Is low comprehension characterized by misinformation or acknowledged lack of information? Are there differences based on insurance type?
Across all three insurance types, one-half of all low comprehension is misinformation and one-half is acknowledged lack of information (Table 3). Misinformation is defined by myth, incorrect statements, or mistaken beliefs. Acknowledged lack of information includes questions about the QCIs, consumer statements of their lack of knowledge, requests for clarification, and questions about the interpretation of the data. Comparing the different insurance types, however, a different picture emerges. The privately insureds' low-comprehension comments are comprised of 37 percent misinformation and 63 percent acknowledged lack of information. This is reversed for the Medicaid and uninsured consumers. The privately insured are much more likely to ask questions about the QCIs, declare their lack of knowledge (e.g., acknowledging unfamiliarity with terms, medical conditions, the QCIs, etc.) and to discuss problems with interpreting the QCIs. This awareness of their own information deficits and increased information-seeking through questions about the QCIs may give them an advantage in being informed consumers. The privately insureds' lower percentage of misinformation or myth may be due to greater access to the medical care system. This may also be due to higher self-esteem or higher comfort levels resulting from their higher levels of socioeconomic status. Although the privately insured have 45 more low-comprehension comments than the uninsured, all are accounted for by acknowledged lack of information. The privately insured sought information or acknowledged a deficit 2.2 times more often than the uninsured and 1.8 times more often than the Medicaid group.
Table 3. Low Comprehension Comments, by Insurance Status.
| Type of Low-Comprehension Comments | Total | Privately Insured | Medicaid | Uninsured |
|---|---|---|---|---|
|
| ||||
| Percent | ||||
| All Indicators | ||||
| Misinformation | 50 (167/333) |
37 (49/132) |
***60 (68/114) |
**57 (50/87) |
| Lack of Information | 50 (166/333) |
63 (83/132) |
**40 (46/114) |
*43 (37/87) |
| Desirable Events | ||||
| Misinformation | 35 (36/104) |
25 (10/40) |
***42 (14/33) |
**39 (12/31) |
| Lack of Information | 65 (68/104) |
75 (30/40) |
*58 (19/33) |
61 (19/31) |
| Undesirable Events | ||||
| Misinformation | 69 (109/157) |
56 (33/59) |
***86 (42/49) |
69 (34/49) |
| Lack of Information | 31 (48/157) |
44 (26/59) |
***14 (7/49) |
**31 (15/49) |
| Patient Ratings | ||||
| Misinformation | 33 (13/39) |
7 (1/15) |
***45 (9/20) |
***75 (3/4) |
| Lack of Information | 67 (26/39) |
93 (14/15) |
***55 (11/20) |
***25 (1/4) |
| Disciplinary Actions | ||||
| Misinformation | 33 (11/33) |
28 (5/18) |
25 (3/12) |
***100 (3/3) |
| Lack of Information | 67 (22/33) |
72 (13/18) |
75 (9/12) |
0 (0/3) |
p<.05
p<.01.
p<.001.
NOTES: Privately insured consumers are the reference group for the Chi-square analysis. Medicaid and uninsured consumers are compared with the privately insured. Parentheses indicate the number of comments coded as misinformation or lack of information over the number of low-comprehension comments for that indicator.
SOURCE: Jewett, J.J., and Hibbard, J.H., University of Oregon, Department of Planning, Public Policy and Management, 1995.
Looking at total percentages by QCI category, the distributions for Desirable Events, Patient Ratings, and Disciplinary Actions are remarkably similar: One third Misinformation and two-thirds Lack of Information. For the Undesirable Event QCIs, however, the distribution is reversed, with two-thirds reflecting misinformation and myth. This is not surprising, since these QCIs are the least understood. In comparing insurance types, the question is whether less experience or access to the system affects comprehension. That is, do the Medicaid and the uninsured consumers differ from the privately insured? Significant Chi-square values are shown by asterisks on Table 3. From an educational perspective, the privately insured may have an advantage over others in their potential to be informed consumers. They may be more amenable to educational intervention by virtue of their greater questioning, greater acknowledgment of deficits, and by their proportionately smaller degree of misinformation. For other groups, additional efforts may be required to dispel myths. Increased motivation may be necessary before those consumers seek information.
Closer Look at Low Comprehension
What specifically is not understood? What “big ideas” can be identified?
Low-comprehension comments are coded to determine the specific content and nature of misinformation, myths, and questions of consumers. A series of questions are posed to determine comprehension of underlying concepts and to identify “big ideas” which can be used in educational efforts. Are deficits in foundational knowledge (e.g., basic definitions) a major barrier to understanding QCIs? Do consumers understand underlying quantitative concepts? What beliefs present barriers to QCI comprehension?
A separate round of coding is required to adequately answer each question. This is necessary because, for both theoretical and empirical reasons, the unit of analysis in this study is the individual consumer comment. Individual comments, however, could be long and contain a number of thoughts and ideas. Splitting comments into code phrases could bias the denominators for an analysis or jeopardize the integrity and rationality of the data. Each comment, therefore, is reviewed and coded for each of the above questions regarding the nature of low comprehension. Thus, each question represents a separate round of coding (each with a separate set of mutually exclusive codes with the potential to sum to 100 percent for that question).
In the following sections, questions regarding the foundational knowledge refer to round 1 coding of low-comprehension comments; questions regarding aggregate or quantitive conceps refer to round 2 coding; and the regarding concepts and beliefs questions refer to round 3 coding.
Is Low Comprehension Associated With Deficits in Foundational Knowledge?
Certain foundational knowledge is prerequisite to understanding how an indicator tells about quality of care. For example, for the indicator “Rate of mammograms in women over 50 in the plan,” misunderstanding of breast cancer or the mammogram procedure can lead to confusion over the value of the QCI or what the QCI tells about quality. To what degree is the primary cause of low QCI comprehension associated with deficits in foundational knowledge? By definition, all comments are within the context of why the QCI is or is not a good indicator of quality. Any confusion that is independent of the QCI (for example, unrelated confusion about breast cancer, in general) is not coded.
The nature or meaning of the underlying medical condition associated with the QCI accounts for 21 percent of all low-comprehension comments. This includes comments such as: “Rates of mammograms in women don't show quality because over 40 percent of all breast cancer is in men and you miss that in your statistics;” “I don't really know what asthma is so how can I use this?;” “Low- birth-weight babies include any baby under 7 lbs, so big deal.”
The nature, parameters, or meaning of the test or procedure associated with the QCI is not understood in 8 percent of the low-comprehension comments. This includes questions or comments such as: “Rate of cholesterol screening is a bad indicator of quality because they can't decide on fasting or non-fasting tests, so the data is no good;” “How can a pap smear prevent cancer? I mean, what's the stuff they use in the smear to prevent it?;” “I have to tell you that I don't even know what a mammogram is.”
In 20 percent of low QCI comprehension, consumers interpret QCI performance opposite to its intended meaning or do not understand whether high or low values indicate good performance. For example,“High asthma hospitalizations tell me they're doing a good job and get kids in at the first sign of a problem;” “High c-section rates show quality because you don't want the baby in any distress;” “High death rates show sensitivity by allowing people to go in peace rather than use heroic means;” “A high rate of mammograms tells me they're missing cancer because it means they don't take the time to read each one carefully.” In the Medicaid population, high disenrollment also had an alternative “positive” interpretation: “High disenrollment is good because it shows more people can afford their own insurance.”
Most of the remaining 51 percent of comments are either direct questions about the QCI or misinformation about the general definition of the QCI such as “patient ratings are how the doctor scores the patient's condition for his charts;” “It's not the hospital's fault that patients go home, do a sloppy job of dressing their wounds, and get infections;” or “Would someone please tell me what the heck this means?”
Differences in Foundational Knowledge?
No differences are found in the comprehension of foundational knowledge among the privately insured, uninsured, and Medicaid consumers. Results for all three groups are consistent with the above aggregated data.
Do Consumers Understand Aggregate or Quantitative Concepts?
Regardless of the form in which comparative data are presented, quality information provided to consumers is inherently quantitative. Overall, 43 percent of all low-comprehension comments reflect some deficit in understanding aggregate or quantitative concepts (Table 4). These concepts include:
Rates (11 percent of all low comprehension): Many consumers openly admitted, “I don't really understand rates.” Others confused rates with similar terms (ratings) or other definitions (fees). For example, mammogram rates were confused with tumor growth rate; the rate of the spread of breast cancer; a clinical measure of disease progression (doctor's rating), and how often one should have a mammogram.
Aggregate or Comparison Data (15 percent of low comprehension): This includes failure to conceptualize that: data can be summarized across a plan; large data differences between plans can tell you something about those plans; rare single case occurrences can not account for high rates in aggregated data. (This category does not include skepticism about the data, but, instead, reflects not understanding the concept of the data).
Population-Based Approach (17 percent of low comprehension): QCIs are defined and compiled using a population-based approach across patients. However, judging your own care by the quality of care for the whole population in your plan is not intuitive and may be counter to consumers more personal, face-value approach. This is compounded by the fact that consumer's have difficulty understanding the probabilities. “How can you say ‘diabetic eye exams’ tell me about quality when I don't ev have diabetes?;” “PAP smears are for women, why should I care?;” “The number of mammograms means nothing. What matters is how well they read the mammogram;” “Rates don't tell me anything about quality, quality of care is what happens in the doctor's office between the doctor and the patient.”
Differences on Understanding of Quantitve and Aggregate Concepts?
The privately insured have a significant advantage in their understanding of aggregate or quantitative concepts (Table 4). Overall, these concepts are reflected in 33 percent of the privately insureds' low comprehension, compared with 45 percent for the uninsured, and 50 percent for Medicaid. Medicaid and uninsured consumers are approximately three times more likely to make low-comprehension comments about rates and more than twice as likely to make them about comparison or aggregate data.
What Concepts and Beliefs Are Reflected in Low Comprehension?
The “face-value” content of consumer comments revealed that a number of concepts and beliefs important in decision-making are not understood by consumers. These fall into the major categories of plan-related concepts and deterministic beliefs.
Plan-Related Concepts: Concepts relating to the nature, structure, or mechanisms of the health care plan are confusing to consumers. Fifty-seven percent of low-comprehension comments are in regard to plan-related concepts (Table 4). In 7 percent of low-comprehension comments, the consumer does not believe that differences in quality exist among health plans or among doctors or hospitals in the plan. For example, “Plans are just for coverage and bill paying…the only differences are in the coverage you pay for;” “Hospitals are just facilities, they are all the same by law.” In 21 percent of low comprehension, the consumer does not see what aspect of care in the plan that the QCI tells about. “Asthma hospitalizations show the emergency room is doing a good job;” “Rate of diabetic eye exams is important to me because I spend a fortune on glasses for my family and I need a plan that covers eye care;” “I just don't see what this has to do with anything. Is this a trick one that you threw in?” In 29 percent of low comprehension, consumers believe that the managed care plan has no role in the QCIs performance. They do not understand that the plan has influence or responsibility for quality of care. “It makes no sense. What does the plan have to do with people dying in the hospital after a heart attack?;” “It's the woman who decides to get pap smears, it shouldn't reflect on the plan in any way;” “The plan has no influence on how much time a doctor spends with you;” “It's not the plan's fault that diabetics don't go get eye exams.”
Deterministic Beliefs: In 26 percent of all low QCI comprehension, consumers believe that the events measured by the QCI are either uncontrollable or inevitable. Eleven percent of consumer comments reflect a belief that the adverse event or the underlying problem associated with the QCI is just not reducible or preventable. For example, “When it's your time to die there's nothing anyone can do. Your number is up;” “Hospitals are full of germs, there is no way to avoid infections;” “Some babies are just small, it's just genetics.” In 15 percent of low-comprehension, consumers do not see the value of public health or preventive measures (e.g., screening or immunization programs). Consumers either fail to see big picture results or do not believe prevention is efficacious. For example, “Screening programs are a scam to make money;” “Too many children die from the shot who wouldn't die of the disease itself;” “Immunizations should be left entirely up to the parents, without the doctor or the plan or the schools telling them what to do.” Also included in this definition is not believing that the test/procedure is an effective preventive measure, e.g., “not only are mammograms not effective in reducing cancer, they cause cancer.”
Differences in Concepts Not Understood?
Fewer of the privately insured (7 percent) believe that the problem related to the QCI is just not reducible, compared with 18 percent for Medicaid consumers and 16 percent for the uninsured (Table 4).
Comprehension of Individual QCIs
Eight QCIs with the lowest comprehension are selected for analyses at the individual indicator level. These QCIs would be the obvious targets for educational efforts and may show differences from aggregated data (which include a number of better understood indicators). For these eight indicators, all previous coding analyses are duplicated except the comprehension of “aggregate and quantitative concepts” (which is not associated with any one QCI but distributed across all QCIs).
Differences in Types of Misunderstanding?
Are there differences in “misinformation” versus acknowledged “lack of information” for the eight target QCIs with the lowest comprehension?
Data for individual target QCIs are consistent with the aggregated data of Table 3. Lack of information predominates each Desirable Event QCI (ranging from 59 percent for the Cholesterol Screening QCI to 77 percent for the Immunization QCI). In the Undesirable Event category, misinformation predominates (with the highest rate being 79 percent for Rates of Low-Birth-Weight Babies). C-section Birth Rates (50 percent misinformation and 50 percent lack of information) is the only exception.
Misunderstanding of QCIs With the Lowest Comprehension
These analyses focus on the key areas of Foundational Knowledge (round 1 coding of low comprehension) and Consumer Beliefs (round 3 coding). Only cells with 20 percent or more low comprehension are included in Table 5. This provides an overview of the areas of QCI comprehension to address more fully with consumers.
Table 5. Comprehension of Individual QCIs Which Are the Least Understood by Consumers (Percent of Low–Comprehension Comments for Each Category).
| Undesirable Events
|
||||||||
|---|---|---|---|---|---|---|---|---|
| Measure of Comprehension | Desirable Events
|
Hospital-Acquired Infections | Cardiac Deaths | C-Sections | Asthma Hospitalization | Low Birth Weight | ||
| Mammograms | Immunizations | Cholesterol | ||||||
|
| ||||||||
| Percent | ||||||||
| Foundational Knowledge (Round 1) | ||||||||
| Doesn't Understand the Nature of the Underlying Medical Condition | 23 | — | — | 54 | 30 | — | 20 | 26 |
| Doesn't Understand the Meaning of the Medical Test or Procedure Associated With the QCI | 30 | — | 46 | — | — | — | — | — |
| Doesn't Understand Whether High or Low Rates of the QCI Indicate High Quality Performance | — | — | — | — | — | 37 | 20 | 30 |
| Plan-Related Concepts (Round 3) | ||||||||
| Doesn't Believe That Differences in Quality Exist Among Plans or Providers in the Plan | — | 48 | 20 | — | — | — | — | — |
| Doesn't See What Aspect of Care in the Plan the QCI Tells About | — | — | 33 | 23 | — | — | 20 | 24 |
| Belief That the Plan has No Role in the QCI's Performance | 20 | 42 | — | 39 | 50 | 26 | 53 | 64 |
| Deterministic Beliefs (Round 3) | ||||||||
| Believes the Problem Related to the QCI is Just Not Reducible | — | — | — | 31 | 46 | — | 20 | — |
| Doesn't See the Value of Preventive Health Measures or Public Health | 55 | — | 38 | — | — | — | — | — |
NOTES: Within coding rounds, codes are mutually exclusive and may sum to 100 percent. Only cells with 20 percent or more low-comprehension comments are included. QCI is quality care indicator.
SOURCE: Jewett, J.J., and Hibbard, J.H., University of Oregon, Department of Planning, Public Policy and Management, 1995.
Foundational knowledge, especially low comprehension of the medical condition, becomes a much more important source of confusion. The belief that the plan has no role in the QCI's performance is almost unanimous for these poorly understood indicators, with percentages much higher than for the overall data with all QCIs included (e.g., 64 percent of low-comprehension comments for rates of low-birth-weight babies). In addition, the specific aspect of care (e.g., hospital care, outpatient care, and preventive care) measured by hospital infection rates, rates of low-birth-weight babies; rates of pediatric asthma hospitalizations, and cholesterol screening rates must be explicitly stated for consumers. An important prerequisite to assessing quality must be an underlying belief that some action can be taken to improve it. Determinism, however, is an important barrier to the use of quality indicators for breast cancer, coronary artery disease, hospital-acquired infections, heart attacks, and asthma (e.g., “There's really nothing that anyone can do about it. It's just in the genes”).
Comparing individual indicators shows interesting variations in what is not understood, even within Desirable Event and Undesirable Event categories. For example, comparisons among “rate of mammograms among female plan members over age 50”; “rate of immunizations among children in the health plan”; and “rates of cholesterol screening.” Low comprehension is more dispersed for the mammogram QCI, perhaps because breast cancer and mammograms are not well understood. Publicized controversies surrounding mammograms have taken their toll, confusing consumers about the importance and efficacy of the procedure. (“Rate of mammograms in women over age 50 can't be effective at early detection because age 50 is already too late.”) Many consumers don't “believe in” cancer screening (for either populations or for individuals), asserting that mammograms are not effective in reducing the number of cancer deaths and, therefore, tell nothing about quality or choosing a plan.
Discussion
Consumers Have a Different “Big Picture” View
Report card efforts were conceived by policymakers operating from the assumptions of quality of care measurement; managed care, and market-driven economics. Those policymakers then looked for specific measurable events to serve as sensitive indicators of plan performance, in accordance with those theories and assumptions. It is difficult, however, for consumers to reconstruct the big picture working backward from these narrow indicators. Without knowledge of these larger assumptions, consumers are left with face-value interpretations for some QCIs. In addition, those larger concepts and theories which are held by consumers are much different than those of policymakers. The consumers' big picture appears to be that of the fee-for-service (FFS) environment: plans are just payers; the doctor- patient relationship is one of paternal advocacy; any ulterior motives are for overtreatment; quality of care by providers is highly regulated and standardized. These different assumptions are reflected in consumer comments across all low-comprehension categories.
The majority of comments show a failure to see any connection between the plan and quality of care (“What on earth can the plan do about it;” “It's not the plan's fault…it happened in the hospital”). Some consumers, however, feel their care is “none of the plan's business,” for example, due to concerns over coverage denial. (“Any plan that wants you to get a cholesterol test can't be trusted. It's because they want to drop you;” “The plan has no right to meddle.”) The doctor-patient relationship view that quality “only takes place in the doctor's office between a doctor and patient” is common. The shift from the independent doctor-patient relationship of FFS to the doctor-plan-patient relationship of managed care did not occur for most consumers. The gatekeeper role and the reversal of monetary incentives for physicians is unseen.
When discussing physicians and hospitals, the ulterior motives cited refer only to overtreatment, e.g., that providers perform many unnecessary procedures for monetary gain or for convenience. Consumers perceive health care as highly regulated by the government or other organizations, for example, believing that hospitals are all the same by law and that standards of quality are monitored continuously. This belief in regulation and monitoring also extends to physicians. Further, population-based definitions of quality of care are difficult to understand and the implications of managed care “group membership” and responsibility for all members are not often seen (“I'm not a woman, so pap smears and mammograms don't tell me anything about quality”). Quality of care is seen at a personal level (“Quality is how good your doctor is…not any of this other stuff;” “Numbers of mammograms tell me nothing…it's how well they read them”).
Consumers often have a face-value view of QCIs and do not understand the meanings that policymakers assign to them. For example, the c-section QCI is “derived” for large purchasers, who are concerned with cost containment. Consumers, however, are concerned with trauma to their infants and some see higher rates as greater concern for infant safety. Others see c-sections through the ulterior motives of making money or convenience for providers. The asthma hospitalization QCI and the low-birth-weight QCI were created by policy-makers as potentially sensitive indicators of ambulatory care. Consumers, however, are likely to make face-value interpretations. Some see the asthma QCI as a measure of access to hospital care or of sensitivity and caring for asthma patients. Some consumers interpret the low-birth-weight QCI as a measure of proficiency in saving tiny babies or as a case-mix variable (“This plan has a lot more drug addicts and is probably in the inner city”). Others openly state that they don't know if the rates of a given QCI should be high or low. These reversed interpretations underscore the difficulty consumers have with QCIs.
What Are the Big Ideas for Educational Efforts?
The findings indicate real barriers to the use of quality information and informed choice for some consumers. Consumers need to understand: what a managed care plan is; how the plan can influence quality; what aspects of care the QCIs are designed to measure and how to interpret each indicator; and that informed consumer choice can influence the quality of care. The big picture, as seen by policymakers, is invisible to consumers and must be explicitly drawn. Report cards must give consumers the context necessary to make QCIs meaningful and ensure their appropriate use. Some of the important big ideas to include in educational efforts are:
New Concept of a Plan–Including the ideas of: plan responsibility for the care of its members; wellness and health maintenance (contrasted with the sick-care model); and rationales for population-based definitions of quality.
Mechanisms of How Plans Influence Quality–Including the mechanisms that: link providers to plans; allow plans to monitor physician and hospital practices; allow plans to influence physician, hospital, and patient behavior.
Implications of the Doctor-Plan-Patient Relationship–Including the dynamics of the new relationship, compared with the doctor-patient relationship of the FFS system; potential conflicts of interest; different provider motives (e.g., the new concern is undertreatment, not overtreatment).
Consumeristic Attitude–Including the messages: Your choice of a plan can make a difference in your health care; quality differences do exist among plans and hospitals (and regulation can not be relied upon to alter quality differences); medical problems (such as asthma, heart attack deaths, and hospital infections) are controllable or reducible; preventive measures are effective and are part of good quality care; you must take an active role in evaluating quality of care.
What Components or Aspects of Care to Evaluate in Choosing a Plan–Including the components of the system, (ambulatory care, hospital care, specialist care) and other aspects, such as preventive, chronic, emergency, respect and caring, monitoring and followup, patient satisfaction, etc.
What each QCI Indicates and How to Interpret It–Explicitly link each QCI to what it measures and how it indicates the plan is doing a good job. To prevent confusion regarding good and bad performance, the values for each QCI should be stated directly (e.g., “low rates of pediatric asthma hospitalizations are good because….”) Provide benchmarks from national data sets or from clinical guidelines. Poorly understood diseases and procedures will need additional statements to dispel specific myths and lack of knowledge. (e.g., “Some people mistakenly believe that hospital infections are unavoidable…;” “ Some people believe that nothing can be done after you have a heart attack, but medical advances have changed that.”) These QCIs do tell about quality.
Other Considerations
All low-comprehension comments by consumers in this study occurred after hearing verbal explanations of managed care and reading definitions for the QCIs (during an observed 6-8 minute pause). This illustrates what educators have long known: Just because material is presented doesn't mean it is understood. Educators use the term “curriculum disability” (Carnine, 1994) to replace outdated student- blaming approaches. Indeed, literacy in this study population probably exceeds that of the general public. Seventy-nine percent of study participants had at least 1 year of college and participants were recruited via newspaper advertisements. In addition, all Medicaid enrollees in Oregon shifted to managed care prior to this study. Thus, they have some exposure to plan concepts and mechanisms. Low-comprehension codes used in this study are conservative. Many attitudes posing significant barriers to QCI use were not coded as “misinformation” but as “consumer perceptions” with some degree of accuracy. Thus, although findings from this nonrepresentative sample may be restricted in their generalizability, the low comprehension of QCIs is not overestimated. The nature and complexity of quality of care information warrants special attention to its presentation. This is particularly important when materials are mailed and may be skimmed or set aside by consumers.
Strategies
Strategies to consider include:
Avoid Quantitative and Aggregate Concepts: Present quality information in ways that do not rely on rates, understanding probabilities, or require consumers to make projections from the data.
Use Simplified Approaches: Translate jargon (prevention, capitation, managed care) into simple language. Software is available to screen for vocabulary, for example, above a sixth grade reading level, and to offer alternative words for replacement. Anything over two pages may not be read, suggesting “bullet” rather than textual approaches for important ideas. “Layer” or append any detailed information.
Video presentation may be required for a significant number of consumers. This is especially true for consumers with no previous exposure to the managed care system.
Person-mediated approaches may also be required. These range from telephone hotlines to the use of “personal consumer advocates” to interpret and explain information for consumers.
As the focus of health care reform shifts to informing consumers, policymakers must focus on the special needs of different populations. Informational needs will vary with: experience with the medical care system, literacy, disavantaged populations, and language barriers. A single type of approach for individuals with different backgrounds and prerequisite knowledge may be inadequate to achieve informational goals. Many consumers in this study voiced feelings of inadequacy or feeling overwhelmed by the task of interpreting QCIs and evaluating plans. Others lamented, “I just want some expert to tell me which one to pick.” Additional learning opportunities and motivational strategies may be necessary to involve these consumers.
Summary
This study serves as an educational diagnosis of consumer comprehension of QCIs and the differences among consumers with greater and lesser disadvantages or access to care. A number of QCIs are not well understood by consumers. Both the degree and type of low comprehension differs among privately insured, uninsured, and Medicaid consumers. The uninsured have less understanding of Desirable Event indicators than the other two groups, perhaps due to less access to preventive care. The privately insured are more aware or more willing to address their own information deficits; have proportionately less misinformation; have fewer deterministic views; and have better understanding of quantitative and aggregate concepts. This may give the privately insured an advantage in being informed consumers. Overall, however, all three consumer groups have low comprehension of QCIs. Undesirable Event indicators are the least understood and have the highest levels of misinformation and myth associated with them.
Informed plan choice cannot occur if QCIs are not understood. Comments reflecting low comprehension are analyzed to determine what, specifically, consumers do not understand. Categories of comprehension deficits include: foundational knowledge; aggregate and quantitative concepts; and consumer beliefs (plan-related and deterministic beliefs). Consumers interpret QCIs in ways that are not intended by policymakers. Misinterpretation of quality information can result in poor choices for individual consumers. But, the effects of poor choice extend beyond the individual. It is not known how many informed consumer choices are required to influence the market for health care reform. The findings here raise concern about whether consumers will be able to understand and use quality information to make informed choices and to collectively influence the market.
Dissemination of report cards must include an educational component, including information about the current health care context and how to interpret and use quality information. However, additional strategies may be required to meet informational goals for individuals with different backgrounds, prerequisite knowledge, and experience with the system.
Technical Note
Quality of Care Indicators Used in Study
Desirable Events (Preventive Measures)
Rate of mammograms among older female plan members (over age 50).
Rate of immunizations among children in the health plan.
Rate of cholesterol screening among plan members.
Rate of eye exams among diabetic plan members.
Rate of pap smears among female plan members over age 50.
Undesirable Events
Hospital-acquired infection rates.
Post-surgery complication rates.
Hospital death rates after heart attacks.
C-section birth rates.
Rates of pediatric asthma hospitalizations.
Rates of low-birth-weight babies.
Patient Ratings of Satisfaction and Quality
Patient ratings overall of the quality of health plans.
Patient ratings of time spent with the doctor.
Patient ratings of respect given to patients.
Patient ratings of doctor's communication.
Disenrollment: number of members who quit the plan.
Disciplinary Actions
Malpractice judgments.
Professional organization disciplinary actions.
Acknowledgments
Support for this study was provided by the Agency for Health Care Policy and Research (AHCPR), U.S. Public Health Service, under Grant Number RO1-08231-01. The authors are with the University of Oregon, Department of Planning, Public Policy and Management. All views and opinions are those of the authors and do not necessarily reflect the views or policy positions of AHCPR, the University of Oregon, or the Health Care Financing Administration (HCFA).
Footnotes
Loss ratios represent the percentage of each dollar in premiums that is spent on health care benefits. The balance, often referred to as “retention,” is devoted to the costs of administration (e.g., claims payment, marketing) and profit.
Reprint Requests: Judith Hibbard, Dr.P.H., Department of Planning, Public Policy and Management, University of Oregon, Eugene, Oregon 97403-1209. E-mail: jhibbard@Oregon.uoregon.edu
References
- Baker SK, Kameenui EJ, Simmons DC, Stahl SA. Beginning Reading: Educational Tools for Diverse Learners. School Psychology Review. 1994;23(3):372–391. [Google Scholar]
- Carnine D. Diverse Learners and Prevailing, Emerging, and Research-Based Educational Approaches and Their Tools. School Psychology Review. 1994;23(3):341. [Google Scholar]
- Cockburn J, White V, Hirst S, Hill D. Barriers to Cervical Screening in Older Women. Australian Family Physician. 1992;21:973. [PubMed] [Google Scholar]
- Fulton J, Buechner J, Scott H, et al. A Study Guided by the Health Belief Model of Predictors of Breast Cancer Screening of Women Forty and Older. Public Health Reports. 1991;106:410. [PMC free article] [PubMed] [Google Scholar]
- Hibbard JH, Jewett JJ. What Type of Quality Information Do Consumers Want in a Health Care Report Card? Medical Care Research and Review. 1996:53–28. doi: 10.1177/107755879605300102. [DOI] [PubMed] [Google Scholar]
- Kottke TE, Trapp MA, Fores MM, et al. Cancer Screening Behaviors and Attitudes of Women in Southeastern Minnesota. Journal of the American Medical Association. 1995 Apr 12;273(14):1099. [PubMed] [Google Scholar]
- Lyons B. Managed Care Attitudes and Information Needs: Medicare and Medicaid Focus Group Results. Paper presented at the American Public Health Association Annual Meeting; San Diego, CA.. November 1995. [Google Scholar]
- National Assessment of Educational Progress. NAEP 1992 Reading Report Card for the Nation and States: Data From the National and Trial States Assessments. U.S. Department of Education; Washington, DC.: 1992. [Google Scholar]
- National Center for Educational Statistics. Adult Literacy in America: A First Look at the Results of the National Adult Literacy Survey. U.S. Department of Education; Washington, DC.: 1993. [Google Scholar]
- National Research Council of the National Committee on Science Education Standards and Assessment. National Science Education Standards: A Sampler. Washington, DC.: 1993. [Google Scholar]
- Stein M, Dixon RC, Isaacson S. Effective Writing Instruction for Diverse Learners. School Psychology Review. 1994;23(3):392. [Google Scholar]
- Williams MV, Parker RM, Baker DW, et al. Journal of the American Medical Association. 1995 Dec 6;274(21):1677. [Google Scholar]