Measuring Beneficiary Knowledge in Two Randomized Experiments

Abstract

This article reports results of two studies that measured beneficiaries' knowledge of the Medicare program and related health insurance options using pre- and post-experimental designs. Knowledge was measured using multiple item indexes before and after receiving new informational materials developed by the Centers for Medicare & Medicaid Services (CMS) as part of the National Medicare Education Program (NMEP). Beneficiaries in both studies showed statistically significant gains in knowledge after receiving the new materials. Policy implications for the measurement of knowledge and creation of future versions of the materials are discussed.

Introduction

The Balanced Budget Act (BBA) of 1997 increased the number and type of health insurance options available to Medicare beneficiaries. With this increase in choices, efforts to disseminate informational resources to beneficiaries are needed if these new options are to be seriously considered. CMS has implemented the NMEP in response to the BBA mandate. The NMEP is a multifaceted program that attempts to increase beneficiary awareness of Medicare options, access to informational resources, understanding of health plan choices, and use of informational resources to inform decisionmaking.

Several studies have demonstrated the relatively low level of health insurance knowledge among Medicare beneficiaries (Hibbard et al., 1998; Isaacs, 1996; McCall, Rice, and Sangl, 1986). In addition to not understanding the basic Medicare program, there is evidence that beneficiaries do not understand their plan choices. Hibbard et al. (1998) investigated the degree to which beneficiaries understood the differences between the major plan design options that beneficiaries face between Medicare managed care and the Original Medicare plan. They reported that nearly 30 percent of all respondents know almost nothing about health maintenance organizations (HMOs) and that the large majority of respondents did not understand the differences between the FFS and managed care delivery systems.

The provision of information to beneficiaries does not guarantee beneficiary understanding. Achieving increases in understanding is a much more difficult task. This is particularly true with regard to helping beneficiaries understand the complexities associated with the different Medicare options. Achieving increases in the comprehension of Medicare plan options is especially challenging given that baseline levels of comprehension are so low. Yet, the ultimate goal of the NMEP is to increase informed choice. Informed choice requires not only an understanding of the options but also an understanding of the advantages and disadvantages associated with each option, and how significant they are for one's own situation. This study assesses early steps toward increasing beneficiary awareness and understanding of Medicare plan options under the NMEP.

One component of the NMEP program is the Medicare & You handbook that contains basic information about the Medicare program, supplemental insurance, managed care, and other plan options. The 1999 version of the handbook was pilot-tested in five States and the Kansas City metropolitan statistical area (MSA) in fall 1998. Following this pilot study, a national randomized evaluation of the handbook was conducted. This article reports the effects of receiving the handbook on beneficiary knowledge in both the Kansas City MSA pilot study and the national evaluation. In both studies, beneficiaries in the treatment group (who received the handbook) demonstrated modest gains in knowledge in Medicare coverage and new plan choices, but the majority of beneficiaries still had low levels of knowledge in several areas (McCormack et al., 2002; McCormack et al., 2001). This article reports how knowledge among control group members changed over time.

The Kansas City study focused on knowledge gains about the basic Medicare program, while the national evaluation focused on knowledge gains in awareness and understanding of the different Medicare plan design options. Specific research questions are: How much and what kind of knowledge is gained from the distribution of the Medicare & You handbook? And, in what areas are beneficiaries the least and most knowledgeable?

Data and Methods

Study Design

We used data from two different surveys of the Medicare population—an evaluation of the pilot version of the Medicare & You 1999 handbook conducted in the Kansas City MSA and a national evaluation of the Medicare & You 2000 handbook. The survey instruments were developed to measure beneficiary knowledge of the Medicare program and related health insurance options, use of Medicare information sources, and attitudes about health plan choice and decisionmaking. They were developed by a multidisciplinary team of RTI and CMS staff members and consultants to reflect information provided in the handbook and some of the key messages that CMS was trying to convey to beneficiaries at the time.

Data for the 2000 handbook evaluation were collected using a mail survey with telephone followup of a national random sample of Medicare beneficiaries drawn from CMS's 1998 enrollment data base. This file contained a random selection of 1 percent of all Medicare beneficiaries, making it representative of the entire Medicare population. Beneficiaries excluded from the sampling frame included those with the following characteristics:

• Were dually eligible for Medicare and Medicaid.

• Lived outside the 50 States and Washington, DC.

• Had end-stage renal disease.

• Were institutionalized in skilled nursing or long-term care facilities.

• Were receiving hospice care.

• Were originally eligible for Medicare because of a disability.

For both studies the design included a treatment and a control group. Data were collected in the preintervention period from the control group only, and in the post-intervention period from both the control and the treatment groups. This analysis focuses on changes in knowledge levels measured at preintervention (assessing controls only) relative to knowledge levels at post-intervention.

Beneficiaries were randomly assigned to a control group who received no information as part of the study or to a treatment group who received the handbook. Because the Medicare & You handbook was customized with information about local health plans, samples for both control and treatment groups were drawn proportionally to the size of the region. The proportion of demographic subgroups (age, sex, and race) was also maintained across these groups and regions. It is important to note that although the control groups received no information as part of the study, they could have received Medicare information from other sources.

For the national evaluation, data collection activities were conducted with the control group from July to October 1999, just before CMS's first national mailing of the handbook (n=1,175 completed interviews), and with the treatment group from October 1999 to February 2000, which was immediately after the national mailing (n=2,563 completed interviews). The overall response rate for the survey was 76 percent. We conducted a followup survey with the control group (only) after the national mailing of Medicare & You 2000 handbook; 843 members responded equating to a 72-percent response rate.

The Kansas City evaluation followed a similar sampling approach and study design except that the study was restricted to residents of the 10-county Kansas City MSA. The survey was conducted using a telephone interview between September 1998 and January 1999 and had the response rate of 62 percent. We interviewed 1,156 experienced beneficiaries who were divided about equally across the control group (that received no information as part of the study) and three separate treatment groups that received different sets of intervention materials. Similar to the national study, we re-interviewed the control group members after dissemination of Medicare & You 1999 handbook.

In the Kansas City study, however, only 104 of the 320 Medicare beneficiaries in the control group participated in the followup survey for a response rate of 33 percent. The large difference in the sample sizes for these two surveys (104 versus 843) may have contributed to a difference in effect size between the studies, with the national evaluation survey having the potential for greater levels of significance. The low response rate of the Kansas City followup survey is also a limitation.

Since the control groups in the two studies eventually received the study intervention (i.e., the national mailing of the handbooks), we refer to the initial (control group) interview as the pre-test period and the followup interview (with the control group) as the post-test period.

Variable Construction

We constructed five types of variables from survey data and CMS administrative data including: (1) one variable for each knowledge question reflecting whether a beneficiary responded correctly, (2) two indexes of beneficiary knowledge, (3) a variable indicating whether a respondent had recently seen a copy of the Medicare & You 2000 handbook (this question was not asked in the Kansas City evaluation), (4) beneficiary characteristics, and (5) other independent variables to be used as controls. A 5-question knowledge index was developed for the questions administered in the Kansas City survey, and a 15-question knowledge index was developed for the national evaluation questions.

Knowledge Indexes

The five knowledge items in the Kansas City survey addressed knowledge of the Original Medicare plan and related Medicare health plans (Table 1). The dominant theme in these items focused on the basic Medicare program. Each question had only one correct answer, and missing responses or responses of “don't know” were coded as incorrect, which was the approach taken in previous studies (Hibbard et al., 1998). The overall knowledge index score was calculated as the proportion of items answered correctly among the five knowledge questions.

Table 1. Correct Responses on the Five Knowledge Questions in the Kansas City Medicare & You 1999 Pilot Survey.

Knowledge Question		Survey		Absolute Increase
		Pre	Post
			Percent
1.	As far as you know, can people on Medicare still get Original Medicare today? (Yes/No)	75.0	80.1	5.1
2.	As far as you know, does Original Medicare pay for all health care costs for people on Medicare? (Yes/No)	77.8	89.5	*11.7
3.	If a person signs up for any of these Medicare health insurance plans is he or she still in the Medicare program? (Yes/No)	65.1	82.6	**17.5
4.	If a person signs up for any of these Medicare health insurance plans does he or she still get at least the same health care benefits as someone in the Original Medicare program? (Yes/No)	52.2	68.8	**16.6
5.	Do people on Medicare who are happy with the health insurance plan they have now have to change plans this year? (Yes/No)	78.2	81.2	3.0

^*Indicates a significant difference at the p=0.05 level between the pre- and post-test distributions of correct and incorrect responses.

^**Indicates a significant difference at the p=0.01 level between the pre- and post-test distributions of correct and incorrect responses.

NOTES: Response options are shown in parentheses following each question, with the correct response shown in bold. A “don't know” response was considered incorrect.

SOURCE: Survey of Medicare beneficiaries in the Kansas City metropolitan statistical area conducted by Research Triangle Institute between September 1998 and January 1999.

A similar scoring algorithm was used for the 15-item national evaluation knowledge index which addressed the following 7 topics: (1) awareness of Medicare options, (2) access to the Original Medicare plan, (3) cost implications, (4) coverage and benefits, (5) plan rules and restrictions, (6) availability of information, and (7) beneficiary rights.¹ The dominant theme in these items focused on the differences between the Original Medicare plan and Medicare managed care. The knowledge index scores were calculated as the percentage of correct answers to all 15 knowledge questions. Again, “don't know” responses and missing responses were assigned a value of incorrect. Although alternative weighting mechanisms were considered when developing the scales, each of the questions was equally weighted in the knowledge indexes.

Theoretically, the absolute gains in knowledge among control group members in both studies may be biased upward because of their participation in the pre-test survey which could have increased the likelihood of looking at the handbook after completing the initial interview. To explore this possibility, we compared the absolute gains in the 15 knowledge questions for the pre- and post-test control groups with the pre-test control group data and treatment groups data. We found that those who were reinterviewed as part of the followup post-test survey scored approximately 1 to 2 percentage points higher than the treatment group (who received the handbook but was only interviewed once) on 9 of the 13 questions (in which either the control or treatment group had a significant gain in knowledge). This suggests that the post-test group may have been influenced by their pre-test experience to a small degree.

We conducted psychometric analyses to assess the internal consistency reliability and construct validity of the two knowledge indexes. Internal consistency reliability measures the degree to which items on a scale are related to each other and therefore appear to be measuring the same construct. The internal consistency reliability of the indexes was estimated using Cronbach's alpha (Cronbach, 1951). A common rule of thumb is to require the coefficient alpha to be 0.70 or above in order for the index score to be considered reliable enough for group comparisons.

The Cronbach's alpha for the national Medicare & You knowledge index was 0.67 at the pre-intervention time point and 0.71 at the post-intervention time point, suggesting that the scale had fairly good internal consistency reliability. The Cronbach's alpha for the Kansas City knowledge index was 0.51 at the pre-intervention time point and 0.56 after the intervention. The lower alpha coefficients for this scale may be due to the smaller number of items included in the scale. Although these values did not reach 0.70, they may still be considered promising. In fact, Helmstadter (1964) indicated that alpha coefficients greater than 0.50 may be considered reliable for new scales under development.

Construct validity concerns whether the scale actually measures what it was designed to measure. One method for evaluating construct validity is to measure the amount of agreement between a scale and other measures of the same construct (Nunnally and Bernstein, 1994). A high level of agreement provides support for the construct validity of the scale. Another test of construct validity is to examine the extent to which the scores on the scale can differentiate between groups who are expected to differ on the construct (McDowell and Newell, 1996). If a scale shows the expected pattern, it will have demonstrated evidence of construct validity. The knowledge indexes in this study were evaluated using both of these validity tests.

First, the relationship of the knowledge indexes with a measure of self-reported understanding was examined. Both the national evaluation and Kansas City surveys contained a question that asked beneficiaries to rate their understanding of the different types of health insurance options for people with Medicare on a 5-point scale from “poor” to “excellent.” Because of the small number of respondents indicating the responses of “excellent” and “very good,” the scale was collapsed into four categories (fair, poor, good, and very good/excellent) to reduce the disparity between the categories.² To assess the construct validity of the knowledge indexes, a general linear model was used to investigate the relationship between these three levels of self-reported understanding and the calculated knowledge index scores.

For the National Evaluation survey, self-reported understanding was significantly positively related to the knowledge index both before the intervention (Wald F(3,817)=24.98, p<0.01) and after the intervention (Wald F(3,819)=27.85, p<0.01). The ordering of the means by self-reported understanding was monotonic. Respondents who had higher self-reported understanding also had higher knowledge scores. For the Kansas City survey, self-reported understanding was again significantly positively related to knowledge scores before the intervention (Wald F(3,102)=3.50, p=0.02). However, self-reported understanding was not significantly related to knowledge scores after the intervention (Wald F(3,102)=1.48, p=.22), perhaps due to a decreased sample size.

In the Kansas City evaluation, the self-reported understanding question was only available at the preintervention time point. Therefore, the preintervention question was used to validate the knowledge index at both the pre- and post-intervention time points. A limitation of this approach is that self-reported understanding may change over time and as a result, the relationship between self-reported understanding and the knowledge index may appear to be smaller than it would if the questions were administered simultaneously.

The next test of the construct validity of the knowledge indexes was to examine whether the knowledge index scores differentiated between individuals expected to differ in level of knowledge. Researchers have consistently found that respondents with more education have higher levels of insurance knowledge (Lambert, 1980; Marquis, 1983; McCall, Rice, and Sangl, 1986; Hibbard et al., 1998; McCormack et al., 2002). Therefore, to assess the construct validity of the knowledge indexes, a general linear model was used to compare the knowledge scores of beneficiaries with different education levels. Respondents were divided into four groups based on educational achievement: (1) less than high school diploma, (2) high school diploma, (3) some college, and (4) college degree. For the national evaluation survey, those with more education had significantly higher knowledge scores both before (Wald F(3,817)=15.03, p<0.0001) and after (Wald F(3,817)=9.63, p<0.0001) the intervention. Similar results were found for the Kansas City survey. Beneficiaries with more education received higher knowledge scores both prior to the intervention (Wald F(3,102)=5.69, p=0.001) and following the intervention (Wald F(3,102)=5.21, p=0.002).

Key Policy Variable

Respondents were asked whether or not they had received the Medicare & You handbook as part of the national evaluation survey. For that study only, we constructed a measure using this question to serve as the key policy variable in the multivariate analysis.

Control Variables

We identified several categories of variables to be used as controls in multivariate analysis, as suggested by theory or that were previously used in the literature. First, we included the beneficiary's pre-test score on the knowledge index as a baseline knowledge measure. We also included scale measures of memory capacity and reading and a measure regarding their degree of computer literacy.

Beneficiary characteristics included in the model were age, sex, race, ethnicity, education, income, and marital status. We also included measures of self-reported health, categories for the number of outpatient visits received, whether the beneficiary had been hospitalized during the year, and whether they had a regular source of care. Several different measures of insurance were also used in the model. We controlled for whether they had employer, individual, or no insurance beyond Medicare, whether they made their insurance decisions alone or had help in doing so, how satisfied they were with their insurance choices, whether they had thought about switching insurance plans in the last year, whether they had ever been in an HMO, and the degree of bias they had regarding HMOs as compared with a FFS arrangement. We also included a measure for the percentage of HMO penetration in the beneficiary's county of residence as we suspected that this may influence beneficiaries knowledge of managed care because of health plan marketing. Finally, we included measures for the amount of information beneficiaries had received on how well doctors in different health plans communicate with their patients (which is a quality of care measure from the Consumer Assessments of Health Plans Study [CAHPS^®]), and the quality of cancer-related services provided by different health plans which is a quality of care measure from the Health Plan Employee Data Information Set (HEDIS^®), expecting that access to other sources of information may influence their knowledge scores. Table 2 shows selected characteristics of beneficiaries in the two study samples.

Table 2. Characteristics of Survey Participants.

Characteristic	Kansas City Control Group (N=104)	National Survey Control Group (N=843)	National Survey Treatment Group (N=3486)
	Percent
Age
65-69 Years	35.1	22.4	26.1
70-74 Years	31.1	30.0	30.0
75-79 Years	33.8	23.3	24.8
80 Years or Over		24.3	19.1
Sex
Male	42.3	41.8	41.0
Female	57.7	58.2	59.0
Race
White	88.0	90.6	92.5
Non-White	12.0	9.4	7.5
Ethnicity
Hispanic	0.0	3.5	3.4
Non-Hispanic	100.0	96.5	96.6
Education
Less than High School	13.4	27.3	27.4
High School Diploma	41.0	31.0	33.1
Some College	17.1	22.8	22.2
College Degree	28.6	18.9	17.4
Marital Status
Married	66.9	58.7	57.9
Not Married	33.1	41.3	42.1
Income
Less than $15,000	17.6	32.0	34.5
$15,000-$30,000	30.2	35.4	34.7
$30,000 or More	33.7	32.6	30.8
Supplemental Insurance
Employer	36.1	38.1	34.3
Individual	46.1	24.7	28.6
None	14.8	29.9	28.1
Type Unknown	3.0	1.1	1.4
Don't Know		6.3	7.6
Decision About Insurance
Alone	51.3	54.6	45.7
Not Alone	48.7	45.4	54.4
Regular Source of Care
Yes	92.2	95.6	95.4
No	7.8	4.4	4.6

SOURCES: Survey of Medicare beneficiaries in the Kansas City metropolitan statistical area conducted by Research Triangle Institute between September 1998 and January 1999. National survey of Medicare beneficiaries conducted by Research Triangle Institute between July 1999 and February 2000.

Analyses

For the descriptive analysis we performed a test of the homogeneity of the marginal distribution of correct and incorrect responses over time for each knowledge question to determine if there was any difference in the percentage of correct responses between pre- and post-test associated with exposure to the Medicare & You handbook. The test measured whether there was a significant difference in the marginal distributions of beneficiary responses. A weighted generalized logistic regression analysis was performed on a four-level dependent variable constructed to reflect the possible distribution of correct and incorrect responses over time (1) correct at both pre- and post-test, (2) incorrect at both pre-and post-test, (3) incorrect at pre-test but correct at post-test, and (4) correct at pre-test but incorrect at post-test). For each question, we tested for a difference between two of the four levels of this variable comparing the frequency of beneficiaries who answered the question incorrectly at the pre-test and correctly at the post-test with the frequency of beneficiaries who answered the question correctly at the pre-test but incorrectly at the post-test. Conceptually, this is similar to McNemar's (1947) test, which also controls for the correlation between observations over time. One limitation of the McNemar's test, is that it does not allow us to control for potentially confounding variables. We used multivariate regression analyses to examine the factors that affected beneficiary knowledge during the post-test period. Both data sets were weighted for the probability of selection and non-response. All analyses were performed in SUDAAN (Shah, Barnell, and Bieler, 1997) to control for the complex designs used in the surveys.

Results

Knowledge Gained from the Medicare & You Handbook

Kansas City Survey

Table 1 shows the percent of correct responses to each of the five knowledge questions before and after the mailing of the Medicare & You handbook. Before the mailing, only about 50 percent of the beneficiaries reported that they were aware of how the benefits in Original Medicare compared with benefits offered by other Medicare health care plan options. (This survey question 4 was the longest and most complex question in the knowledge index, and the low pre- and post-test scores could be a function of this.) About 65 percent of the beneficiaries understood Medicare's relationship with other Medicare health insurance plans (question 3). Three-quarters of beneficiaries were knowledgeable about the availability of Original Medicare (question 1). Almost 80 percent of beneficiaries recognized that Original Medicare will not pay for all health care costs (question 2), and that they could remain in the health care plan of their choice (question 5). Responses of “don't know” ranged from about 10 percent on question 2 to 37 percent on question 4.

After the mailing of the Medicare & You handbook, over 80 percent of the beneficiaries were able to answer four of the five knowledge questions correctly. Nearly 90 percent of the beneficiaries clearly understood that Original Medicare does not pay for all health care costs for people on Medicare. However, only 70 percent of beneficiaries were aware that they get at least the same health care benefits as someone in the Original Medicare program regardless of which Medicare health insurance plan they enroll in. The greatest increases were found for questions related to whether:

• Original Medicare pays for all health care costs for people on Medicare.

• Beneficiaries can stay in the Medicare program if they sign up for any of the Medicare health insurance plans.

• They can still get at least the same health care benefits as in the Original Medicare program if they sign up for any of the Medicare health insurance plans.

To evaluate whether the changes of knowledge are statistically significant, we collapsed the “don't know” responses for each question with the incorrect response category, assuming that beneficiaries do not have the knowledge if they said “don't know,” otherwise they would answer correctly. A form of McNemar's test was used to test the significance of the effect of the mailing on beneficiaries' responses on each of the knowledge questions. As shown in Table 1, the test statistic was significant for three of the five questions, indicating that the mailing materials improved beneficiaries' knowledge in those areas. It was not significant for the first question concerning whether beneficiaries can still get the Original Medicare plan and the fifth question concerning whether people had to change plans if they were happy with their current choice.

National Medicare & You Survey

Fifteen questions in this survey were used to assess beneficiaries' knowledge of the Medicare program and health-related insurance options. There was a significant increase in knowledge scores in 12 of the 15 questions as a result of receiving the Medicare & You materials, with respondents gaining between 2 and 17 percentage points across the questions. A majority of respondents were able to answer correctly only 5 of the 15 survey questions though, indicating a generally low level of beneficiary knowledge, particularly about choices beyond Original Medicare coverage. Responses of “don't know” on these questions ranged from about 11 percent regarding the types of costs paid for under Original Medicare to 87 percent regarding whether the Medicare program had its own Web site. In the post-test scores, 68 percent of beneficiaries knew that they would probably save money by seeing a doctor who accepts assignment, the degree to which the Original Medicare plan pays for health care costs (84 percent), about coverage of services under Original Medicare (58 percent), and which plan option is least likely to cover prescription drugs (57 percent). Sixty-one percent knew which type of health insurance option gives them more freedom to choose their doctors or hospitals.

Areas of Higher and Lower Knowledge

Although knowledge about general Medicare coverage and benefits was high, questions that required beneficiaries to differentiate between Original Medicare and Medicare managed care benefits proved more challenging. In post-test scores, less than one-quarter of beneficiaries, even after the mailing of the Medicare & You handbook, knew whether Original Medicare and Medicare managed care covered specific benefits such as a 6-month stay in the nursing home (22 percent), emergency health care (19 percent), and preventive health care services (13 percent). Another area of low knowledge among beneficiaries is related to supplemental insurance. Beneficiaries in the post-test survey did not know that if their medigap policy was dropped, a beneficiary could only become a policyholder again under certain conditions (20 percent), and that availability of medigap coverage was contingent on the health of an individual (15 percent). Further, knowledge about the availability of Medicare information was also low, as was knowledge about beneficiary rights. Thus, beneficiary knowledge is low in the areas most critical to informed choice—being able to differentiate between Original Medicare and Medicare managed care, understanding the circumstances under which one may obtain a medigap policy, and how to get Medicare information.

The greatest percentage point gains in knowledge occurred among beneficiaries who became aware of the availability of Medicare informational resources and their possession of certain beneficiary rights, perhaps because only about one-quarter of beneficiaries knew about these issues before receiving the Medicare & You materials. Even in post-test scores, only 41 percent of beneficiaries were aware that information and counseling services were available, and only 23 percent of beneficiaries were aware of the Medicare Web site. Considering post-test scores on beneficiary rights, only 32 percent knew, that at the time of the survey, that they could leave a Medicare managed care plan at any time, only 39 percent were familiar with the reasons for contacting the peer review organization for their State, and only 21 percent knew they had the right to appeal decisions about what a Medicare plan would pay for.

The McNemar's test showed that exposure to the handbook had a significant effect in 12 of the 15 questions. It did not have a significant effect for the question about coverage of nursing home benefits, the question about coverage of preventive health services, and the question about freedom to choose doctors or hospitals, where percentage points gains were relatively small.

Table 3 shows the mean score of the knowledge index for the national Medicare & You survey among different subgroups. Seeing the handbook raised the mean knowledge index score from 0.32 to 0.38 for the average beneficiary in the sample. Mean index scores improved for beneficiaries in all subgroups in the post-test period. What is interesting to note is that gains achieved by the more educated are no greater than those achieved by the less educated, suggesting that education level influences where one's Medicare knowledge level begins, but does not have as much of an impact on how much can be learned.

Table 3. Mean Knowledge Index Scores for Selected Subgroups in the National Medicare & You 2000 Survey.

Characteristic	Test Score		Percentage Point Gain
	Pre	Post
		Percent
Overall	32	38	7
Age
65-69 Years	33	40	7
70-74 Years	33	40	7
75-79 Years	32	40	8
80 Years or Over	28	34	6
Sex
Female	31	37	6
Male	33	40	8
Race
White	32	39	7
Non-White	24	29	6
Ethnicity
Hispanic	26	39	*13
Non-Hispanic	32	39	7
Education
Less than High School	26	34	7
High school Diploma	31	38	7
Some College	36	42	6
College Degree	36	44	7
Marital Status
Married	34	41	7
Unmarried	28	35	7
Income
Less than $15,000	25	32	7
$15,000-$30,000	33	40	7
$30,000 or More	37	43	6
Supplemental Insurance
Employer-Sponsored	33	40	7
Individually Purchased	36	42	5
No Supplemental Insurance	29	35	6
Decision About Insurance
By Self Alone	30	37	7
Not by Self Alone	33	40	6
Regular Source of Care
Yes	32	39	7
No	28	36	8
Information on Doctor Communication
A Lot/Some	36	41	5
A Little	34	38	4
None	32	40	** 8
Information on Cancer Care
A Lot/Some	33	39	5
A Little	37	43	6
None	32	39	6

^*Indicates a significant difference at the p=0.05 level between the pre- and post-test distributions of correct and incorrect responses.

^**Indicates a significant difference at the p=0.01 level between the pre- and post-test distributions of correct and incorrect responses.

NOTE: Knowledge index is the percentage of the 15 knowledge questions that were correctly answered.

SOURCES: Survey of Medicare beneficiaries in the Kansas City metropolitan statistical area conducted by Research Triangle Institute between September 1998 and January 1999. National survey of Medicare beneficiaries conducted by Research Triangle Institute between July 1999 and February 2000.

The descriptive results indicate that individuals who benefited from the Medicare & You 2000 handbook were those who at the pretest reported having no information on doctor communication. In comparison, beneficiaries who reported having any information on doctor communication had a smaller increase in their knowledge scores. Given the many tests of significance in Table 3, additional analysis is needed to determine if these differences are due to chance alone. Due to low sample size, the significant findings for Hispanic beneficiaries should be interpreted with caution.

Multivariate Results

The impact of seeing the Medicare & You 2000 handbook on post-test knowledge index scores for beneficiaries in the national survey was evaluated using a weighted ordinary least square regression model. The post-knowledge index score was used as the dependent variable and the pre-knowledge index score was used in the model to control for beneficiary knowledge prior to the mailing of the handbook.

As shown in Table 4, beneficiaries who reported seeing the Medicare & You handbook scored 8.5 percentage points higher on their post-test knowledge index score than beneficiaries who had not seen the handbook controlling for other factors. This modest gain in knowledge is equivalent to beneficiaries having answered slightly more than one additional question correctly on the 15-item knowledge index after seeing the handbook. A strong relationship was identified between the pre-test and post-test knowledge index scores. For every one percentage point increase in the pre-index score, the post-index score increased by approximately one-half of a percentage point. This measure was highly significant and may have captured the explanatory effect usually demonstrated by a beneficiary's level of education, which was not a significant predictor of knowledge. Reading and memory measures were not significant predictors of beneficiary knowledge, perhaps for the same reason.

Table 4. Ordinary Least Square Regression on the 15-item National Medicare & You 2000 Knowledge Index.

Characteristic	Beta	Standard Error
Intercept	0.1327	-0.0506
Saw Medicare & You 2000 Handbook
Yes	*0.0853	-0.0124
Knowledge Index at Pre-Intervention Time Period	*0.5299	-0.0397
Age Group
65-74 Years	-0.0009	-0.0025
75 Years or Over	-0.0022	-0.0016
Sex
Female	-0.0155	-0.0141
Race
Non-White	*-0.0538	-0.0219
Ethnicity
Hispanic	0.0236	-0.0369
Education
High School Diploma or General Equivalency Degree	0.0128	-0.0165
Some College or Technical Degree	0.0124	-0.0197
College Degree	0.0170	-0.0196
Income
$15,000-$30,000	0.0215	-0.0153
$30,000 or More	0.0130	-0.0189
Marital Status
Married	0.0150	-0.0152
Supplemental Insurance
Employer	-0.0024	-0.0155
Individual	-0.0005	-0.0159
Unknown What Type	0.0939	-0.0640
Don't Know	0.0403	-0.0275
Have a Regular Doctor
Yes	-0.0081	-0.0264
Number of Doctor Visits in Last Month
1 Visit	-0.0034	-0.0167
2 Visits	0.0021	-0.0186
3 or More Visits	-0.0018	-0.0195
Any Hospitalizations in Last Year
Yes	0.0074	-0.0152
Memory Index Score	-0.0061	-0.0085
Reading Index Score	-0.0005	-0.0024
Penetration of Medicare HMO Enrollment	-0.0001	-0.0004
Information on Doctor Communication
A Lot/Some	-0.0079	-0.0184
A Little	*-0.0453	-0.0173
Don't Know	*-0.0429	-0.0212
Information on Cancer Care
A Lot/Some	-0.0023	-0.0236
A Little	*0.0439	-0.0216
Don't Know	0.0219	-0.0189
Decision About Insurance
Not Alone	-0.0080	-0.0129
Satisfaction with Insurance
Not Very/at All Satisfied	—	—
Somewhat Satisfied	0.0119	-0.0258
Very/Extremely Satisfied	0.0396	-0.0258
Thinking About Switching
A Little	-0.0256	-0.0192
A Fair Amount	*0.0696	-0.0255
A Lot	0.0555	-0.0514
Use of Medicare Web site
Somewhat Likely	*0.0600	-0.0187
Very Likely	0.0376	-0.0199
Don't Know	0.0143	-0.0207
Ever in HMO
Yes	0.0162	-0.0158
Don't Know	0.0425	-0.0381
Degree of HMO Bias
About the Same	0.0136	-0.0179
Better	0.0189	-0.0319
Don't Know	*-0.0429	-0.0138

^*P-value is significant at the 0.05 level.

R²=0.47.

NOTES: Omitted categories are individuals who did not see the handbook, beneficiaries who are male, white, non-Hispanic, have less than a high school education, less than $15,000 annual income, are not married, no supplemental insurance, no regular source of care, had no recent health care utilization, no information at the plan level about quality of care, makes insurance decisions alone, are not satisfied with their insurance choice, have not changed plans recently, are not very likely to use the Medicare Web site, have never been in an HMO, and who think the quality of care given by HMOs is worse than in other types of plans. HMO is health maintenance organization.

SOURCE: National survey of Medicare beneficiaries conducted by Research Triangle Institute between July 1999 and February 2000.

Other significant predictors of the post-index score were being white, having received a little information on how well doctors in different health plans communicate with their patients, and a little information on cancer services provided by different health plans, being somewhat likely to use the computer to gather information, and having thought about switching health plans a fair amount.

The post-index score of beneficiaries who were non-white was 5.4 percentage points lower than the post-index score for those who were white, after controlling for all other factors. Although non-white and white beneficiaries had a similar percentage point gain between the pre- and post-time periods, the baseline mean index score among non-white beneficiaries was lower than the baseline mean index score among white beneficiaries. Therefore, although the handbook improved knowledge scores among both groups, non-white gained less knowledge.

Results were mixed for beneficiaries who reported having a little quality of care information. Beneficiaries who reported that they were “somewhat likely” to use the Medicare Web site to gather information about Medicare had a significantly larger increase in their post-index score compared with those who said it was “not very likely” that they would use this resource. A positive but non-significant increase was also observed among beneficiaries who reported being “very likely” to use it. Beneficiaries who have access to the Internet may have more informational resources available to them or may be more proactive in accessing and utilizing them.

Finally, beneficiaries who had been thinking about switching health plans (a fair amount) had a 7-percentage point gain in their post-index score than beneficiaries who had not considered switching health plans. Beneficiaries planning to switch health plans may search for information to inform their decisionmaking, and may be more interested in reading the Medicare & You handbook than those beneficiaries who were not considering, or those who had already made up their mind to switch.

Discussion

Overall, we found modest gains in Medicare-related knowledge in both the Kansas City and the national evaluation studies, which used pre-post experimental designs and random assignment of control and treatment group members. Knowledge increased significantly in 3 of the 5 questions in the Kansas City study and 12 of the 15 questions in the national evaluation. The absolute gains in knowledge ranged from 2 to 17 percentage points after receiving the handbook.

The highest increases in knowledge occurred with respect to availability of information from Medicare and their rights as beneficiaries, while the increases were smaller for questions about Medicare coverage and plan options. While the gains on some of the individual questions were impressive, overall levels of knowledge in the population are fairly low, with a majority of beneficiaries knowing the correct answer at post-test on only 10 of the 20 questions (when the descriptive results from the two surveys are considered together). Further, the areas of low knowledge are those that are most critical to informed choice—being able to differentiate between Original Medicare and Medicare managed care, issues in obtaining a medigap policy, and knowing how to get information on the Medicare program.

The increase in knowledge index scores among members of the control group in the national evaluation survey is consistent with findings illustrating gains in knowledge among treatment group members who were exposed to the handbook (McCormack et al., 2002). While these findings show the benefit of educational materials, additional interventions may be needed to increase knowledge in areas that did not exhibit significant gains. Materials could be targeted at subgroups in the greatest need of the information, such as those considering changing plans, or certain key messages could be directed at selected subgroups.

The pre-test knowledge score of beneficiaries was a strong predictor of the post-test knowledge score. Beneficiaries with lower pre-test scores may have more to gain than beneficiaries with higher pre-test scores, given that the latter group may have less remaining to learn. If so, there may be decreasing returns for a unit investment in knowledge at the higher end of the knowledge scale.

The findings are useful for planning future educational efforts in this population. Issues that are most important for beneficiaries to know may need to be highlighted using multiple information channels as part of a broad campaign. This might include motivational messages that inform beneficiaries about the availability of the handbook and related materials.

Future assessments of beneficiary knowledge should include an effort to uncover not just lack of knowledge but also misconceptions that beneficiaries hold. Misconceptions can be major barriers to making appropriate choices and are best addressed with messages that directly target the misconception. Other possible barriers to explore are: cognitive declines, literacy problems, perceptions about availability of choice, and motivations to have and use Medicare information.

Finally, it will be important to conduct further research that examines different approaches to educating beneficiaries. What methods work with different segments of the population? What is an acceptable level of knowledge (among beneficiaries or their families) to make satisfying and appropriate choices? There is also a need to assess the costs of producing targeted knowledge levels. What are the costs associated with different approaches that yield a similar level of knowledge gain? Given that the Medicare program continues to change over time and new beneficiaries enter the program, analyses of CMS materials on beneficiary knowledge will continue to be needed.