Usability of the Medicare Health Web Site

To the Editor: The use of Internet-based applications to support health management tasks has grown enormously.¹ Studies investigating these applications have focused on prevalence of or reasons for use. We evaluated the ability of a sample of older adults to use the Medicare.gov Web site to make decisions concerning eligibility for services and prescription drug plans.

Methods

Two hundred eighteen people responded to newspaper advertisements and flyers in community and senior centers in the greater Miami area through telephone contact. Participants were required to be aged at least 50 years; be cognitively unimpaired (Mini-Mental State Examination² score >26); have had computer experience (assessed via questionnaire³); be English speaking; and not have depressive symptoms or severe visual or hearing impairments. Following screening, 201 were eligible for participation, 64 of whom expressed lack of interest or did not report for the study. Participants were enrolled consecutively; 112 completed the protocol and were compensated $50 (Table 1). To characterize the sample, data were collected on age, education, and race/ethnicity (based on self-report using fixed categories⁴). Data were collected at the University of Miami between February 2006 and May 2007. The study was approved by the university’s institutional review board, and all participants provided written consent.

Table 1.

Characteristics of the Study Sample

	Participants (n = 112)	Nonparticipants (n = 89)a	P Value
Age, mean (SD), y	63.72 (9.39)	65.80 (9.10)	.12b
Sex, No. (%)  Men	32 (28.6)	19 (21.3)	.24c
Women	80 (71.4)	70 (78.7)
Race/ethnicity, No. (%)  Hispanic	43 (38.4)	32 (36.0)	.64c
Non-Hispanic white	43 (38.4)	34 (38.2)
Non-Hispanic black	19 (17.0)	13 (14.6)
Non-Hispanic otherd	7 (6.3)	10 (11.2)
Education level, median (IQR), y	14 (14–17)	14 (12–16)	.12e
Overall health, median (IQR)f	3 (3–4)	NA
Health literacy, median (IQR), S-TOFHLA scoreg	35 (33–36)	NA
Computer experience, median (IQR)h	73.5 (53.3–93.8)	NA
Internet experience, median (IQR)i	10 (6–15)	NA

Abbreviations: IQR, interquartile range; NA, not applicable; S-TOFHLA, Short Test of Functional Health Literacy in Adults.

^aNonparticipants included 25 persons who enrolled but did not complete the study for the following reasons: 14 were disqualified (8 failed Internet training, 1 reported depressive symptoms, 4 had low Mini-Mental State Examination scores,² 1 had language problems) and 11 quit (5 before starting the task, 6 during the task).

^bUsing t test.

^cUsing χ² test.

^dExcluded from χ² test because this category includes individuals reporting a large variety of ethnicities in small number.

^eUsing Mann-Whitney test.

^fHealth was rated using a 5-point scale (1 = poor to 5 = excellent).⁴

^gThe S-TOFHLA is a 36-item scale that requires participants to answer queries based on 2 passages using multiple choice.⁵ Scores can range from 0 to 36, with a higher score indicating better health literacy. A score lower than 17 indicates inadequate health literacy.

^hAmount of experience with various input devices, computer operations, window operations, and software applications (scores range from 0 to 133 with higher scores indicating greater breadth of experience).³

ⁱScore indicating length of time and frequency of use of the Internet and range of Internet activities (scores range from 0 to 25 with higher scores indicating greater breadth of experience).³

Participants were asked to use the Medicare.gov Web site to (1) determine eligibility for home health care services, (2) select a home health agency to meet specified needs, (3) make a decision about enrollment in Medicare Part D based on specified criteria, and (4) select a drug plan and determine associated costs based on a specified medication regime. They provided written responses. Participants also evaluated Web site usability and the value of Internet health information via questionnaire. Responses were rated by 2 investigators using a scoring sheet and through analysis of video records.⁶ Interrater reliability on 2 problems in a random sample of 20 participants was good (r=0.96 and r=0.97).

During data collection, changes were made to the appearance of the home page and sizing of selection buttons on the Web site; content was not altered. To assess comparability, we compared performance scores, usability ratings, and characteristics of participants who used the initial (n=82) and modified (n=30) versions using χ² tests, t tests, and Wilcoxon tests. No differences were found for ratings of usability or performance scores. A greater percentage of Hispanic (46.2%) and non-Hispanic white (44.9%) participants used the initial version than non-Hispanic black participants (9.0%), but more non-Hispanic black participants (42.9%) used the modified version than Hispanic (28.6%) or non-Hispanic white participants (28.6%) (P<.001). Analyses were performed using SPSS 15.0 (SPSS Inc, Chicago, Illinois), with 2-sided significance set at P<.05.

Results

Most participants were unable to specify all eligibility criteria for home health services (n=77 [68.8%; 95% confidence interval {CI}, 59.8%–76.8%]), choose the correct home health agency (n = 90 [80.4%; 95% CI, 72.3%–86.9%]), or execute computation procedures needed for making a plan enrollment decision (n=94 [83.9%; 95% CI, 76.3%–89.8%]). Only about half (n=64 [57.1%; 95% CI, 47.9%–66.0%]) were able to make an enrollment decision. Most participants (n=81 [72.3%; 95% CI, 63.5%–80.0%]) were unable to select a drug plan or had problems navigating to the necessary Web pages, locating needed information, or executing actions required for plan selection. Participants also reported problems with usability (Table 2), but almost all indicated that the Internet was a valuable source of health information.

Table 2.

Task Performance and Usability Ratings (n = 112)

	Time, Score, or Rating
Task performance times, median (IQR), min  Specify eligibility for home health care	10 (8.3–11.8)
Choose correct home health agency	20 (18–22)
Execute annual cost computation procedure	15.5 (11.3–20)
Selection of prescription drug plan	29 (21–31)
Task performance measures, median (IQR)  Specify eligibility for home health carea	0 (0–4)
Navigation scoreb	9 (2–12)
Performance scorec	7 (0–9)
Interpretation scored	2 (0–3)
Medicare Web site usability ratings, No. (%) [95% CI]e  I found it difficult to navigate within the site—I was getting lost	69 (61.6) [52.4–70.2]
I became frustrated using the Web site to search for information	68 (60.7) [51.5–69.4]
In general, I found it difficult to locate the information that I needed within the Web site	61 (54.5) [45.2–63.5]
Overall, I found the Web site difficult to use	55 (49.1) [40.0–58.3]
Rating of Internet information, No. (%) [95% CI]e  I think the Internet is a valuable tool for finding health information	106 (94.6) [89.3–97.7]
In general, I would use the Internet to find health-related information	107 (95.5) [90.5–98.3]

Abbreviations: CI, confidence interval; IQR, interquartile range.

^aThe correct number of 4 required criteria provided by the participant (scores range from 0 to 4 with higher scores indicating better performance).

^bThe number of required Web pages accessed and method of access for selection of prescription drug plan (scores range from 0 to 14 with higher scores indicating better performance).

^cThe number of required actions correctly executed for selection of prescription drug plan (scores range from 0 to 13 with higher scores indicating better performance).

^dCorrect interpretation of required out-of-pocket costs for selection of prescription drug plan (scores range from 0 to 5 with higher scores indicating better performance).

^ePersons responding strongly agree or agree on a 5-point scale (strongly agree, agree, neutral, disagree, strongly disagree).

Comment

A sample of relatively well-educated adults with computer experience had difficulty using the Medicare.gov Web site to determine eligibility for services and enroll in a drug plan. Participants also reported problems navigating within the Web site and locating information. Although the sample was relatively small and restricted to people aged 50 years or more, it was diverse and larger than samples included in most usability studies. It is likely that persons with less computer experience would have even greater difficulty. To ensure that electronic health tools reach their full potential, broad and inclusive input from consumers should serve as the basis for design.