Abstract
Objective
The use of the internet as a source of health information and link to healthcare services has raised concerns about the ability of consumers, especially vulnerable populations such as older adults, to access these applications. This study examined the influence of training on the ability of adults (aged 45+ years) to use the Medicare.gov website to solve problems related to health management. The influence of computer experience and cognitive abilities on performance was also examined.
Design
Seventy-one participants, aged 47–92, were randomized into a Multimedia training, Unimodal training, or Cold Start condition and completed three healthcare management problems.
Measurement and analyses
Computer/internet experience was measured via questionnaire, and cognitive abilities were assessed using standard neuropsychological tests. Performance metrics included measures of navigation, accuracy and efficiency. Data were analyzed using analysis of variance, χ2 and regression techniques.
Results
The data indicate that there was no difference among the three conditions on measures of accuracy, efficiency, or navigation. However, results of the regression analyses showed that, overall, people who received training performed better on the tasks, as evidenced by greater accuracy and efficiency. Performance was also significantly influenced by prior computer experience and cognitive abilities. Participants with more computer experience and higher cognitive abilities performed better.
Conclusions
The findings indicate that training, experience, and abilities are important when using complex health websites. However, training alone is not sufficient. The complexity of web content needs to be considered to ensure successful use of these websites by those with lower abilities.
Keywords: Older adults, e-health, training
Background and significance
Within the healthcare arena, there is a growing emphasis on health self-management, where consumers are increasingly being asked to assume new responsibilities with respect to health management activities. The internet is becoming a focal point for these activities and has become a popular means of communicating information about health topics and services. The US government is also increasingly using the internet as a forum for the exchange of information and services.
In 2010, 80% of internet users and 57% of the total adult population searched online for information about a disease or medical problem, medical treatment, or health professionals.1 Consumers also report that internet health information has an influence on decisions about seeking care, treatment choices, and physician interactions.2 Unfortunately, many ‘e-health’ applications have been designed without considering the needs and characteristics of diverse user groups and thus have increased the complexity of the demands of healthcare engagement.3 Particularly vulnerable groups include those who are less educated, less literate, and older.
Although older adults are receptive to using the internet for healthcare tasks,4 5 they often have more difficulty engaging in internet-based health information seeking.4 6 For example, in a prior study,4 we found that older adults with prior computer experience had difficulty using the Medicare website (Medicare.gov) to solve problems related to benefits coverage. We also showed that older adults with less technology experience and lower cognitive abilities are less successful when engaging in health information seeking.6 7 Likewise, data from the Health and Retirement Survey8 indicate that people with lower cognitive abilities were less likely to enroll in the Medicare Part D program and less likely to use the internet for enrollment. Lack of access to internet-based health services among older adults is particularly problematic given that they typically have more health problems9 and use more healthcare services than younger adults.10 This access must extend beyond simple access to the equipment and the internet to also include having the requisite skills, technical support, and the appropriate and meaningful content.
If the benefits of e-health applications are to be realized by all consumer groups, information is needed about factors related to users' acceptance of, and ability to understand and use, internet health applications.11 Intervention strategies are also needed to help people effectively use these applications. Perceived usefulness and perceived ease of use are two important predictors of patients' acceptance and self-reported use of web-based health self-management technology.12
In this study, we built on our previous research4 and investigated the influence of training, cognitive abilities, and prior internet experience on the ability of adults (aged 45+ years) to use the Medicare.gov website to perform tasks related to health insurance benefits and coverage. Specifically, we examined the potential benefits of two online training strategies: a Unimodal program (text only) and a Multimedia program (text, speech and animation). We chose online training, as e-learning is increasingly becoming a popular form of instruction. Potential benefits of e-learning include flexibility, convenience, and working at one's own pace and from any location where the requisite technology is available.13
Multimedia formats may be especially advantageous to older users. By emphasizing the visual and auditory sensory modalities, these formats offer the potential for reducing cognitive load by distributing information over both of these modalities which, by themselves, are each limited in capacity.14 15 Effective use of narration can also be used to reduce the amount of text information and afford more optimal use of the different limited capacity memory systems in working memory.16 The auditory modality also allows an opportunity to provide cues on where to focus attention to find and process textual or pictorial information more efficiently. Overall, multimedia formats provide a potentially stronger basis for integrating information in working memory, thereby ensuring deeper learning and a greater ability for concepts to be available for future recall.17 18 This may be particularly important for older learners who may have decreased information-processing capacities and may be more prone to cognitive overload in new learning situations. Prior data19 indicate that multimedia-based instruction seems promising for older adults; however, as cautioned by the investigators, these formats need to be tested on more complex and varied applications.
One important caveat is that the Medicare.gov website is broad and provides users with the opportunity to perform a wide variety of activities and solve a variety of different and relatively complex tasks. In this study our objective was not to examine the use of training as a means for learning the rules, regulations, and content of the Medicare system. The focus instead was on presenting a broad tutorial to help orient the user to the homepage and content of the website and to facilitate navigation through the website. In this regard, we examined the impact of the training strategies on a set of representative tasks that could be commonly performed on the website. We also investigated the role of cognitive abilities on performance, as our previous work has shown that these abilities are important to internet-based health information seeking. Understanding the link between these abilities and performance provides important information for the design of health websites and aiding tools. We also examined the role of prior internet experience, as understanding how prior experience influences performance also yields important information for the design of training programs.
Methods
Sample
The sample included 61 adults aged 45+, recruited via advertisement from the South Florida community. This included advertisements describing the study and entry criteria in community newspapers and newsletters, placing flyers in senior centers and libraries, and speaking at senior centers and community agencies. All participants were required to be non-cognitively impaired (a score ≥27 on the Mini Mental Status Examination (MMSE),20) have no indications of depressive symptoms (a score ≥28 on the Center for Epidemiological Studies Depression Scale (CESD)21), have at least 20/40 near and far vision with or without correction, have non-impaired hearing with or without correction, be English speaking, and have internet experience. The institutional review board of the University of Miami approved the study. All data were collected at the University of Miami Miller School of Medicine in a laboratory setting. Written consent was obtained from all participants who were compensated with US$65.00.
Procedure
Overview
The tasks were performed on a Dell desktop computer using the Windows XP operating system. The system was configured with Microsoft Internet Explorer 6.0, the Hypercam 2.10 screen capture utility,22 and Windows Media Player. The Hypercam screen capture utility enabled each participant's onscreen interactions to be recorded in the form of a Windows-based digital movie.
Data were collected on an individual basis over 2 days. After telephone screening, eligible participants were scheduled for participation. On day 1, participants were administered the MMSE, CESD, and vision23 and hearing24 tests, and they completed a background questionnaire, a technology, computer and internet experience questionnaire,25 and a test of Health Literacy (STOFHLA).26 Scores on the STOFHLA range from 0 to 36 and are classified according to level of health literacy: inadequate (0–16); marginal (17–22); adequate (23–36). They then completed a battery of standardized tests that assessed a broad range of their component cognitive abilities—specifically: episodic memory (California Verbal Learning Test27 (CVLT)) and meaningful memory28; fluid intelligence (Alphabet Span,29 Trail Making Test,30 Letter Sets31); perceptual speed (Digit Symbol Substitution,32 and Number of Comparison31); psychomotor speed (Choice Reaction Time33 and Simple Reaction Time33); crystallized intelligence (Shipley Vocabulary34 and Wechsler Adult Intelligence Scale (WAIS)-III Information35) (online appendix 1). Participants were then randomly assigned to one of the following study conditions: (1) Cold Start/No Training; (2) Unimodal Training; (3) Multimedia Training.
On day 2, to ensure that everyone had familiarity with basic computing and internet skills, participants were given a review and practice on basic mouse and Window operations.6 Following a break, they proceeded to the training or cold start protocols, had an additional break, and performed the experimental tasks.
Overview of the training conditions
Both training conditions were designed to provide an overview of the content on the Medicare.gov website and instructions on how to navigate the website and access various features. Training was targeted toward facilitating navigation through the site and not on skills related to completing a particular task (eg, determining eligibility requirements, selecting a health insurance plan). The goal was to provide information about the content on the site and how to navigate pages using links, buttons, menus and check boxes, and the Medicare.gov home page (figure 1A) including the banner, sidebar, spotlights area, and search tools area.
Figure 1.
(A) Overview of the Medicare website homepage and areas of training focus. (B) Review menu for Multimedia segments.
Multimedia Training
The Multimedia Training module (developed using Adobe Flash) presented the information in both a visual and auditory format and included graphics and animation. The training was divided into segments and each segment could be reviewed as often as desired. Participants had the opportunity to return to any segment using a menu containing links to all segments (figure 1B). The narrated audio component ‘walked’ the user through a segment and was complemented by on-screen text that highlighted important features in the narration. Participants had up to 45 min to complete the training.
Unimodal Training
The Unimodal Training presented content identical with the Multimedia condition but in a visual format within the context of a Microsoft PowerPoint presentation. Participants were able to review the information on the slides as needed and had up to 45 min to complete the training.
There were no differences in time spent in training between the two training conditions (F(1,28) =0.65, p>0.05). On average, those in the Multimedia condition spent about 27 min (M=26.66, SD=3.29) interacting with the training module, and those in the Unimodal condition spent about 25 min interacting with the module (M=24.36, SD=10.27).
Cold Start/No Training condition
To control for contact time differences between the Cold Start/No Training condition and the training conditions, those assigned to the Cold Start condition were asked to spend 30 min browsing on the internet.
The experimental tasks
Participants were asked to use the Medicare.gov website (http://www.medicare.gov) to complete three fictitious health-management tasks (online appendix 2). The first task required participants to find the three closest dialysis centers in a given neighborhood to help them with the care of their mother, and to select a center open at specific hours that had the best quality ratings for anemia control. The second task required participants to appeal a prescription refill denial. The third task involved selecting a prescription drug plan, determining annual costs and monthly premium costs associated with the plan, and assessing changes in costs with generic versions of medications. Participants were provided with a fictitious address and a list of three required medications and required dosages (eg, Liptor, 10 mg daily). All three tasks had multiple steps (stages) that required completion. Participants were asked to write down their responses to the tasks. In addition, their performance was captured using Hypercam. Participants were allotted 25 min to complete tasks 1 and 2 and 45 min to complete task 3. These times were based on prior experience with these types of tasks for this population.4
Results
Outcome measures
Three performance measures were derived for each task on the basis of the written responses and data from the Hypercam screen capture utility. An accuracy score (A) was computed which was a composite measure of the correctness of the written response (C= actual score (X)/maximum score) and the number of correct actions completed (F = actual actions (Y)/correct actions). Correct actions represented actions the participant needed to complete to access the information needed to solve the problem (eg, click on a link) and reflected the extent to which a participant traversed the website correctly. Task 1 required a total of five correct written responses (C = X/5) and 16 actions (F = Y/16), and thus A = X/5 + Y/16. Task 2 required eight correct written responses (C= X/8) and six correct actions (F= Y/6), and thus A = X/8 + Y/6. Task 3 required eight correct written responses (C = X/8) and 22 correct actions (F = Y/22) resulting in A = X/8 + Y/22.
An efficiency measure was defined as the accuracy score for the task divided by the natural log of time taken for the task (A/T). A measure of navigation was derived by dividing each task into stages and subsequently computing the degree to which a participant traversed through these stages. Tasks 1 and 2 had six stages and task 3 had seven stages.
Coding of the written responses and the video capture data was standardized and guided by coding sheets. The coding was based on methods developed in our prior research4 6 and involved a comprehensive identification of all steps and actions required for completion of the task. Examples of coded items from the screen capture data included: clicking essential buttons and links, entering information (eg, a zip code), and viewing webpages.
Analyses
Descriptive statistics were used to summarize performance on the tasks and responses to questionnaires. χ2 tests and one-way analyses of variance (with condition as the grouping factor at three levels) were used for comparing participants on demographics, cognitive abilities, and performance outcomes. Pearson product moment correlations were computed to examine relationships between participant characteristics (eg, age, cognitive abilities) and performance.
Given that the three tasks were chosen to have generalizability across common tasks that are performed using the website and that training focused on general use of the site, the values of each performance measure across the three tasks were combined, resulting in an average measure of accuracy, efficiency, and navigation.
We conducted hierarchical regression analyses to examine the role of individual characteristics such as cognitive abilities and internet experience as predictors of performance, and to examine the impact of providing training on performance after controlling for differences on these variables. All data analysis was conducted using IBM SPSS Statistics V.19.
Study participants
Of the 148 participants who were screened for eligibility, 16 expressed disinterest after screening, 51 did not attend their scheduled appointment, six were disqualified on the basis of the MMSE, and four on the basis of the CESD. Of the remaining 71 who were randomized to one of the three conditions, five quit after randomization, two were disqualified because of lack of computer experience, two were disqualified because of problems with English, and one was disqualified because of tremors associated with Parkinson's disease (figure 2).
Figure 2.
Flow of participants. *Problems with language comprehension, tremors, and discrepancy in reporting computer experience. CESD, Center for Epidemiological Depression Scale; MMSE, Mini Mental Status Examination.
The 61 study participants who completed the study included 25 men (41%) and 36 women (59%) ranging in age from 47 to 92 years (M=70.38, SD=10.5). With respect to ethnicity, the sample included whites (69%), Hispanics (13%) and Black/African Americans (15%). The participants were fairly well educated; 56% of the participants had a college degree or beyond (table 1). Most of the participants were not working (72.1%) and reported that they were healthy. The mean score for health literacy was about 34 (M=34.08, SD=1.91), indicating that on average the sample had adequate health literacy.
Table 1.
Sample description
| Cold Start (n=21) | Unimodal (n=21) | Multimedia (n=19) | Overall (n=61) | |
| Age, mean (SD) | 70.38 (10.50) | 68.81 (8.26) | 69.37 (11.00) | 69.87 (9.80) |
| Gender | ||||
| Male, n (%) | 14 (67%) | 5 (24%) | 6 (32%) | 25 (41%) |
| Female, n (%) | 7 (33%) | 16 (76%) | 13 (68%) | 36 (59%) |
| Education | ||||
| No college, n (%) | 9 (43%) | 9 (43%) | 9 (47%) | 27 (44%) |
| College, n (%) | 4 (19%) | 6 (29%) | 5 (26%) | 15 (25%) |
| Advanced degree, n (%) | 8 (38%) | 6 (29%) | 5 (26%) | 19 (31%) |
| Race/ethnicity | ||||
| Hispanic, n (%) | 2 (10%) | 3 (14%) | 3 (16%) | 8 (13%) |
| Non-Hispanic white, n (%) | 17 (81%) | 11 (52%) | 14 (74%) | 42 (69%) |
| Non-Hispanic black, n (%) | 2 (10%) | 6 (29%) | 1 (5%) | 9 (15%) |
| Non-Hispanic other, n (%) | 0 (%) | 1 (5%) | 1 (5%) | 2 (3%) |
| Occupational status | ||||
| Working for pay, n (%) | 2 (10%) | 7 (3%) | 8 (42%) | 17 (28%) |
| Not working for pay, n (%) | 19 (90%) | 14 (67%) | 11 (58%) | 44 (72%) |
| General health | ||||
| Fair, n (%) | 1 (5%) | 4 (19%) | 2 (11%) | 7 (12%) |
| Good, n (%) | 7 (33%) | 6 (29%) | 8 (42%) | 21 (34%) |
| Very good, n (%) | 8 (38%) | 6 (29%) | 7 (37%) | 21 (34%) |
| Excellent, n (%) | 5 (24%) | 5 (24%) | 2 (11%) | 12 (20%) |
| Health literacy, mean (SD) | 34.19 (2.11) | 34.32 (1.42) | 33.74 (2.13) | 34.08 (1.91) |
| Computer experience, mean (SD) | 62.33 (30.31) | 76.24 (30.98) | 68.00 (26.75) | 68.89 (29.58) |
| Internet experience, mean (SD) | 11.90 (7.59) | 13.10 (7.38) | 9.63 (7.18) | 11.61 (7.41) |
| Cognitive abilities | ||||
| California Verbal Learning Test Total Trials, mean (SD) | 52.29 (10.75) | 52.90 (9.09) | 51.42 (9.79) | 52.22 (9.77) |
| California Verbal Learning Test—Delayed, mean (SD) | 10.29 (3.55) | 11.25 (2.20) | 11.21 (2.84) | 10.90 (2.92) |
| Letter Set—correct count, mean (SD) | 15.71 (4.38) | 12.90 (4.48) | 14.47 (5.22) | 14.38 (4.76) |
| Number Comparison—correct count, mean (SD) | 45.14 (9.65) | 42.25 (7.32) | 42.68 (10.32) | 43.40 (9.11) |
| Simple Alphabet Span, mean (SD) | 2.93 (0.73) | 4.08 (0.95) | 4.03 (0.84) | 4.01 (0.83) |
| Absolute Alphabet Span, mean (SD) | 25.33 (8.52) | 26.40 (11.99) | 25.68 (9.70) | 25.80 (9.99) |
| WAIS Information, mean (SD) | 20.90 (3.87) | 19.40 (3.59) | 19.95 (4.43) | 20.10 (3.95) |
| Meaningful memory, mean (SD) | 10.52 (4.95) | 12.65 (5.02) | 11.58 (4.26) | 11.57 (4.77) |
| Shipley Institute of Living Scale, mean (SD) | 34.38 (4.08) | 32.80 (3.66) | 31.68 (4.84) | 33.00 (4.29) |
| MMSE, mean (SD) | 29.05 (0.97) | 29.29 (0.72) | 28.63 (1.07) | 29.00 (0.95) |
| Log of Trail A Time in Sec, mean (SD) | 3.44 (0.35) | 3.66 (0.38) | 3.62 (0.35) | 3.57 (0.37) |
| Log of Trail B Time in Sec, mean (SD) | 4.41 (0.44) | 4.48 (0.34) | 4.38 (0.37) | 4.43 (0.38) |
| Digit Symbol Substitution, mean (SD) | 63.10 (12.71) | 59.10 (10.86) | 62.89 (13.90) | 61.66 (12.43) |
| Log of mean overall simple two-choice reaction time, mean (SD) | 5.73 (0.19) | 5.88 (0.30) | 5.78 (0.16) | 5.80 (0.23) |
| Performance measures | ||||
| Accuracy, mean (SD) | 0.79 (.43) | 0.75 (.43) | 0.94 (.33) | 0.82 (.41) |
| Performance efficiency, mean (SD) | 0.11 (.06) | 0.11 (.07) | 0.13 (.05) | 0.12 (.06) |
| Navigation, mean (SD) | 12.95 (4.62) | 11.62 (5.16) | 14.76 (4.31) | 13.00 (4.83) |
MMSE, Mini Mental Status Examination; WAIS, Wechsler Adult Intelligence Scale.
Twenty-one participants were randomized to the Cold Start condition, 21 to the Unimodal condition, and 19 to the Multimedia condition. There were no differences among the three conditions with respect to participant age, educational level, employment status, health status, prior computer experience, or component cognitive abilities (all p values >0.05). However, there was a difference in prior internet use (F(2, 58) =4.68, p=0.013, η 2=0.14) (table 1). Those assigned to the Cold Start condition reported more internet use, which was assessed via questionnaire36 that asked participants to rate, on a four-point scale, the frequency with which they performed each of 11 activities (eg, banking/money management, entertainment) online. Only 23% of the sample had enrolled in Medicare-Part D at the time of the study, and, of those who enrolled, only one enrolled online and this person used the Medicare website.
Comparison of the training conditions
The analyses of variance revealed that there were no differences among the three conditions in performance accuracy or performance efficiency (all p values >0.05). However, there was a trend in the data indicating that those in the Multimedia condition had higher accuracy than those in the other two conditions (table 1).
We also examined the number of participants who successfully navigated through the stages of the tasks. Forty-two per cent of the sample navigated through all six stages of task 1, 66% of the sample navigated through all six stages of task 2, and 44% navigated through all seven stages of task 3. There were no differences in navigation across the three conditions (F(2, 55) =2.07, p=0.14, η 2=0.07).
Predictors of performance
For the regression analyses, prior internet experience was entered as the first step, the measures of cognitive abilities were added as the second step, and training was added as the third step, which was followed by age. We added training as the third step to examine if training had an impact after differences in prior internet experience and cognitive abilities had been controlled for. As there were no differences between the two training conditions in the accuracy and efficiency measures, we combined the conditions and compared training with no training. Selection of the cognitive ability measures was based on the results of correlational analyses. Age was included in the last step to determine if age had an impact on performance after differences in prior internet experience, cognitive abilities and training had been controlled for. This analysis was also performed for our measure of navigation.
Given the large number of cognitive predictors, we initially examined the relationship between the cognitive ability measures and the performance outcomes using correlational analyses. These analyses revealed that the following measures were most closely correlated with the performance outcomes: CVLT, Letter Set Test, Number Comparison Test, Shipley Institute of Living Scale, and Log of Trails B (all p values <0.01) (table 2). Owing to our relatively small sample and concerns about multi-collinearity, we then examined the relationships among these ability measures. On the basis of these analyses, we retained the following ability measures for our model: Shipley Institute of Living Test (general knowledge), CVLT (memory), and Letter Set Test (reasoning).
Table 2.
Correlation between cognitive variables and the outcome measures
| Accuracy | Performance efficiency | Navigation | |
| California Verbal Learning Test total trials, mean (SD) | 0.41** | 0.41** | 0.27** |
| California Verbal Learning Test—delayed, mean (SD) | 0.36** | 0.35** | 0.28* |
| Letter Set—correct count, mean (SD) | 0.41** | 0.41** | 0.25 |
| Number Comparison—correct count, mean (SD) | 0.29* | 0.28* | 0.19 |
| Simple Alphabet Span, mean (SD) | 0.19 | 0.20 | 0.02 |
| Absolute Alphabet Span, mean (SD) | 0.25 | 0.25 | 0.13 |
| WAIS Information, mean (SD) | 0.27* | 0.25 | 0.23 |
| Meaningful memory, mean (SD) | 0.29* | 0.28* | 0.34** |
| Shipley Institute Of Living Scale, mean (SD) | 0.31* | 0.31* | 0.39** |
| MMSE, mean (SD) | 0.24 | 0.24 | 0.10 |
| Log of Trail A Time in Sec, mean (SD) | −0.23 | −0.22 | −0.17 |
| Log of Trail B Time in Sec, mean (SD) | −0.39** | −0.39** | −0.23 |
| Digit Symbol Substitution, mean (SD) | 0.23 | 0.24 | 0.07 |
| Log of mean overall simple two-choice reaction time, mean (SD) | −0.31* | −0.29* | −0.32* |
*p≤0.05, **p≤0.01.
MMSE, Mini Mental Status Examination; WAIS, Wechsler Adult Intelligence Scale.
As shown in table 3, age was not significant in any of the models after differences in internet experience and cognitive abilities had been controlled for; therefore we chose model 3 as our final model. The final model for performance accuracy (adjusted R2=0.46) contained four significant individual difference predictors: prior internet experience, CVLT, Letter Set Test, and the Shipley Independent Living Test. Prior internet experience accounted for 12% of the variance, and the cognitive ability measures resulted in a significant increment in R2, accounting for an additional 34% of the variance. Examination of the cognitive variables indicated that the Letter Sets Test was the most influential cognitive ability (β=0.39), followed by the Shipley Scale (β=0.33) and the CVLT (β=0.31) (table 4). The results also indicated that after these variables had been controlled for, training was a significant predictor of performance and resulted in a significant change in R2 of 0.04 (table 3).
Table 3.
Regression models
| R2 | Adjusted R2 | ΔR2 | ΔF | DF | p Value | |
| Accuracy | ||||||
| Model 1* | 0.12 | 0.11 | 0.12 | 7.94 | 1, 56 | 0.007 |
| Model 2† | 0.47 | 0.43 | 0.34 | 11.37 | 3, 53 | 0.000 |
| Model 3‡ | 0.51 | 0.46 | 0.04 | 4.02 | 1, 52 | 0.050 |
| Model 4§ | 0.53 | 0.48 | 0.03 | 2.92 | 1, 51 | 0.094 |
| Performance efficiency | ||||||
| Model 1* | 0.14 | 0.12 | 0.14 | 8.81 | 1, 56 | 0.004 |
| Model 2† | 0.47 | .43 | 0.33 | 11.04 | 3, 53 | 0.000 |
| Model 3‡ | 0.51 | 0.47 | 0.04 | 4.65 | 1, 52 | 0.036 |
| Model 4§ | 0.54 | 0.49 | 0.03 | 3.38 | 1, 51 | 0.072 |
| Navigation | ||||||
| Model 1* | 0.09 | 0.07 | 0.09 | 5.26 | 1, 55 | 0.026 |
| Model 2† | 0.36 | 0.31 | 0.27 | 7.31 | 3, 52 | 0.000 |
| Model 3‡ | 0.40 | 0.34 | 0.05 | 3.82 | 1, 51 | 0.056 |
| Model 4§ | 0.40 | 0.33 | <0.00 | .05 | 1, 50 | 0.830 |
Variables: internet experience.
Variables: internet experience, CVLT, Letter Set Test, Shipley Scale.
Variables: internet experience, CVLT, Letter Set Test, Shipley Scale, Training.
Variables: internet experience, CVLT, Letter Set Test, Shipley Scale, Training, Age.
Table 4.
Standardized coefficients (β) for the final regression model
| Accuracy | Performance efficiency | Navigation | |
| Internet experience | 0.28** | 0.30** | 0.27* |
| CVLT | 0.31** | 0.31** | 0.24* |
| Letter Set Test | 0.39*** | 0.38*** | 0.26* |
| Shipley Scale | 0.33** | 0.33** | 0.47*** |
| Training | 0.20* | 0.22* | 0.23 |
*p≤0.05, **p≤0.01, ***p≤0.001.
CVLT, California Verbal Learning Test.
The final model for performance efficiency (adjusted R2=0.47) contained the same four predictors. In this model, prior internet experience accounted for 14% of the variance, and cognitive abilities resulted in a significant increment in R2, accounting for an additional 33% of the variance (table 3). The Letter Set Test was the most influential cognitive ability (β=0.38), followed by the Shipley Scale (β=0.33) and the CVLT (β=0.31) (table 4). After internet experience and cognitive abilities had been controlled for, training was also a significant predictor of performance and resulted in a significant change in R2 of 0.04.
Examination of the model for navigation indicated similar results (adjusted R2=0.34). Specifically, prior internet experience accounted for 9% of the variance, and, after this variable had been controlled for, the cognitive ability measures resulted in a significant increment in R2, accounting for an additional 27% of the variance. Interestingly, the Shipley Scale was the most important cognitive ability (β=0.47), followed by the CVLT (β=0.24) and Letter Sets (β=0.26) (table 4). In this model there was a small trend indicating that training also contributed to R2 (increment in R2=0.05, p=0.06) (table 3). Overall, the results indicate that individuals with prior internet experience and higher cognitive abilities were better able to navigate the website.
Discussion
The internet is becoming a critical medium for accessing health information and services. Prior research has shown4 that older adults often have difficulty using internet-based health applications. In this study we examined the impact of providing basic training about a health website (the Medicare.gov website) on the ability of middle-aged and older adults to use the website to successfully complete a set of tasks. We also compared two training formats: a Unimodal format and a Multimedia format. Additional goals of the study were to examine the influence of prior internet experience and component cognitive abilities on performance.
Overall, our findings from the regression analyses (Table 3) indicated that the training was beneficial. After controlling for internet experience and cognition, participants who received training were able to perform the tasks more accurately and efficiently. This finding is important, especially given that the training was relatively brief and only focused on the general content of the website and navigation; there was no specific task-related training. Furthermore, the participants only had limited exposure to the training and it was confined to one session. Thus our results suggest that health websites that include basic instructional programs may be helpful to consumers and facilitate their ability to use e-health applications. However, the content and format of the instructional program needs to be carefully designed according to current usability guidelines and tested with user groups that include older adults.37
Generally, we did not find the Multimedia format to be significantly more beneficial than the Unimodal format. Participants who received Multimedia training were not more accurate or efficient in terms of their performance on the tasks or able to better navigate the website. It may be that the potential benefits of Multimedia in terms of reducing cognitive load were not realized because of the complexity of the tasks that needed to be performed. Also, as noted, the training was not focused on skills related to a specific health management task. It may also be that a different combination of multimedia formats would have been more effective. Several investigators38 39 have shown that the effectiveness of different multimedia combinations (eg, text with animation, text with narration) varies according to task and performance requirements. Generally, our findings indicate that multimedia formats need further investigation and need to be tested on more complex and varied applications,19 especially with adult user groups, who have more limited processing capacity. The usability of the training protocol should also be evaluated. In addition, these formats should be tested with larger and diverse samples. The sample size in our study was relatively small and a post hoc analysis revealed that the study was not sufficiently powered (π=0.38) to detect group differences. However, increasing the sample size (N=169) would have allowed us to achieve sufficient power (π=0.80) to obtain a meaningful training effect.
The data also revealed that prior internet experience was an important predictor of all three measures of performance. It is not surprising that the participants who had a greater amount of internet experience performed the tasks more accurately and efficiently and could navigate the website more effectively. These findings underscore the importance of basic internet skills to successful use of internet-based health applications. Today, even with the increased availability of the internet in the USA, it is not ubiquitous, and some populations such as those aged 65+, especially those who are less educated, minorities and of lower socioeconomic status, are less likely to have access or internet skills. Unless, these groups of consumers have ‘meaningful access’, they may eventually be disadvantaged in terms of access to medical information and services. Training on basic computer and internet skills may also improve the self-efficacy and comfort of older adults with respect to use of these applications. Confidence in one's ability to use technology and comfort using technology are important predictors of technology adoption.23
A unique feature of this study is the examination of cognitive abilities that are important to use of the website. Our results showed that people who had better reasoning, memory and general knowledge were more successful using the website and performing the tasks. This finding parallels our previous findings6 7 regarding the importance of cognitive abilities to the performance of internet-based health tasks. Our sample was non-cognitively impaired according to criteria for the MMSE. However, as is typical with most adult samples, there is still variability in cognitive abilities and the results clearly show that, even among non-impaired samples, cognitive abilities are important to use of health management websites. To enable people with lower cognitive abilities to successfully use e-health applications, efforts need to be directed toward the development of design interventions that help compensate for cognitive declines. This would include making the content of websites less technical and easier to comprehend, minimizing the demands on memory, and providing aiding tools such as search aids. Although the population of older adults is very heterogeneous, cognitive abilities such as memory and reasoning often decline with age, particularly in the later decades. In summary, age-related differences in cognitive abilities, in addition to technical skills, need to be taken into account in the design of these e-health applications.
Limitations of our study include the relatively small sample size, brevity of the training program and limited exposure of the participants to the website. Also, our sample was fairly well educated compared with the general older population and somewhat limited with respect to representation of ethnic/cultural minorities. Despite these limitations, the findings from our study underscore the importance of training and website design to the use of e-health applications.
Conclusions
e-Health applications are intended to provide consumers with easier access to health information and foster their engagement in health self-management. The benefits of these applications to broad populations of user groups will only be achieved if the abilities, needs, and preferences of diverse users are accounted for in the design of these applications. This requires having an understanding of the potential user population and involving them in the design process using a user-centered design approach. Unless these objectives are met, e-health applications will not achieve their intended goals, but can instead lead to increased healthcare disparities. As cogently noted by Atkinson and colleagues,40 poorly designed websites can be a challenge for all user groups but an even greater challenge for older adults who may be experiencing some functional declines, illness, or disabilities.
Supplementary Material
Acknowledgments
The authors would like to acknowledge Drs Christopher Hertzog and David Loewenstein for their assistance with the preparation of this manuscript.
Footnotes
Contributors: Conceived and designed the study: JC, JS, CCL, MAH, SNN. Performed the experiments: CCL, MAH. Analyzed the data: SNN, CCL, MAH, SHF, NA. Wrote the paper: SJC, JS, CCL, SNN. Conceived computational methods and interpreted the results: SJC, JS, SN, MH, CCL. Supervised the research: SJC, JS.
Funding: This study was supported by a grant from the National Institute on Ageing 5 P01-AG-017211-10.
Competing interests: None.
Ethics approval: This study was conducted with the approval of the Institutional Review Board of the University of Miami.
Provenance and peer review: Not commissioned, externally reviewed.
References
- 1. Fox S. Health Topics. Pew Internet & American Life Project, 2011. http://pewinternet.org/Reports/2011/HealthTopics.aspx [Google Scholar]
- 2. Fox S. Online Health Search. Pew Internet & American Life Project, 2006. http://pewinternet.org/Reports/2006/Online-Health-Search-2006.aspx [Google Scholar]
- 3. US Department of Health and Human Services Expanding the Reach and Impact of Consumer e-Health Tools. 2006. http://www.health.gov/communication/ehealth/ehealthTools/executivesummary.htm (accessed 24 Jul 2008). [Google Scholar]
- 4. Czaja SJ, Sharit J, Nair SN. Usability of the Medicare health website. JAMA 2008;300:790–2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Taha J, Sharit J, Czaja S. Use of and satisfaction with sources of health information among older Internet users and non-users. Gerontologist 2009;49:663–73 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Sharit J, Hernandez M, Czaja SJ, et al. Investigating the roles of knowledge and cognitive abilities in older adults information seeking on the web. ACM Trans Comput Hum Interact 2008;15:1–25 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Czaja SJ, Sharit J, Hernandez M, et al. Variability among older adults in Internet information seeking performance. Gerontechnology 2010;9:46–55 [Google Scholar]
- 8. Weir DR. The health and retirement study. Paper presented at the Cognitive Aging Summit. Washington, DC: The National Institutes of Health, 2007. [Google Scholar]
- 9. Center for Disease Control Improving and Extending Quality of Life Among Older Americans: At a Glance 2010. 2010. http://www.cdc.gov/chronicdisease/resources/publications/AAG/aging.htm (accessed 3 Dec 2010). [Google Scholar]
- 10. Administration on Aging. A profile of older Americans: 2010. 2010. http://www.aoa.gov/aoaroot/aging_statistics/Profile/2010/docs/2010profile.pdf (accessed 30 Nov 2010).
- 11. US Department of Health and Human Services Expanding the Reach and Impact of Consumer e-Health tools. 2006. http://www.health.gov/communication/ehealth/ehealthtools/pdf/ehealthreport.pdf [Google Scholar]
- 12. Or CK, Karsh BT, Severtson DJ, et al. Factors affecting home care patients' acceptance of a web-based interactive self-management technology. JAMIA 2011;18:51–9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Willis SL. Technology and learning in current and future older cohorts. In: Pew RW, Van Helmel SB, eds. Technology for Adaptive Aging. Washington, DC: National Academies Press, 2004 [PubMed] [Google Scholar]
- 14. Sweller J. Cognitive load theory, learning difficulty, and instructional design. Learn Instruct 1994;4:295–312 [Google Scholar]
- 15. Wickens CD, Lee JD, Liu Y, et al. An Introduction to Human Factors Engineers. 2nd edn Upper Saddle River, NY: Pearson Prentice Hall, 2004 [Google Scholar]
- 16. Baddely A. Working Memory. New York: Oxford University Press, 1986 [Google Scholar]
- 17. Mayer RE, Moreno R. Nine ways to reduce cognitive load in multimedia learning. Educ Psychol 2003;38:43–52 [Google Scholar]
- 18. Moreno R. Learning in high-tech and multimedia environments. Curr Dir Psychol Sci 2006;15:63–7 [Google Scholar]
- 19. Van Gerven PW, Paas F, Van Merrienboer JJ, et al. The efficiency of multimedia learning into old age. Br J Educ Tech 2003;73:489–505 [DOI] [PubMed] [Google Scholar]
- 20. Folstein MF, Folstein SE, McHugh PR. Mini-mental state: a practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res 1975;12:189–98 [DOI] [PubMed] [Google Scholar]
- 21. Radloff L. The CES-D scale: a self-report depression scale for research in the general population. Appl Psych Meas 1977;1:385–401 [Google Scholar]
- 22. HyperCam [Computer Program]. Murrysville, PA: Hyperionics Technology, 2006 [Google Scholar]
- 23.Snellen Eye Chart. AliMed Inc., 2010.
- 24. Earscan Acoustic Impedance Audiometer. Murphy, NC: Micro Audiometrics, 2000 [Google Scholar]
- 25. Czaja SJ, Charness N, Fisk AD, et al. Factors predicting the use of technology: findings from the center for research and Education on Aging and Technology Enhancement (CREATE). Psychol Aging 2006;21:333–52 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Baker DW, Williams MV, Parker RM, et al. Development of a brief test to measure functional health literacy. Patient Educ Couns 1999;38:33–42 [DOI] [PubMed] [Google Scholar]
- 27. Delis DC, Kramer JH, Kaplan E, et al. California Verbal Learning Test: Adult version. San Antonio, TX: The Psychological Corporation, 1987 [Google Scholar]
- 28. Hakstian AR, Cattell RB. Manual for the Comprehensive Ability Battery[CAB], 1982 Ed. Champaign, Il: Institute for Personality and Ability Testing, Inc, 1982 [Google Scholar]
- 29. Craik FI. A functional account of age differences in memory. InKlix F, Hagendorf H, eds. Human Memory and Cognitive Abilities. Amsterdam: Elsevier, 1986:409–22 [Google Scholar]
- 30. Reitan RM. Validity of the trail making test as an indication of organic brain damage. Percept Mot Skills 1958;9:271–6 [Google Scholar]
- 31. Ekstrom RB, French JW, Harman HH, et al. Manual for Kit of Factor-Referenced Cognitive Tests. Princeton, NJ: Educational Testing Services, 1976 [Google Scholar]
- 32. Wechsler D. Manual for Wechsler Memory Scaled Revised. New York: The Psychological Corp, 1981 [Google Scholar]
- 33. Wilkie FK, Eisdorfer C, Morgan R, et al. Cognitiion in early HIV infection. Arch Neurol 1990;46:433–40 [DOI] [PubMed] [Google Scholar]
- 34. Shipley WC. Shipley Institute of Living Scale. Los Angeles: Western Psychological Services, 1986 [Google Scholar]
- 35. Wechsler D. Wechsler Adult Intelligence Scale III. 3rd edn San Antonio, TX: The Psychological Corporation, 1997 [Google Scholar]
- 36. Czaja SJ, Charness N, Dijkstra K, et al. Computer and Technology Experience Questionnaire. Technical report # CREATE-2006-03. Miami, FL: CREATE/University of Miami, 2006 [Google Scholar]
- 37. Fisk AD, Rogers WA, Charness N, et al. Designing for Older Adults: Principles and Creative Human Factors Approach. 2nd edn Boca Raton, FL: CRC Press, 2010 [Google Scholar]
- 38. Kehoe EJ, Bednall TC, Yin L, et al. Training adult novices to use computers: effects of different types of illustrations. Comput Hum Behav 2009;25:275–83 [Google Scholar]
- 39. Bhowmick A, Khasawneh MT, Bowling SR, et al. Evaluation of alternate multimedia for web-based asynchronous learning. Internal Journal of Industrial Ergonomics 2007;37:615–29 [Google Scholar]
- 40. Atkinson NL, Massett HA, Mylks C, et al. Assessing the impact of user-centered research on a clinical trial eHealth tool via counterbalanced research design. J Am Med Inform Assn 2011;18:24–31 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.