Testing Reliability and Validity of the eHealth Literacy Scale (eHEALS) for Older Adults Recruited Online

Abstract

Currently, vast amounts of health information and health management tools are available to the public online. To maximize the benefits of these eHealth technologies, it is important to assess the eHealth literacy of individuals. The eHealth Literacy Scale (eHEALS) has been used widely in the past several years, but mainly in younger populations. The purpose of this study was to test the psychometric aspects of the eHEALS for older adults using a secondary data analysis (N = 866, mean age, 62.8 ± 8.5 years). Reliability of the eHEALS was examined by calculating alpha coefficients and conducting test-retest procedures. Its validity was assessed using exploratory factor analysis and the hypothesis testing procedure. Findings demonstrated that eHEALS was internally consistent (α = .94) and stable (t [244] = −1.48, p = .140). The exploratory factor analysis yielded a single factor structure explaining 67.3% of the variance. The hypothesis testing also supported the validity of eHEALS. In recent years, there have been great efforts to use eHealth interventions to engage patients in health care and to help them manage their own health. Our study suggests that the eHEALS, a short screening tool for eHealth literacy, can be successfully used for older adults.

Introduction

As the population ages, increasing numbers of Americans are living with chronic illnesses and managing complex health information.¹ Recently, many individuals started using information technologies to manage their health.^2,3,4,5 In 2014, 87% of American adults are Internet users,⁶ and 59% of them searched for health information online.⁷ Although older generations are late adopters, they are rapidly becoming active Internet users (50 – 64 yrs, 88%; ≥65 yrs, 57%).⁸ Online health information can be particularly beneficial to older adult populations as they face significantly more health challenges than younger adults.

Currently, the vast amount of health information is available online to the public. To find and understand the information they need online, however, individuals must have the appropriate knowledge and skill sets to accomplish those tasks. The World Health Organization (WHO) defines health literacy as “the cognitive and social skills which determine the motivation and ability of individuals to gain access to, understand and use information in ways which promote and maintain good health.”⁴ Low health literacy is a significant challenge in health care globally, and improvement of health literacy is a public health goal for the 21st century.⁹

Traditional health literacy refers to an individuals’ ability using printed information.¹⁰ In this digital era, healthcare providers must be aware of their patients’ health literacy levels in cyber space to maximize the benefits of eHealth technologies. Older adults in particular are late adopters of the Internet, and their competency in using the Internet varies greatly. Despite an extensive amount of research on health literacy,¹¹ there has been a lack of studies on eHealth literacy. The knowledge and skills sets required for health literacy in eHealth need to be expanded, adding certain levels of computer competencies.¹² Based on their conceptual framework on eHealth literacy, Norman and Skinner¹³ developed the eHealth Literacy Scale (eHEALS).¹² The tool has been used a great deal in the past several years,^14–21 but mainly in younger populations.^14,15,17 The purpose of this study was to test the reliability and validity of the eHEALS for older adults using a secondary data analysis.

Literature Review

eHealth literacy was built upon the traditional definition of health literacy. This review of the literature briefly addresses concepts and measures in both areas.

Health Literacy

The Institute of Medicine (IOM)¹⁰ defined health literacy as the capacity of individuals to “obtain, process, and understand basic information and services needed to make appropriate decisions regarding their health.” A health-literate individual will be able to search, evaluate, comprehend and use the information that he or she has gathered. Prior studies have shown positive associations between individuals’ health literacy and self-efficacy for health behaviors and between individuals’ health literacy and online health information seeking.¹⁷ It is likely that individuals with low health literacy would obtain less health information and have a more difficult time understanding the information than those with high literacy. Patients with limited literacy tend to have lower efficacy for maintaining healthy behaviors, which could lead to increased incidence of chronic illnesses and health care costs.^22,23 The importance of health literacy has been highlighted as a public health goal in Healthy People 2020⁹ and there are several other federal policy initiatives to address low health literacy issues, including the Affordable Care Act of 2010²⁴ and the National Action Plan to Improve Health Literacy.²⁵

Assessment of an individual’s health literacy is important in health care delivery to ensure the benefits of interventions delivered. Two widely used “gold standard” measures for health literacy are the Test of Functional Health Literacy in Adults (TOFHLA)²⁶ and the Rapid Estimate of Adult Literacy in Medicine (REALM).^27,28 The Newest Vital Sign (NVS) is another recent measure of health literacy.²⁹ The TOFHLA, a commonly used instrument of health literacy evaluation, measures patients’ ability to read and understand things that they might commonly encounter in the healthcare setting.²⁶ Prior studies have reported the reliability of the TOFHLA (Cronbach’s α = .98), as well as its validity (WRAT-R, r = .74; REALM, r = .84).³⁰ The REALM is a short brief screening tool that assesses patients’ ability to read common medical words.^27,28 There has been sufficient evidence for the reliability of the tool as demonstrated by internal consistency (Cronbach’s α = .96) and stability (test-retest r = .99). Validity of the REALM was also reported by comparing its results with three established standardized reading recognition tests.³⁰ The NVS, which is more recent than other health literacy tools, assesses reading, comprehension, and numeracy skills using a nutrition label.²⁹ The NVS has shown to be reliable (Cronbach’s α = .76) and valid (moderate correlation with the TOFHLA, r = .59).³⁰

Finding and understanding appropriate health information online and using it to manage health adds additional attributes to health literacy, since the skill sets involve use of technologies. However, existing tools measuring health literacy do not incorporate these aspects.

eHealth Literacy

In IOM’s seminal report, “Health literacy, eHealth, and communication: Putting the consumer first: Workshop summary,” the IOM highlighted the importance of eHealth literacy and introduced the eHEALS developed by Norman and Skinner.¹⁰ Although there has been limited research, Norman and Skinner’s model of eHealth literacy (Lily model) and the eHealth Literacy Scale (eHEALS) are most frequently used in studies to evaluate eHealth literacy.^12,13 eHEALS was developed based on the Lily model.¹³ In this model, the researchers identified six core skills or literacies: (1) traditional literacy, (2) health literacy, (3) information literacy, (4) scientific literacy, (5) media literacy, and (6) computer literacy. Based on these core literacies, the 8 items of the measure assess consumers’ knowledge, comfort, and perceived skills at finding, evaluating, and applying electronic health information to health problems. The measure also includes 2 additional questions that ask about the participants’ perception of the Internet as a decision support tool and its usefulness to gather health information (not included in the total score). The eHEALS uses a 5-point Likert scale (1-strongly disagree, 5-strongly agree), and the score ranges from 8 to 40, with a higher score indicating higher literacy. (Table 1)

Table 1.

eHEALS and the Means of Item in Older Adults from Bone Health Study at baseline (N=866)

Items	Mean	SD+
(Q1) I know what health resources are available on the Internet	3.81	.923
(Q2) I know where to find helpful health resources on the Internet	3.90	.888
(Q3) I know how to find helpful health resources on the Internet	3.99	.847
(Q4) I know how to use the Internet to answer my health questions	4.01	.861
(Q5) I know how to use the health information I find on the Internet to help me	3.97	.818
(Q6) I have the skills I need to evaluate the health resources I find on the Internet	3.90	.871
(Q7) I can tell high quality from low quality health resources on the Internet	3.65	.968
(Q8) I feel confident in using information from the Internet to make health decisions	3.71	.923
Item means	3.87

⁺SD: Standard Deviation

Norman CD, Skinner HA, eHEALS: The eHealth Literacy Scale, J Med Internet Res 2006;8(4):e27, URL: http://www.jmir.org/2006/4/e27/, doi: 10.2196/jmir.8.4.e27. Items Reprinted with permission

The original psychometric aspects of the eHEALS was tested employing an adolescent sample (N = 664; mean age, 14.95 + 1.24). The calculated coefficient alpha was .88 and the correlation between a test-retest reliability was .68.¹² Validity of the tool was assessed using factor analysis that yielded a single factor solution (eigenvalue = 4.48, 56% of the variance explained).¹² In the past several years, the eHEALS has been used for adults in a variety of study settings.^31,32 In our previous study with patients recruited from an ambulatory clinic (N = 69; mean age, 45.5 + 13.30)³³ the calculated alpha coefficient was .93. The patients who spent more time on the web demonstrated higher scores on the eHEALS (r = .42, p = .03). Although the eHEALS has been frequently used in younger populations, psychometric aspects of the tool have rarely been tested in older adult populations.

Methods

Design

This was a secondary data analysis of data collected at baseline and 8-week follow-up in a large scale online bone health intervention study (Bone Power Study).³⁴ In the original study, older adult participants were randomized into either the control group or the intervention group that used the Bone Power program (learning modules, discussion boards, virtual libraries, and an Ask-the-Experts section) for 8 weeks. The data were collected using online surveys.

The reliability of the eHEALS was tested for internal consistency and stability. Validity was assessed using construct validity with exploratory factor analysis and hypothesis testing. The internal consistency and the validity of eHEALS were assessed using the baseline data of the original study. The test-retest reliability testing was conducted using a subset of the data, the no-intervention control group data (baseline and 8-week). Use of this subset was necessary for the test-retest reliability procedure as it requires data from two time points without an intervention that could possibly influence the scores of the eHEALS.

Sample and Setting

The original sample of the Bone Power Study included 866 older adults.³⁴ Individuals were eligible if they were age 50 or older, could read and write English and use the Internet/e-mail independently. Participants were recruited from two online communities, SeniorNet (seniornet.org), a leading older-adult online community, and My HealtheVet (myhealth.va.gov), a national web portal for veterans’ and Veterans Affairs employees’ health benefits and services (11/30/2010–7/9/2011). The study was approved by the Institutional Review Boards (IRBs) at the University of Maryland, Baltimore, and the Department of Veterans Affairs.

Data Analyses

The data were screened for missing values or any invalid data patterns (e.g., selection of “0”s for all questions). Descriptive statistics (mean, frequencies, percentages) were calculated for demographics, Internet experience, and computer knowledge. Item analyses were performed to assess the quality of the items (e.g., item means).³⁵

Reliability

Internal consistency of the eHEALS was assessed with Cronbach’s alpha coefficients for the overall scale as well as each item. An alpha of .7 or more was considered to be evidence of reliability.³⁵ Stability of the measure was tested using a paired t-test analysis with eHEALS scores of the control group measured at baseline and 8 weeks.

Validity

Validity was examined using exploratory factor analysis and hypothesis testing. Exploratory factor analysis was used to identify the underlying latent variable that can explain the variance in the measure. Maximum likelihood extraction method was applied and factors with eigenvalue of 1 or above were extracted as valid based on the Kaiser Criterion. Scree plot was used to help determine the number of factors to be retained.

Construct validity was assessed using a hypothesis testing approach. Based on prior studies,^13,16 it was hypothesized that participants who used the Internet more frequently would have higher eHealth literacy scores. In addition, participants with higher PC knowledge were expected to have higher eHealth literacy scores. The statistical software SPSS version 21 (IBM Corp, Armonk, NY) was used for data analyses.

Results

The total number of older adults from the Bone Power Study included in the analysis was 866. More than half of the participants were male (n = 549, 63.4%), with a mean age of 62.84 ± 8.52 years. The majority were white (n = 776, 89.6%) with some college or higher education (n = 755, 87.2%). The average duration of Web experience in the sample was 14.46 ± 6.61 years, and their average Web use per week was 15.4 ± 13.75 hours. (Table 2) The majority of the participants (n = 724, 83.6%) reported having a competent or higher level of computer knowledge. The average total score on the eHEALS was 30.94 ± 6.00 (range, 8–40). (Table 2) The mean of items in eHEALS was 3.87 (range, 1–5) with means ranging from 3.65 to 4.01. (Table 1)

Table 2.

Characteristics of Older Adults Recruited Online in Bone Health Study (N=866)

Characteristics	Categories	N(%)
Site	My HealtheVet	683(78.9)
	SeniorNet	183(21.1)
Age (Mean ± SD, yrs)		62.8 ± 8.5
Gender	Male	549 (63.4)
	Female	317 (36.6)
Race	White	776(89.6)
	Black/African American	64(7.4)
	Others+	26(3.0)
Education	Below college	111(12.8)
	College and above	755(87.2)
Income	Low Income	122(14.4)
	High Income	725(85.6)
Web experience (Mean ± SD, yrs)		14.46 ± 6.61
Web use (Mean ± SD, hrs/wk)		15.4 ± 13.75
PC Knowledge	Beginner	142(16.4)
	Above Competent	724(83.6)
eHEALS (Mean ± SD)		30.94 ± 6.00

⁺Others include American Indian/Alaska Native, Asian, and Native Hawaiian/Other Pacific Islander.

Reliability and Validity Testing

Finding from the study suggested that the eHEALS was internally consistent and stable. The calculated Cronbach’s alpha coefficient was .94, and the test-retest analysis showed no significant difference between the 2 time-points (t [244] = −1.48, p = .140) supporting the stability of eHEALS.

The study also supported validity of the eHEALS. For the factor analysis, significant findings from Bartlett’s test of sphericity (χ² [28] = 6381.03, p < .001) supported the factorability of the correlation matrix, and the high value of the Kaiser-Meyer-Olkin (KMO) test (.91) showed adequate sampling.³⁶ Using exploratory factor analysis, a single factor was retained based on an initial eigenvalue of 5.74 (Figure 1). Using this single factor model, the sum of squared loadings of the 8 items on the extracted factor based on maximum likelihood method was 5.39 accounting for 67.3% of the variance in the scale. Table 3 summarizes factor loadings of items.

Figure 1.

Scree Plot of eHEALS

Table 3.

Factor Loadings and Factor Score Coefficients for eHEALS and Single Extracted Factor at baseline (N=866)

eHEALS items	Factor Loadings a
(Q1) I know what health resources are available on the Internet	.827
(Q2) I know where to find helpful health resources on the Internet	.885
(Q3) I know how to find helpful health resources on the Internet	.910
(Q4) I know how to use the Internet to answer my health questions	.861
(Q5) I know how to use the health information I find on the Internet to help me	.834
(Q6) I have the skills I need to evaluate the health resources I find on the Internet	.763
(Q7) I can tell high quality from low quality health resources on the Internet	.719
(Q8) I feel confident in using information from the Internet to make health decisions	.746
Sums of Squared Loadings	5.387

Note. Extraction Method: Maximum Likelihood.

^a1 factor extracted. 4 iterations required

Findings from the hypothesis testing further supported the construct validity of the eHEALS as evidenced by significant correlations between eHealth literacy and the experience of Web use in years (r = .18, p < .001), and hours of Web use per week (r = .18, p < .001). Stronger correlations were found between the eHEALS and computer knowledge (r = .32, p < .001). The results suggest adequate conformity of eHEALS items to assess eHealth literacy.³⁵

Discussion

Based on 2013 national data, 85% of American adults have access to the Internet, and the penetration rates between White (86.0%) and Black (85.0%) are similar.⁶ In recent years, there has been a rapid growth in the use of eHealth programs to promote key principles of self-management, namely education, empowerment, and health communication.^3,5,37 This trend will continue with the national push for using Health IT. For example, Healthy People 2020, specifically highlights the importance of using eHealth technologies to promote the health of the public.⁹ The recent Health Information Technology for Economic and Clinical Health (HITECH) Act authorizes incentive payments to healthcare providers who use Electronic Health Records (EHRs) meaningfully (“Meaningful Use” [MU]).³⁸ Stage 2 MU focuses on health information exchange and engagement of patients in their own care. It requires that care providers allow their patients to access their EHRs and use eMessaging. To meet MU requirements,³⁹ hospitals nationwide are actively implementing new information systems that include “tethered personal health records,” (PHRs), which allows patients to access their EHRs and “patient portals” that incorporate several health tools like eMessaging and prescription refills.^40,41 Thus, assessment of eHealth literacy will become more important in effective delivery of health care in the U.S.

Prior studies have shown that eHEALS is a reliable and valid tool to assess eHealth literacy among younger adults, including teenagers (13 years and above).^{12,14,15,17,19–21} The findings from this study suggest that the eHEALS can also be used to assess eHealth literacy in older adults. This is particularly important as the number of older adults using the Internet to find health information is growing. eHEALS seems to be particularly applicable for older adults who are less familiar with technologies as it is a short (8 items) scale with easily understandable items. Although eHEALS is a short measure, it showed high internal consistency (α = .94) and there were no concerns for multicollinearity indicated (no bivariate correlations among items, r > .90).

Findings from the factor analysis showed significant correlation between eHealth literacy, use of the Internet, and computer knowledge indicating construct validity of eHEALS. Prior studies support the relationship between eHealth literacy, use of Internet, and computer knowledge.^13,16 The area of eHealth, however, is an ever evolving area with the emergence of new technologies. In a more recent article, Norman addresses the limitations of the original eHEALS in the context of Web 2.0.⁴² Psychometrics of the scale needs to be further investigated in the Web 2.0 environment.

In our study, all participants were online users with some college or higher education (87.2%), and this population may not representative of general community-dwelling older adults. Based on national data, only 57%–88% of adults age 50 and above are online users.⁷ The majority of the participants in this study were white (89.6%) and male (63.4%). Considering that 13% of U.S. population is Black, and slightly more whites (87.0%) are online users than blacks (80.0%), the racial mix of our sample seems to be similar to the U.S. population.⁴³ Generally, the gender distribution of online users is similar; however, our sample includes more males. This might have been due to one recruitment community (My HealtheVet) having more males than females.

Limitations

One limitation of the study is that the sample included in this study may not be representative of the majority of community-dwelling older adults. Older adults who opted to participate in the online study might have been more technology savvy and more interested in online health information than usual older adult online users. In addition, the majority of the participants were white with high levels of education. Further studies are needed using diverse samples with varying health conditions.

The original study specifically focused on bone health, although the content included general health promotion, including a balanced diet, exercise, and smoking and drinking cessation. eHealth literacy may vary depending on content and further studies are needed using samples with other disease conditions. In addition, eHealth interventions can use various technology infrastructures that can affect eHEALS scores. Considering the rapidly changing technology landscape, this is a complex issue in eHealth literacy and researchers must put forth more efforts to address this issue.

Due to the intervention performed in the original Bone Power study, the test-retest reliability procedures were conducted using a sub-set of the control group. The interval between the initial and subsequent measurements was 8 weeks, while usually the retest measurements are generally collected within 2 days to 2 weeks from the initial measurement.⁴⁴ In this study, results were not significant; however, there was a chance that participants could become more eHealth-literate due to the exposure to eHealth information.^45,46

Conclusion

In recent years, there have been great efforts to use eHealth interventions to engage patients in health care and to help them manage their own health. These trends are being accelerated by national policies that govern our nation’s use of eHealth (meaningful use). To optimize the benefits of eHealth programs for our patients, health care providers must use the intervention that allows the patient to access, understand and use information to the best of their ability. To accomplish this, eHealth literacy of those individuals must be assessed. The eHEALS is a short screening scale that can be used conveniently. The findings of the study suggest that eHEALS is a reliable and valid tool that can be used for older adults. Further studies are needed employing different samples in other settings.