Perils of providing visual health information overviews for consumers with low health literacy or high stress

Abstract

This pilot study explores the impact of a health topics overview (HTO) on reading comprehension. The HTO is generated automatically based on the presence of Unified Medical Language System terms. In a controlled setting, we presented health texts and posed 15 questions for each. We compared performance with and without the HTO. The answers were available in the text, but not always in the HTO. Our study (n=48) showed that consumers with low health literacy or high stress performed poorly when the HTO was available without linking directly to the answer. They performed better with direct links in the HTO or when the HTO was not available at all. Consumers with high health literacy or low stress performed better regardless of the availability of the HTO. Our data suggests that vulnerable consumers relied solely on the HTO when it was available and were misled when it did not provide the answer.

Introduction

Millions of people, both healthy and ill, look online for health information. Health-related websites are among the most popular on the internet, even though much of the information is for many too difficult to understand. Rewriting all information to be understandable is not a feasible solution, so other means of support need to be found. Current trends are toward assisting consumers with specific tools, such as symptom checkers, or personalizing information. Regardless of the solution chosen, it is important that its use improves laypersons' understanding since understanding affects healthcare-related decisions and even clinical visits.¹

The goal of this study was to evaluate, in a controlled setting, the effects of using a Health Topics Overview (HTO) on reading comprehension. The HTO is an automatically created and visualized table of contents. It minimizes the amount of navigation required and reduces the presence of irrelevant information. The two combined are expected to reduce cognitive load and so facilitate reading comprehension. Reducing the cognitive load is beneficial for consumers with limited cognitive abilities, consumers who are stressed, and consumers who have reduced memory capacity. Furthermore, the HTO can provide an easier to understand starting point for those with limited English skills. For example, knowing that a difficult, repeating term is a disease and not a drug reduces the overall complexity of the text. We also looked at two consumer characteristics commonly associated with increased vulnerability: low health literacy and high stress.

Case description

The HTO functions much as a table of contents (see figure 1). We developed the algorithms that build a HTO for a document, without human intervention, by matching noun phrases to medical concepts from the Unified Medical Language System (UMLS) and the Consumer Health Vocabulary.² By using concepts, phrases that are synonymous or refer to the same concept can be grouped. Moreover, each concept is matched to at least one semantic type, for example, ‘galactose’ is matched to ‘Carbohydrate’ and ‘Indicator, Reagent, or Diagnostic Aid’. We used these UMLS semantic types to group information into categories. For this study, we present the four categories that are most frequently searched for by consumers^3–5: ‘Body Parts’, ‘Diseases and Injuries’, ‘Drugs and Chemicals’, and ‘Medical Procedures’. A complete description and evaluation of the underlying algorithms can be found in Miller et al.⁶

Figure 1.

Health topics overview example.

In the HTO, concepts that appear more frequently in the text are represented by a more prominent link using a larger font size. Clicking on a category title in the HTO, for example, Drugs and Chemicals, highlights all the drugs and chemicals in the original text. Clicking instead on one concept in the HTO, for example, ‘galactose’, displays a list of text snippets for each occurrence of that concept and highlights them in the original text. In the HTO, the text snippets act as links: clicking one takes the user to its location within the original text. Users can highlight several concepts at once to get a broad overview of how the information is distributed in the text and highlighting can also be undone.

Methods

We evaluated the effect of the HTO's presence (independent variable) and of user characteristics (health literacy and stress) on reading comprehension as measured with a question answering task (dependent variable). Question answering tasks have been used for decades in schools to evaluate students and are also frequently used in computer and information science studies. Participants were recruited through convenience and snowball sampling in Southern California and received $15 for completing the study. Acquaintances, students, and staff members at the home institution of the authors were invited and often introduced new participants from their own circle of acquaintances. Each participant participated in both experimental conditions and received one text without HTO (baseline) and one with HTO (treatment). We conducted analyses of variance and include partial eta-squared (shown as η²), as provided in SPSS 16.0, to indicate the proportion of total variability due to a specific factor.

To avoid a learning effect from the first to the second text, we used two different texts (galactosemia and hypothyroidism)⁷ (available as an online data supplement) and changed the order of conditions to avoid confounding learning or order effects, leading to the following groupings: (1) HTO (hypothyroidism)+Baseline (galactosemia), (2) HTO (galactosemia)+Baseline (hypothyroidism), (3) Baseline (hypothyroidism)+HTO (galactosemia), (4) Baseline (galactosemia)+HTO (hypothyroidism). As participants signed up for the study, they were assigned to a grouping, for example, the first participant to grouping 1, the second to grouping 2, and so on.

Participants received a short demographic questionnaire including questions about color-blindness, prior knowledge about the topics, and previous education. We measured health literacy by administering the Rapid Estimate of Adult Literacy in Medicine (REALM), a commonly used metric of health literacy.⁸ We measured perceived stress with the Perceived Stress Scale-10 (PSS-10).⁹ Participants were then given an example text and question with HTO and explained that a correct answer received 1 point, no answer received 0 points, but for an incorrect answer 1 point would be deducted. This scoring method was chosen for its external validity; it represents the consequences of medical information retrieval. A correct answer represents correct knowledge and will lead to better decisions. An omitted answer represents the realization that knowledge is lacking and will lead to more information gathering. An incorrect answer represents false knowledge which may lead to unwise and costly decisions. This was also explained to the participants.

With each text, we measured reading comprehension with 15 multiple choice questions (four choices) in randomized order. The questions had to be answered within 20 min so that the entire study could be completed within 1 h to avoid tiring participants. The answers could all be found in the text. The questions were designed in such a manner that when the HTO was available, the answer to five questions could be found by following a prominent link in the HTO, the answer to five other questions could be found by following a normal link in the HTO, while there were also five questions which did not have a link to the answer. In the baseline condition, no HTO was present.

Results

Forty-eight adults participated, 19 (40%) were male and 29 (60%) were female. The average age was 31 with the youngest participant 22-years-old and the oldest 56-years-old. We verified that none of the participants was color blind, had any medical education or knowledge of the study topics. Most subjects had earned a graduate degree: 81% had completed some graduate school while 19% did not. More than half of the participants performed well on the REALM: 35% performed at ‘8th grade or below’ while 65% performed at ‘high school or above’ level. Half of the participants had low stress and half had high stress. Forty-six per cent of the participants were native English speakers, 54% were not.

Effects of the HTO

Table 1 shows an overview of all data discussed. We first evaluated the scores for the entire group (n=48) using repeated-measures analysis. The overall score for answering the 15 questions was very similar in both conditions: 11.8 points without HTO (baseline) and 11.5 points with the HTO.

Table 1.

Overview question-answering data

		N	Time* (n)	Question-answering score†
				Mean	SD	Min	Max
Baseline	All	48	15 (41)	12	2.8	6.5	15
	Grouped according to stress levels
	Low stress	24	14 (21)	12	2.9	6.5	15
	High stress	24	16 (20)	12	2.7	7.0	15
	Grouped according to literacy levels*
	Low literacy	17	15 (21)	11	2.7	6.5	15
	High literacy	31	15 (20)	13	2.6	7.0	15
HTO	All	48	17 (41)	12	3.2	4	15
	Grouped according to link prominence (5 questions per condition)**
	Without link	48	–	4	1.5	0	5
	Normal link	48	–	4	1.6	−1	5
	Prominent link	48	–	4	1.0	1	5
	Grouped according to stress levels*
	Low stress	24	18 (21)	13	2.6	4.5	15
	High stress	24	16 (20)	10	3.4	4	15
	Grouped according to literacy levels***
	Low literacy	17	16 (21)	9	3.1	4	15
	High literacy	31	18 (20)	13	2.0	8	15

Significance of mean question-answering scores: *p<0.05; **p<0.01; ***p<0.001.

^*Time in minutes, no time data for seven subjects, however, none of the 48 worked longer than 20 min.

^†One point deducted for an incorrect answer.

We then performed a second analysis taking the prominence of the links into account. We found that the prominence of the links in the HTO mattered (F(2,46)=7.566, p<0.01, η²=0.248) and explained 24.8% of the variability. For the five questions without a link, the average score was 3.5. Similarly, with a normal link, the average score was 3.6. However, with a prominent link leading to the answer, the average score was 4.3.

Effects of health literacy

To evaluate the effects of health literacy, we divided the participants in two groups using their REALM scores. The low health literacy group consisted of 17 individuals with REALM scores of 8th grade or below. The high health literacy group consisted of the 31 participants with REALM scores of high school level and above.

As expected, the low health literacy group scored worse than the high health literacy group in question answering. When the HTO was removed, the low health literacy group scored 10.6 points while the high health literacy group scored 12.5 points, a significant difference (F(1,47)=5.841, p<0.05, η²=0.113). When the HTO was made available to participants, the low health literacy group scored 8.7 points and the high health literacy group scored 13.0 points, again a significant difference (F(1,47)=33.138, p<0.001, η²=0.419).

In a follow-up analysis, we looked in detail at the scores with the HTO. The results show that link prominence in the HTO did not affect the scores of the high literacy group but affected the scores of the vulnerable, low literacy group (see figure 2). Participants with low health literacy scored significantly differently depending on the availability and prominence of links to the answers (F(2,16)=13.8, p<0.001, η²=0.464). The lowest score was 2.1 points for the five questions without a direct link to the answer. The group scored on average 2.6 points for the five questions with a normal link and 4.1 points for the five questions with a prominent link in the HTO.

Figure 2.

Question-answering scores with the health topics overview.

Effects of stress

To evaluate the effect of stress, we divided the group participants into two groups of 24 using their PSS-10 scores. The average PSS-10 stress score was 11 for the low stress group and 19 for the high stress group.

Without HTO, the low and high stress groups scored very similar, respectively 11.8 points and 11.9 points on the 15 questions. With the HTO, the high stress group scored lower with 10.4 points than the low stress group with 12.6 points, a significant difference (F(1,46)=6.067, p<0.05, η²=0.117).

Again we performed the follow-up analysis and looked in detail at the scores with the HTO. Link prominence in the HTO did not affect the scores of the low stress group but again affected the scores of the vulnerable, high stress group (see figure 2). Participants with high stress levels scores significantly different depending on the availability and prominence of links to the answers (F(2,46)=7.641, p<0.001, η²=0.249). The lowest score was 2.8 points for the five questions without a direct link to the answer. They scored 3.3 points for the five questions with a normal link in the HTO and 4.2 points for the five questions with a prominent link to the answer.

Discussion

Increasing consumer understanding of medical information can reduce healthcare costs by helping to avoid misunderstandings and unwise decisions. Many researchers are actively trying to find scalable solutions to accomplish this. This study evaluated the use of a health topics overview (HTO) for consumers with high or low health literacy and high or low stress levels. Vulnerable consumers, such as those with low health literacy or high stress levels, interacted differently with the HTO than the other consumers. When the HTO did not link directly to the answer, the vulnerable group scored poorly on the question-answering task even though the answer was available in the accompanying text. The vulnerable consumers relied on the HTO for this task. This led to very good performance when the necessary shortcuts were readily available but poor performance when they were not.

Peer-reviewed work typically emphasizes the advantages of information technology, while adverse effects are seldom published. Nevertheless, being aware of and understanding disadvantages is important. For example, personalization is usually seen as beneficial to consumers, but a recent study on risk predictors for Type-2 diabetes patients found that personalization led to a decrease in information processing by the consumers.¹⁰ Similarly, we found a mix of beneficial and detrimental effects of a HTO: it was helpful for vulnerable consumers when it linked directly to the required information; however it was a deterrent for this group when no direct link was available.

This study is limited since it used a question-answering task to approximate reading comprehension. In addition, it is also possible that not all participants fully understood the penalty scoring used, even though it was explained. Future work will include more additional evaluations, such as free recall or a reasoning task. Once the effects are better understood, careful presentation of information as part of an education program could be combined with longitudinal behavior observations.

User characteristics, such as health literacy and stress, influence users' interactions with information technology and their ability to understand and learn information. Detailed studies are needed to evaluate the different effects of information technology on users with different characteristics. For example, implicit trust in information technology may be a mediating factor in reading behaviors of vulnerable consumers. If these effects are also true for other tools such as symptom checkers or personalized health information, there may be large-scale, unintended consequences. Studies designed to expose weaknesses as well as strengths of these tools for users with different skill levels are needed. Furthermore, since technology can be used to persuade users to behave in a specific manner, future studies should evaluate and compare the use of information technology which limits information, for example, personalization in personal health records, with persuasive technology developed to encourage users to gain in-depth and complete information.