Abstract
Background
Websites are common sources of health information to stroke survivors and caregivers for continual management of stroke and its long-term sequelae. The presence of risk factors and mortality rates related to stroke are high in Korean Americans. A vast majority of this group are active Web users and rely on the Web-based information due to lack of insurance and, thus, limited access to long-term stroke care. Thus, it is critical to evaluate existing stroke websites for their trustworthiness, readability, and suitability.
Objective
The objective of our study was to provide a systematic evaluation of stroke-related websites regarding (1) trustworthiness, (2) readability, and (3) suitability for stroke prevention and self-management for Korean Americans.
Methods
We selected a total of 156 websites using search terms “stroke,” “CVA,” “중풍 (jungpung),” and “뇌졸증 (noejoljung)” on Google and Yahoo. After eliminating duplicates and irrelevant websites (n=116), we evaluated a total of 42 websites (15 in English and 27 in Korean) using the National Library of Medicine’s health website’s evaluation tool for trustworthiness; Simple Measure of Gobbledygook for readability; and Suitability Assessment of Materials for suitability. All three instruments used the 3-point Likert scale: superior (=2), adequate (=1), or not suitable (=0).
Results
Of the 42 websites evaluated, we rated 62% (26/42) websites as “adequate” or above for trustworthiness. The information on 48% (20/42) websites had not been updated for more than a year, which indicates poor currency; 33% (14/42) websites failed to provide the publisher and contact information, which yields poor authority; 50% (21/42) websites did not cite sources of health information, which indicates lack of accuracy. Only 2 websites met the recommended readability (5th grade or lower reading level). The suitability was also suboptimal; only 1 website was rated as “superior”; 60% (25/42) websites were “adequate,” and 38% (16/42) were “not suitable.” Most websites were limited in graphical directions, interactive motivations for desired healthy behaviors, and multiple language translations.
Conclusions
The existing stroke-related websites in either English or Korean are trustworthy and suitable, yet precise citation of evidence-based information will improve trustworthiness. The contents requiring high reading level may set a barrier to the utilization of Web-based health information for Korean Americans with a lower level of education. In addition, supplementing graphical examples, interaction features, and culturally relevant information in multiple languages are the areas for improvement in suitability. The improved features can reduce the reading burden of stroke patients or caregivers and build their confidence when applying the information for stroke management in daily living. These strategies are especially crucial to Korean Americans, who inevitably seek Web-based information to fill the gap between their demand and access to health care for a long-term self-management after a stroke.
Keywords: stroke, website evaluation, trustworthiness, readability, suitability
Introduction
With advancements in the internet services and communication technologies, both release and seeking of health information through the Web have been exponentially growing. In the United States, 89% of adults use the internet, and among them, 72% lookup online for health information [1,2]. Use of mobile phones has made the access to health information ubiquitous. Recent surveys have reported that 77% of US adults own a mobile phone, and among them, 62% have used their phone to get information about a health condition [3,4].
Use of Web-based health information is common among Korean Americans [5-8], who constitute about 1.8 million and are the fifth largest Asian American population in the United States [9]. One possible reason for this population using websites as an important resource for health information may be the limited access to health care professionals due to lack of health insurance [10]. Korean Americans record one of the lowest rates of health insurance coverage among all racial and ethnic groups living in the United States [11-13]. The limited English proficiency, which is predominant in first-generation (75.5%) and monolingual (37%) immigrants, has also contributed to limited health care access among this group [14,15]. Limited health care access increases Korean Americans’ sense of self-responsibility and may lead individuals to be more active in seeking health information through available sources, such as Web-based information [10].
Universally, individuals with chronic or stigmatized diseases such as stroke are more likely to search for health information on the internet than those without the health conditions [16-19]. Stroke is the foremost cause of serious long-term disability, with high health care cost [20], and puts an increasing economic burden on health care resources [21]. The varying degrees of long-standing disability, as a result of stroke, lead to patients and their caregivers living with daunting long-term management. Stroke often results in emotional, cognitive, and physical impairments, which tend to be visible to others and cause stigmatizing social experiences after stroke [22,23]. With this burden, stroke survivors and caregivers often use websites as resources of information that they need [24].
Stroke is the third leading cause of death among Korean Americans [25]. The proportionate mortality ratios of stroke, especially hemorrhagic stroke, have been reported to be higher among Korean American women (2.07) and men (1.89) than among non-Hispanic white women (1.06) and men (0.94) [26]. Furthermore, the elderly have a higher prevalence of stroke risk factors, such as hypertension and dyslipidemia, than Caucasians, and the former lack knowledge about stroke [27]. Thus, prevention and appropriate long-term management of stroke and the sequelae with quality information is critical in this population.
Obtaining Web-based health information can be beneficial for or harmful to people. The explosion and proliferation of health information available online are on the promise that these online resources can confirm or broaden patients or families’ understanding of diseases and treatment opinions that influence health care decisions and empower them to effectively self-manage health conditions [28,29]. However, the quality of the Web-based health information, generally in trustworthiness, readability, and suitability aspects, is often questioned. Online health information seekers could be at risk of finding unreliable or inappropriate information from the websites. Incorrect or inappropriate health information can be used in an improper way and can cause detrimental outcomes by negatively influencing health care-related decisions [30,31]. As patients have a wide range of health information literacy, some patients are unable to critically assess health information or might misinterpret it [31]. Online health information can be advantageous only if it is understandable to the consumers [29].
Considering the high demand of Web-based health information in native or English language among Korean Americans, who have limited English proficiency, it is critical for the websites to be trustworthy with up-to-date, reliable, and accurate information, which is readable and suitable for this specific population. Thus, it is important to examine the trustworthiness, readability, and suitability of health information available in native and English languages on the websites. The study findings will facilitate devising strategies to better design and implement stroke-related websites.
The purpose of this study was to address the following question: Is the Web-based stroke information adequate for Korean Americans to read, understand, and engage in stroke care? Specifically, we aimed to evaluate stroke-related websites in terms of their (1) trustworthiness, (2) readability, and (3) suitability for stroke prevention and self-management in this group.
Methods
Selection of the Websites for Stroke
Initially, we considered a question that how Korean Americans would seek stroke information on the internet. One of the common behaviors exhibited was bilingual searches in their health information seeking as a technique for coping with limited English proficiency [32]. In general, people selected health websites within only the first one or two pages of search results (10 results per page) when searching the internet; about half of the internet users entered a single query [32,33]. In van Deursen and van Dijk’s [33] study, nobody used advanced search features (eg, Boolean operators like AND, OR, and NOT). In fact, Google and Yahoo have been the most popular search engines among Koreans and Americans [2,32].
Based on a review of these studies about online health information-seeking behaviors of Korean Americans, we searched stroke-related websites in Google and Yahoo search engines using the following terms: “Stroke,” “CVA,” “중 (jungpung),” and “뇌졸증 (noejoljung).” The term, “Stroke” is health care consumers’ preferred term listed in the Consumer Health Vocabulary Initiative [34]. The term, “CVA’” is another representative term for stroke. Furthermore, “중풍” and “뇌졸증” are the most commonly used Korean terms to indicate “stroke.” We retrieved a total of 156 stroke-related websites that appeared on the 1st and 2nd pages only after searching. With the use of terms “Stroke” and “CVA,” Google revealed 45 sites, while Yahoo led to 35 sites. Searching with the use of terms “중풍” and “뇌졸증” resulted in 36 sites from Google and 40 sites from Yahoo. Then, irrelevant websites, which were newsletters, commercial and noncommercial advertisements, and homonyms such as stroke motion in swimming or tennis, were excluded. After duplicates and the irrelevant sites (n=116) were eliminated, a total of 42 websites (15 in English and 27 in Korean), including those of the not-for-profit organizations, clinics, magazines, and blogs, were selected.
Evaluation Tools
Trustworthiness
We initially evaluated each website for its trustworthiness using the following three criteria for health websites’ evaluation endorsed by the National Library of Medicine (NLM) [35]: currency or timeliness (when was the website last updated?); authority (who published the website?); and accuracy (are the sources cited reliable?). Each criterion was rated on a 3-point Likert scale, where 0=not suitable, 1=adequate, and 2=superior. The descriptions for each point are presented in Table 1. Thus, the possible total score per site ranged from 0 to 6, with higher scores indicating better trustworthiness.
Table 1.
Trustworthiness of stroke websites by the National Library of Medicine criteria.
| Evaluation criteriaa | Frequency, n (%) |
ICCb | ||
| Currency or timeliness: when was the website last updated? |
|
.969 | ||
|
|
Superior (2) | The published date of the information is indicated and is less than 1 year old (last year) | 13 (31) |
|
|
|
Adequate (1) | The published date of information is indicated, but is older than 1 year | 20 (48) | |
|
|
Not suitable (0) | No indication | 9 (21) | |
| Authority: who published the website? |
|
.980 | ||
|
|
Superior (2) | The publisher’s information (individuals or organizations) and contact information can be easily found | 28 (67) |
|
|
|
Adequate (1) | The publisher’s information (individuals or organizations) can be found. But there is no contact information | 5 (12) | |
|
|
Not suitable (0) | No indication | 9 (21) | |
| Accuracy: are the sources cited reliable? |
|
.987 | ||
|
|
Superior (2) | The information is drawn based on sound medical research, and the information sources are cited | 14 (33) |
|
|
|
Adequate (1) | The information sources are cited but not based on medical research | 7 (17) | |
|
|
Not suitable (0) | No indication | 21 (50) | |
aScore shown in parentheses.
bICC: intraclass correlation coefficient.
Readability
We evaluated the readability of the information on selected websites using the Simple Measure of Gobbledygook (SMOG) Readability Test and the Reading Grade level of the Suitability Assessment of Materials (SAM) instrument. The SMOG has been validated as very easy to compute and provide a reasonably accurate measure of readability when evaluating consumer-oriented health care materials in many studies. We used the English version of the SMOG measure for evaluating the websites published in English [36] and the Korean version [37] for the websites in Korean. The formula requires counting 10 consecutive sentences at the beginning, middle, and end of website pages (30 total sentences). Second, it requires counting the number of words with ≥3 syllables in the 30-sentence sample. Then, with the total number of polysyllabic words counted, the grade level is determined using the SMOG conversion table. When using the SAM, the reading grade level can be measured by using different reading formulas, including the SMOG. Studies on the readability of patient education materials have often used both the SMOG and SAM tools together. For example, Rosenfeld et al [38] and Shieh and Hosei [39] computed the SMOG scores and integrated the scores to SAM 3-point categories. Similarly, in this study, the grade levels computed by the SMOG formula were converted to the SAM’s 3-point scale, where 0=not suitable (9th-grade level and above), 1=adequate (6th-, 7th-, or 8th-grade level), and 2=superior (5th-grade level or lower).
Suitability
Each website was evaluated by the SAM instrument [40]. The SAM has been used in many studies to evaluate the suitability of online health information [41-46]. It consists of 22 items grouped under 6 factors, namely (1) Content, (2) Literacy Demands, (3) Graphics, (4) Layout and Typography, (5) Learning Stimulation and Motivation, and (6) Cultural Appropriateness. We added one item to the list, which was “multiple language translations,’’ under the factor of Cultural Appropriateness. Each of the 23 items was rated on an ordinal scale, where 0=not suitable, 1=adequate, and 2=superior. Raw scores were summed to yield an overall score. This overall score was then converted to a percent of the possible total score for each website using the following formula: converted percent=total score/total possible score (46=23 items×2 maximum score per item)×100. A converted percent of 70%-100% indicates a superior website, 40%-69% indicates an adequate website, and 0%-39% indicates a not suitable website [40].
When previous studies assessed the suitability of health information across multiple sources, the converted percent was also applied to the present the quality of the health information per SAM criterion [47,48]. In this study examining which criterion was suitably met across the websites, the mean score per criterion was computed and then converted to a percent of the possible best score using the following formula: the converted percent=([n of websites×0 point]+[n of websites×1 point]+[n of websites×2 points])/Total possible best score (84=42 websites×2 points)×100.
Then, if the converted percent for a criterion was in the range of 70%-100%, the quality of the information regarding the criterion was considered as “superior” across websites. If the percentage was in the range of 40%-69%, the quality of the information per criterion was considered at an “adequate” level. If the percentage was in the range of 0%-39%, the information regarding a criterion was considered as “not suitable” across the websites.
Evaluation and Analyses
Three content experts evaluated each website independently in the first round and collected the ratings. In the second round, the three raters reviewed the websites synchronously to ensure the accuracy of understanding of each criterion and, then, shared their rationales for the ratings. The frequencies and intraclass correlation coefficients (ICCs) per the evaluation criteria were obtained using SPSS 24. The final interrater agreement levels for the three evaluation criteria were as follows: NLM, ICC=.969-.987; SMOG, ICC=.810; and SAM, ICC=.626-.994.
Results
General Characteristics
Out of the 42 appraised websites, 15 stroke websites were in English and 27 were in Korean. Of all, 12 websites were those of the not-for-profit organizations, 7 were blogs on stroke, 5 were published by clinics or hospitals, 5 were Wikipedia pages, 4 were medical magazine sites, 3 were general magazine sites, 2 were postings on broadcasting websites, 1 was a pharmacological company website, 1 was an insurance company website; 1 was a general online forum site, and 1 was a medical information website.
Trustworthiness
Overall, 62% (26/42) websites received a rating of >3.0, which presents “adequate” level. Regarding the currency or timeline, 79% (33/42) of the selected websites indicated published date of the information on their websites, whereas 21% (9/42) did not indicate a published date of the information. The updated dates of the information on 31% (13/42) websites were <1 year old, while on 48% (20/42) websites these were older than 1 year. In terms of authority, 21% (9/42) websites did not present the publisher or contact information. Regarding accuracy, 33% (14/42) websites posted health information based on medical research evidence along with citations, whereas 50% (21/42) of the websites did not cite any source of information (Table 1).
Readability
A total of 19 of 42 (45%) websites were presented at the 9th-grade level or above (0=not suitable). Twenty-one (50%) websites were presented at reading levels between 6th and 8th grade (1=adequate). Only 2 websites (5%) were presented at the 5th grade- or lower level (2=superior).
Suitability
Table 2 presents the frequencies per rating score, ICC of the three raters per the SAM criterion, and the converted suitability percent score. The overall converted suitability percent score of all 42 websites was 55.7%, which represents “adequate” suitability. Out of 42 websites, only 1 website of the American Stroke Association was rated as “superior”; 60% (25/42) websites were “adequate,” and 38% (16/42) websites were “not suitable.” The websites were superior in the quality of layout and typography (85.3%). Furthermore, the websites were at an adequate level regarding the Content (65.5%); Literacy Demands (68.6%); Graphic Illustration, Lists, Tables, and Charts (42.4%); and Learning Stimulation and Motivation (47.2%), even though there were some individual items with inadequate suitability. However, Cultural Appropriateness was not suitable (22.2%).
Table 2.
The evaluation of the stroke websites based on Suitability Assessment of Materials.
| Category and criteria | Frequency, n (%) | ICCa | Converted %b | |||
|
|
|
Not suitable (0) | Adequate (1) | Superior (2) |
|
|
| Content |
|
|
|
|
|
|
|
|
Purpose | 0 (0) | 8 (19) | 34 (81) | .626 | 90.5 |
| Scope | 2 (5) | 13 (31) | 27 (64) | .896 | 79.8 | |
| Content topics | 6 (14) | 25 (60) | 11 (26) | .875 | 56.0 | |
| Summary and review | 23 (55) | 8 (19) | 11 (26) | .993 | 35.7c | |
| Average |
|
|
|
|
65.5 | |
| Literacy demands |
|
|
|
|
|
|
|
|
Reading grade level | 19 (45) | 21 (50) | 2 (5) | .810 | 29.8c |
| Writing Style | 1 (2) | 14 (33) | 27 (64) | .859 | 81.0 | |
| Sentence construction | 1 (2) | 17 (41) | 24 (57) | .727 | 77.4 | |
| Vocabulary | 2 (5) | 19 (45) | 21 (50) | .746 | 72.6 | |
| Learning enhancement by advance organizers | 6 (14) | 3 (7) | 33 (79) | .962 | 82.1 | |
| Average |
|
|
|
|
68.6 | |
| Graphic illustration, lists, tables, and charts |
|
|
|
|
||
|
|
Cover graphic | 20 (48) | 13 (31) | 9 (21) | .950 | 36.9c |
| Type of illustrations | 14 (33) | 15 (36) | 13 (31) | .950 | 48.8 | |
| Relevance of illustrations | 15 (36) | 11 (26) | 16 (38) | .975 | 51.2 | |
| Graphical direction: lists, tables, charts, and forms | 25 (60) | 10 (24) | 7 (17) | .956 | 28.6c | |
| Captions are used to announce or explain graphics | 14 (33) | 17 (41) | 11 (26) | .976 | 46.4 | |
| Average |
|
|
|
|
42.4 | |
| Layout and typography |
|
|
|
|
|
|
|
|
Typography | 1 (2) | 9 (21) | 32 (76) | .863 | 86.9 |
| Layout | 1 (2) | 12 (29) | 29 (69) | .813 | 83.3 | |
| Subheadings and chunking | 4 (10) | 4 (10) | 34 (81) | .965 | 85.7 | |
| Average |
|
|
|
|
85.3 | |
| Learning, stimulation, and motivation |
|
|
|
|
||
|
|
Interaction included in the text or graphics | 16 (38) | 15 (36) | 11 (26) | .967 | 44.0 |
| Desired behavior patterns are modeled | 14 (33) | 12 (29) | 16 (38) | .975 | 52.4 | |
| Motivation | 16 (38) | 14 (33) | 12 (29) | .965 | 45.2 | |
| Average |
|
|
|
|
47.2 | |
| Cultural appropriateness |
|
|
|
|
|
|
|
|
Cultural match: logic, language, experience | 24 (57) | 8 (19) | 10 (24) | .992 | 33.3c |
| Cultural image and examples | 29 (69) | 10 (24) | 3 (7) | .987 | 19.0c | |
| Multiple languages translation | 36 (86) | 0 (0) | 6 (14) | .994 | 14.3c | |
| Average |
|
|
|
|
22.2c | |
| Overall average | 55.7 | |||||
aICC: intraclass correlation coefficient.
bConverted Percent=([n of websites×0 point]+[n of websites×1 point]+[n of websites×2 points])/total possible best score (84=42 websites×2 points)×100. Interpretation: 70%-100%=superior; 40%-69%=adequate; 0%-39% =not suitable.
cInformation rated as “not suitable” per criterion across the evaluated websites.
Under the Content category, the quality of purpose and scope was superior, with scores of 90.5% and 79.8%, respectively. Regarding the extent of the content topics, 26% (11/42) websites aimed at desirable behavior rather than at nonbehavioral facts. In addition, 60% (25/42) websites showed <40% of the content topics focusing on desirable behaviors or actions. However, 6 websites did not present such contents. Many of the appraised websites (23/42, 55%) did not present summaries or reviews well to convey key messages.
Regarding the Literacy Demands, the selected websites were suitable in writing style (81.0%), sentence construction (77.4%), vocabulary use (72.6%), and organization using road signs (82.1%). However, the reading grade level was not suitable across websites (29.8%).
The overall suitability of the Graphic Illustration, Lists, Tables, and Charts category was rated as “adequate” (42.4%). In this category, the type of illustrations (48.8%), relevance of illustrations (51.2%), and captions (46.4%) were “adequate” at the lower levels, but cover graphic (36.9%) and graphical direction (28.6%) were “not suitable.” In more detail, 48% (20/42) of the websites did not present cover graphic to attract attention and to clearly portray the purpose of the website.
In addition, 60% (25/42) of the websites presented graphics without explanation and 24% (10/42) showed too brief “how to” directions with graphics for readers; only 17% (7/42) websites provided step-by-step directions with an example that will build self-efficacy (confidence). The layouts and typography were superiorly presented in most appraised websites, with the converted suitability scores of 83.3%-86.9%.
In the Learning Stimulation and Motivation category, the suitability of each criterion was rated as “adequate,” even though the suitability percent scores were not high, ranging 44.0%-52.4%. In more details, 38% (16/42) websites did not provide interaction learning or stimulation and 36% (15/42) used passive interactions with the Questions & Answer format. Regarding the desired behavior pattern presentation, 38% (16/42) websites demonstrated instructions for specific behavior and skills by using specific, familiar instances with the rating of “superior.” In addition, 29% (12/42) websites were rated as “adequate” as they provided the information in a mix of technical and common language that the reader may not easily link with daily living activities. In terms of Motivation, only 29% (12/42) websites reached the “superior” level, which means that complex topics are subdivided so that readers may experience small successes in understanding or problem solving, leading to self-efficacy (confidence). Furthermore, 38% (16/42) websites did not have features of motivation.
The Cultural Appropriateness across websites was “not suitable”; the converted suitability percent scores ranged from 14.3% to 33.3%. Regarding cultural match, 57% (24/42) websites did not present the information in a culturally similar logic, language, and experience of the target population culture. The cultural image and examples were rarely shown in 69% (29/42) websites. Moreover, 86% (36/42) of the websites were presented in only one language and could not be translated into other languages.
Discussion
Principal Findings
This study is the first to provide a systematic evaluation of the stroke-related websites considering Korean Americans. The findings of this study present valuable information about the trustworthiness, readability, and suitability of stroke-related English and Korean websites, which were trustworthy and marginally suitable, but not easily readable. In addition, areas that need improvement were identified in each criterion. This information can be utilized by clinicians and researchers in improving or designing stroke websites as well as by people who seek stroke information in their self-assessment of the quality of the information on websites.
Trustworthiness
Patients and caregivers look up the websites as they are easy to access, not expensive, and, in general, believed to be timely [32]. Patients are often unsure about which websites to trust and are concerned about accessing potentially misleading or illegitimate health information [49,50]. The outdated, inaccurate, and unreliable Web-based information can mislead the public in their understanding of stroke and stroke management. In the limited chance of education on which website and its information are trustworthy, Korean Americans consider merely the repetition of specific information as a criterion for evaluating the trustworthiness of websites; if the same information appears in several locations on the internet, Korean Americans are likely to simply believe that the information or the website is reliable [32]. The trustworthiness of the website information could be easily checked by the NLM’s [35] three criteria of health website evaluations. These evaluation criteria need to be educated to online health information seekers, including the population in question.
In this study, the information on 20 websites was published more than 1 year ago and 9 websites did not indicate a published date of the information. Notably, half of the websites did not cite any source of information on their websites. Agarwal et al [30] also found in their evaluation of educational resources in 3 stroke-related websites that users criticized the lack of citations and references for the material and facts provided on all 3 websites. It is important to present the valid and reliable primary information with the published date and the information sources for health consumers to help them with right decision making regarding their health issues [16,30]. It is recommended to scrutinize or peer review before presenting the information on the website. Health care providers should be aware of, at least, these three criteria when providing health care information to patients and caregivers via papers or websites. Furthermore, health care providers should provide opportunities to educate patients and caregivers on how to check and whether to rely on health care information in any type of Web information platform.
Readability
Reading level is an essential component of health literacy and is included as one of the Literacy Demand criteria in the SAM tool. Concerned about health literacy, the Joint Commission [51] has recommended patient education materials to be written at or below the 5th-grade level and has developed an action plan to promote patient literacy. In this study, 19 out of 42 websites were written at the 9th-grade level or above, which is rated as “not suitable” and categorized as “difficult” in the US readability categories [52]. Only 2 websites were written at the 5th grade- or lower level. This result is consistent with those of previous studies, which evaluated some stroke-related websites: the mean readability level of stroke information was found to be 10th grade by Griffin et al [53] in their evaluation of 30 stroke educational websites; these websites were different from the set of websites appraised in our study, except 3 overlapping websites. Furthermore, Sharma et al [24] found that over half of 100 stroke websites were produced at the 12th-grade level or above when the readability was measured with the SMOG. The high reading levels may be due to complex medical terminologies without explanations or translation into layperson’s terms.
The health information is only useful if the consumer can comprehend the presented information. The consequences of not understanding health information can negatively affect both a person’s health and his or her utilization of health care services. The use of consumer health vocabulary may help lower the reading grade level on both English and Korean stroke websites. In the United States, the Open-Access, Collaborative Consumer Health Vocabulary provides 156,826 consumer-friendly health phrases and synonyms for professional clinical terms [34,54]. In South Korea, a consumer vocabulary system for health information was developed in a study [55]. Also, the complex medical terms or sentences could be provided with additional understandable explanations so that readers become familiar with them and can use them appropriately [38]. Korean Americans’ limited English proficiency should also be taken into consideration. Limited English proficiency was identified as a critical source of health vulnerability; it impacted health perceptions, led to higher health risk, and negatively influenced health care utilization [56-58]. Those with limited English proficiency scored lower on all measures of health conditions than their English-proficient counterparts [57]. Considering the limited English proficiency among Korean Americans, terms and sentence structures of stroke care information on English websites should be used at an appropriate reading level.
Suitability
The overall suitability of all 42 websites was rated as “adequate” (55.7%); however, there are needs for improvement in many areas. Regarding the Content, the majority of the websites explicitly showed the purpose and the scope of the content. However, they did not present summaries or reviews well to convey key messages. In order to ensure the delivery of important messages and to assist the readers’ comprehension, it is necessary to retell key messages in different words or examples and provide their summaries.
The overall suitability of Graphic Illustration was marginally adequate (42.4%). Half of the websites did not present cover graphic to attract attention and clearly portray the purpose of the websites. In addition, 60% (25/42) of the websites presented graphics without explanation and 24% (10/42) showed too brief “how to” directions for readers. This finding is similar to the lack of graphics and other nontextual media covering patient education materials in Agarwal et al’s [30] evaluation of educational resources in 3 stroke-related websites. Images and videos combined with the text can act as supplements to difficult topics covered on the site [30] and can increase the ability of the user to understand and retain the material [59]. Since the medical information is relatively unfamiliar to laypersons, and the reading level is high, step-by-step directions with graphical examples can reduce the reading burden of stroke patients and caregivers. Furthermore, use of strategies to enhance readers’ understanding of the information on websites is especially important to Korean Americans, who inevitably seek online information due to lack of insurance and limited English proficiency.
The Learning Stimulation and Motivation category also received the rating of “adequate,” but it was not high (47.2%). No interaction learning or stimulation was provided on 38% (16/42) websites. Regarding the desired behavior patterns presentation, the criterion of the “superior” rating probed if websites demonstrated instructions for specific behavior and skills by using specific, familiar instances; more than half of the websites failed to reach this level. Only 29% (12/42) websites ranked “superior” in the Motivation criterion. Mere access to Web-based health information does not necessarily empower consumers and patients. Providing actionable health information with accuracy is the top vision in the National Action Plan to Improve Health Literacy published by the US Department of Health and Human Services [60]. In addition, practical guidelines demonstrating specific behavior and skills applicable to daily living will motivate the readers to experience small success in understanding or problem solving, leading to self-efficacy. Multimedia and interactivity features can assist information users to build more confidence by tailoring to individual needs and progress of condition management and rehabilitation. The incorporation of self-assessments of risks or symptom changes and modeling of desirable health behaviors into the website will promote self-efficacy in learning. Miller and Leroy [54] suggested that it would be ideal if a consumer would go to a website, answer a few questions, and in a few seconds, a document written specifically for their needs and appropriate to their reading skill would appear in their Web browser. The features of dynamic discussions and communication will lead to co-learning with each other [28,61]. Chat rooms and email services with health care providers on websites are other recommendable features to satisfy patients’ information needs.
Of all, 86% (36/42) websites were presented in only one language and could not be translated into other languages. More than half of the websites in English or Korean did not present information matched to the Korean culture in texts, images, or examples. The cultural factors influence health information acquisition and access to social support for ethnic minorities [62]. Korean consumers rely heavily on Korean-specific health information, regardless of education levels [32]. Korean Americans have a lower level of knowledge on stroke risk factors and symptoms than the general US population [63]. Websites are important sources and channels of health information for them. If websites provide more diverse and culturally appropriate information in both Korean and English, the health information can be delivered to a broader population of Korean Americans. Then, it may contribute to increase in knowledge about stroke prevention and self-management among this group.
Limitations
We evaluated the websites that appeared on the first and second pages only on Google and Yahoo browsers. Thus, the results cannot be generalized to reflect the features of all stroke-related websites. However, these websites can be the most popular or important ones because studies have found that 73%-95% of Web users never view past the first search engine results page and have emphasized the value of the first page [64-67]. In addition, it may indicate the increasing ability of Web search engines to retrieve the relevant Web-based information more effectively [64]. The websites evaluated in this study were stroke specific, which may limit the application of this study results to websites of other specific health conditions.
Conclusions
The quality of health information for patients with stroke is vital to ensure the good recovery and to improve the quality of life. Easily obtainable, user-friendly, accurate, and reliable online resources could help people make appropriate decisions about how best to maintain or improve their health condition.
We found that the existing stroke-related websites in either English or Korean are trustworthy and suitable. However, it was notable that the accuracy of information on these websites was low due to lack of citations and references. Providing accurate, reliable, high-quality, and evidence-based information is a critical responsibility of health professionals. In addition, the study findings signified the need for diverse features to reduce the high reading level of the information on the websites. Graphical examples, multimedia, and interactive features can reduce the reading burden of stroke patients and caregivers, as well as build more confidence or self-efficacy when applying the information for condition management and rehabilitation in daily living. Rather than posting the information in a nonspecific way, practical guidelines demonstrating specific behavior and skills applicable to daily living will motivate the reader to experience small success in stroke self-management, leading to self-efficacy.
These strategies to enhance readers’ understanding of Web-based information are especially important to Korean Americans, who inevitably seek online information due to lack of familiarity with the US health care system, inadequate health insurance coverage, and language barriers. Culturally sensitive, high-quality health information relevant to this group through websites may have great potential to increase the knowledge of stroke self-management and make significant contributions to promote the health of Korean American stroke survivors. In the future, the evaluation of stroke websites by Korean American laypersons using the evaluation tools used in this study might be an educational health literacy intervention to learn how to evaluate and consider the design and content aspects in the websites. The examination of how this population navigates stroke websites and apply the health information obtained from these websites for their daily health behaviors should be considered for future studies.
Acknowledgments
The processing of this manuscript was supported by Texas Woman’s University Libraries’ Open Access Fund.
Abbreviations
- CVA
cardiovascular accident
- ICC
intraclass correlation coefficient
- NLM
National Library of Medicine
- SAM
Suitability Assessment of Materials
- SMOG
Simple Measure of Gobbledygook
Footnotes
Conflicts of Interest: None declared.
References
- 1.Pew Research Center. 2018. Feb 05, [2018-03-15]. Internet/Broadband Fact Sheet http://www.pewinternet.org/fact-sheet/internet-broadband/
- 2.Fox S, Duggan M. Pew Research Center. 2013. Jan 15, [2018-03-15]. Health Online 2013 http://www.pewinternet.org/files/old-media/Files/Reports/PIP_HealthOnline.pdf .
- 3.Pew Research Center. 2018. Feb 05, [2018-03-15]. Mobile Fact Sheet http://www.pewinternet.org/fact-sheet/mobile/
- 4.Smith A. Pew Research Center. 2015. Apr 01, [2018-03-15]. US Smartphone Use in 2015 http://www.pewinternet.org/2015/04/01/us-smartphone-use-in-2015/
- 5.Kim S, Yoon J. The use of an online forum for health information by married Korean women in the United States. Information Research. 2012 Jun;17(2):514. http://informationr.net/ir/17-2/paper514.html . [Google Scholar]
- 6.Islam NS, Patel S, Wyatt LC, Sim SC, Mukherjee-Ratnam R, Chun K, Desai B, Tandon SD, Trinh-Shevrin C, Pollack H, Kwon SC. Sources of health information among select Asian American immigrant groups in New York City. Health Commun. 2016;31(2):207–216. doi: 10.1080/10410236.2014.944332. http://europepmc.org/abstract/MED/26266574 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Oh KM, Kreps GL, Jun J, Ramsey L. Cancer information seeking and awareness of cancer information sources among Korean Americans. J Cancer Educ. 2011 Jun;26(2):355–364. doi: 10.1007/s13187-010-0191-x. [DOI] [PubMed] [Google Scholar]
- 8.Oh KM, Kreps GL, Jun J, Chong E, Ramsey L. Examining the health information-seeking behaviors of Korean Americans. J Health Commun. 2012 Aug;17(7):779–801. doi: 10.1080/10810730.2011.650830. [DOI] [PubMed] [Google Scholar]
- 9.United States Census Bureau. 2017. Mar 14, [2018-03-15]. FFF: Asian-American and Pacific Islander Heritage Month: May 2017 https://www.census.gov/newsroom/facts-for-features/2017/cb17-ff07.html .
- 10.Oh KM, Zhou QP, Kreps G, Kim W. The influences of immigration on health information seeking behaviors among Korean Americans and Native Koreans. Health Educ Behav. 2014 Apr;41(2):173–185. doi: 10.1177/1090198113496789.1090198113496789 [DOI] [PubMed] [Google Scholar]
- 11.Kao D. Factors associated with ethnic differences in health insurance coverage and type among Asian Americans. J Community Health. 2010 Apr;35(2):142–155. doi: 10.1007/s10900-009-9209-x. [DOI] [PubMed] [Google Scholar]
- 12.Yoo G, Kim B. Korean immigrants and health care access: Implications for the uninsured and underinsured. In: Kronenfeld JJ, editor. Inequalities and Disparities in Health Care and Health: Concerns of Patients, Providers and Insurers (Research in the Sociology of Health Care Vol 25) Bingley, United Kingdom: Emerald Group Publishing Limited; 2007. pp. 77–94. [Google Scholar]
- 13.American Association of Retired Persons. 2015. Jan, [2018-03-15]. The health and healthcare of Asian Americans and Pacific Islanders Age 50+: An AARP Report https://www.aarp.org/content/dam/aarp/home-and-family/asian-community/2015-01/AARP-Report-Health-and-Healthcare-AAPIs-50-Jan2015.pdf .
- 14.Hoeffel E, Rastogi S, Kim MO, Shahid H. United States Census Bureau. 2012. Mar, [2018-03-15]. The Asian Population: 2010 https://www.census.gov/prod/cen2010/briefs/c2010br-11.pdf .
- 15.Pew Research Center. 2013. Apr 04, [2018-03-15]. The rise of Asian Americans http://www.pewsocialtrends.org/files/2013/04/Asian-Americans-new-full-report-04-2013.pdf .
- 16.Amante DJ, Hogan TP, Pagoto SL, English TM, Lapane KL. Access to care and use of the Internet to search for health information: Results from the US National Health Interview Survey. J Med Internet Res. 2015;17(4):e106. doi: 10.2196/jmir.4126. http://www.jmir.org/2015/4/e106/ v17i4e106 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Bansil P, Keenan NL, Zlot AI, Gilliland JC. Health-related information on the Web: Results from the HealthStyles Survey, 2002-2003. Prev Chronic Dis. 2006 Apr;3(2):A36. http://www.cdc.gov/pcd/issues/2006/apr/05_0155.htm .A36 [PMC free article] [PubMed] [Google Scholar]
- 18.Berger M, Wagner TH, Baker LC. Internet use and stigmatized illness. Soc Sci Med. 2005 Oct;61(8):1821–1827. doi: 10.1016/j.socscimed.2005.03.025.S0277-9536(05)00120-6 [DOI] [PubMed] [Google Scholar]
- 19.Bundorf MK, Wagner TH, Singer SJ, Baker LC. Who searches the internet for health information? Health Serv Res. 2006 Jun;41(3 Pt 1):819–836. doi: 10.1111/j.1475-6773.2006.00510.x. http://europepmc.org/abstract/MED/16704514 .HESR510 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Benjamin EJ, Blaha MJ, Chiuve SE, Cushman M, Das SR, Deo R, de FSD, Floyd J, Fornage M, Gillespie C, Isasi CR, Jiménez MC, Jordan LC, Judd SE, Lackland D, Lichtman JH, Lisabeth L, Liu S, Longenecker CT, Mackey RH, Matsushita K, Mozaffarian D, Mussolino ME, Nasir K, Neumar RW, Palaniappan L, Pandey DK, Thiagarajan RR, Reeves MJ, Ritchey M, Rodriguez CJ, Roth GA, Rosamond WD, Sasson C, Towfighi A, Tsao CW, Turner MB, Virani SS, Voeks JH, Willey JZ, Wilkins JT, Wu JH, Alger HM, Wong SS, Muntner P, American Heart Association Statistics Committee and Stroke Statistics Subcommittee Heart Disease and Stroke Statistics-2017 Update: A report from the American Heart Association. Circulation. 2017 Mar 07;135(10):e146–e603. doi: 10.1161/CIR.0000000000000485.CIR.0000000000000485 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Burridge JH, Lee ACW, Turk R, Stokes M, Whitall J, Vaidyanathan R, Clatworthy P, Hughes A, Meagher C, Franco E, Yardley L. Telehealth, wearable sensors, and the Internet: Will they improve stroke outcomes through increased intensity of therapy, motivation, and adherence to rehabilitation programs? J Neurol Phys Ther. 2017 Jul;41 Suppl 3 Supplement, IV STEP Special Issue:S32–S38. doi: 10.1097/NPT.0000000000000183.01253086-201707001-00006 [DOI] [PubMed] [Google Scholar]
- 22.Anderson S, Whitfield K. Social identity and stroke: 'they don't make me feel like, there's something wrong with me'. Scand J Caring Sci. 2013 Dec;27(4):820–830. doi: 10.1111/j.1471-6712.2012.01086.x. [DOI] [PubMed] [Google Scholar]
- 23.Feigin VL, Forouzanfar MH, Krishnamurthi R, Mensah GA, Connor M, Bennett DA, Moran AE, Sacco RL, Anderson L, Truelsen T, O'Donnell M, Venketasubramanian N, Barker-Collo S, Lawes CM, Wang W, Shinohara Y, Witt E, Ezzati M, Naghavi M, Murray C, Global Burden of Diseases, Injuries, and Risk Factors Study 2010 (GBD 2010) and the GBD Stroke Experts Group Global and regional burden of stroke during 1990-2010: Findings from the Global Burden of Disease Study 2010. Lancet. 2014 Jan 18;383(9913):245–254. doi: 10.1016/s0140-6736(13)61953-4. http://europepmc.org/abstract/MED/24449944 .S0140-6736(13)61953-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Sharma N, Tridimas A, Fitzsimmons PR. A readability assessment of online stroke information. J Stroke Cerebrovasc Dis. 2014 Jul;23(6):1362–1367. doi: 10.1016/j.jstrokecerebrovasdis.2013.11.017.S1052-3057(13)00483-7 [DOI] [PubMed] [Google Scholar]
- 25.The Korean American Health Coalition. 2017. [2018-03-16]. Korean American Health Coalition Strategic Plan 2016-2018 http://kahealthcoalition.org/strategic-plan/
- 26.Jose PO, Frank AT, Kapphahn KI, Goldstein BA, Eggleston K, Hastings KG, Cullen MR, Palaniappan LP. Cardiovascular disease mortality in Asian Americans. J Am Coll Cardiol. 2014 Dec 16;64(23):2486–2494. doi: 10.1016/j.jacc.2014.08.048. https://linkinghub.elsevier.com/retrieve/pii/S0735-1097(14)06578-4 .S0735-1097(14)06578-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Song S, Song J, Park H, Chung J, The STROKE-COPE Community Advisory Panel. Ryan G, Vickrey B. Exploring stroke knowledge in Korean-American seniors: The strengthening our Korean elders through community-partnered education (STROKE-COPE) Project. Neurology. 2013;80(7 Supplement) http://n.neurology.org/content/80/7_Supplement/P05.237 . [Google Scholar]
- 28.Du H, Ma J, Li M. High-quality health information provision for stroke patients. Chin Med J (Engl) 2016 Dec 05;129(17):2115–2122. doi: 10.4103/0366-6999.189065. http://www.cmj.org/article.asp?issn=0366-6999;year=2016;volume=129;issue=17;spage=2115;epage=2122;aulast=Du .ChinMedJ_2016_129_17_2115_189065 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Walsh TM, Volsko TA. Readability assessment of internet-based consumer health information. Respir Care. 2008 Oct;53(10):1310–1315. http://www.rcjournal.com/contents/10.08/10.08.1310.pdf . [PubMed] [Google Scholar]
- 30.Agarwal N, Kommana SS, Hansberry DR, Kashkoush AI, Friedlander RM, Lunsford LD. Accessibility, reliability, and usability of neurosurgical resources. J Neurosurg. 2017 Apr;126(4):1263–1268. doi: 10.3171/2015.12.JNS151102. [DOI] [PubMed] [Google Scholar]
- 31.Tonsaker T, Bartlett G, Trpkov C. Health information on the Internet: Gold mine or minefield? Can Fam Physician. 2014 May;60(5):407–408. http://www.cfp.ca/cgi/pmidlookup?view=long&pmid=24828994 .60/5/407 [PMC free article] [PubMed] [Google Scholar]
- 32.Yi YJ, Stvilia B, Mon L. Cultural influences on seeking quality health information: An exploratory study of the Korean community. Libr Inf Sci Res. 2012 Jan;34(1):45–51. doi: 10.1016/j.lisr.2011.06.001. [DOI] [Google Scholar]
- 33.van Deursen AJAM, van Dijk JAGM. Using the Internet: Skill related problems in users’ online behavior. Interact Comput. 2009 Dec;21(5-6):393–402. doi: 10.1016/j.intcom.2009.06.005. [DOI] [Google Scholar]
- 34.The University of Utah, Department of Biomedical Informatics Open Access, Collaborative Consumer Health Vocabulary Initiative. [2018-03-16]. Consumer Health Vocabulary Wiki http://consumerhealthvocab.chpc.utah.edu/CHVwiki/
- 35.National Network of Libraries of Medicine. [2018-03-16]. Evaluating Health Websites https://nnlm.gov/initiatives/topics/health-websites .
- 36.McLaughlin G. SMOG grading: A new readability formula. J Reading. 1969;12(8):639–646. http://www.jstor.org/stable/40011226 . [Google Scholar]
- 37.Sung N, Lee D, Park K. Suitability assessment of patients' education materials made by Korean Academy of Family Medicine. Korean J Fam Med. 2004;25(9):669–677. http://www.kjfm.or.kr/journal/view.php?number=545 . [Google Scholar]
- 38.Rosenfeld L, Shepherd A, Agunwamba AA, McCray AT. Iterative evaluation of a web-based health information resource. J Health Commun. 2013 Aug;18(8):974–990. doi: 10.1080/10810730.2012.757395. http://europepmc.org/abstract/MED/23577665 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Shieh C, Hosei B. Printed health information materials: evaluation of readability and suitability. J Community Health Nurs. 2008;25(2):73–90. doi: 10.1080/07370010802017083.792795066 [DOI] [PubMed] [Google Scholar]
- 40.Doak CC, Doak LG, Root JH. Assessing suitability of materials. In: Doak CC, Doak LG, Root JH, editors. Teaching patients with low literacy skills. 2nd ed. Philadelphia, PA: Lippincott Company; 1996. pp. 41–60. [Google Scholar]
- 41.Guan Y, Maloney KA, Roter DL, Pollin TI. Evaluation of the informational content, readability and comprehensibility of online health information on monogenic diabetes. J Genet Couns. 2018 Jun;27(3):608–615. doi: 10.1007/s10897-017-0155-y.10.1007/s10897-017-0155-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Tran BNN, Singh M, Lee BT, Rudd R, Singhal D. Readability, complexity, and suitability analysis of online lymphedema resources. J Surg Res. 2017 Jun 01;213:251–260. doi: 10.1016/j.jss.2017.02.056.S0022-4804(17)30106-3 [DOI] [PubMed] [Google Scholar]
- 43.Robins S, Barr HJ, Idelson R, Lambert S, Zelkowitz P. Online health information regarding male infertility: An evaluation of readability, suitability, and quality. Interact J Med Res. 2016 Oct 21;5(4):e25. doi: 10.2196/ijmr.6440. http://www.i-jmr.org/2016/4/e25/ v5i4e25 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Tian C, Champlin S, Mackert M, Lazard A, Agrawal D. Readability, suitability, and health content assessment of web-based patient education materials on colorectal cancer screening. Gastrointest Endosc. 2014 Aug;80(2):284–290. doi: 10.1016/j.gie.2014.01.034.S0016-5107(14)00097-2 [DOI] [PubMed] [Google Scholar]
- 45.Nasser S, Mullan J, Bajorek B. Assessing the quality, suitability and readability of internet-based health information about warfarin for patients. Australas Med J. 2012;5(3):194–203. doi: 10.4066/AMJ.2012862. http://europepmc.org/abstract/MED/22952566 .2012862 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Finnie RK, Felder TM, Linder SK, Mullen PD. Beyond reading level: A systematic review of the suitability of cancer education print and Web-based materials. J Cancer Educ. 2010 Dec;25(4):497–505. doi: 10.1007/s13187-010-0075-0. http://europepmc.org/abstract/MED/20237884 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Garnweidner-Holme LM, Dolvik S, Frisvold C, Mosdøl A. Suitability assessment of printed dietary guidelines for pregnant women and parents of infants and toddlers from 7 European countries. J Nutr Educ Behav. 2016 Feb;48(2):146–151.e1. doi: 10.1016/j.jneb.2015.10.004.S1499-4046(15)00723-X [DOI] [PubMed] [Google Scholar]
- 48.Okuhara T, Ishikawa H, Okada H, Kiuchi T. Readability, suitability and health content assessment of cancer screening announcements in municipal newspapers in Japan. Asian Pac J Cancer Prev. 2015;16(15):6719–6727. doi: 10.7314/apjcp.2015.16.15.6719. http://journal.waocp.org/?sid=Entrez:PubMed&id=pmid:26434901&key=2015.16.15.6719 . [DOI] [PubMed] [Google Scholar]
- 49.Sbaffi L, Rowley J. Trust and credibility in Web-based health information: A review and agenda for future research. J Med Internet Res. 2017 Jun 19;19(6):e218. doi: 10.2196/jmir.7579. http://www.jmir.org/2017/6/e218/ v19i6e218 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Vega LC, Montague E, Dehart T. Trust between patients and health websites: A review of the literature and derived outcomes from empirical studies. Health Technol (Berl) 2011 Nov 18;1(2-4):71–80. doi: 10.1007/s12553-011-0010-3. http://europepmc.org/abstract/MED/22288026 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.The Joint Commission. Oakbrook Terrace, IL: 2010. [2018-03-16]. Advancing Effective Communication, Cultural Competence, and Patient- and Family-Centered Care: A Roadmap for Hospitals https://www.jointcommission.org/assets/1/6/ARoadmapforHospitalsfinalversion727.pdf . [Google Scholar]
- 52.The U.S. Centers for Medicare & Medicaid Services. Toolkit for making written material clear and effective. [2018-03-16]. 2012 https://www.cms.gov/Outreach-and-Education/Outreach/WrittenMaterialsToolkit/index.html .
- 53.Griffin E, McKenna K, Worrall L. Stroke education materials on the World Wide Web: An evaluation of their quality and suitability. Top Stroke Rehabil. 2004;11(3):29–40. doi: 10.1310/JKJK-N2MP-9LQ1-FF2N. [DOI] [PubMed] [Google Scholar]
- 54.Miller T, Leroy G. Dynamic generation of a Health Topics Overview from consumer health information documents. Int J Biomed Eng Technol. 2008;1(4):395–414. doi: 10.1504/IJBET.2008.020069. [DOI] [Google Scholar]
- 55.Shin Y, Kim J. A study for building a system of consumer vocabulary for health information. J Korean Soc Med Inform. 2009 Mar;15(1):31–40. doi: 10.4258/jksmi.2009.15.1.31. [DOI] [Google Scholar]
- 56.Jang Y, Kim G, Chiriboga DA. Health, healthcare utilization, and satisfaction with service: Barriers and facilitators for older Korean Americans. J Am Geriatr Soc. 2005 Sep;53(9):1613–1617. doi: 10.1111/j.1532-5415.2005.53518.x.JGS53518 [DOI] [PubMed] [Google Scholar]
- 57.Jang Y, Yoon H, Park NS, Chiriboga DA. Health vulnerability of immigrants with limited English proficiency: A study of older Korean Americans. J Am Geriatr Soc. 2016 Jul;64(7):1498–1502. doi: 10.1111/jgs.14199. http://europepmc.org/abstract/MED/27305524 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58.Sentell T, Braun KL. Low health literacy, limited English proficiency, and health status in Asians, Latinos, and other racial/ethnic groups in California. J Health Commun. 2012;17 Suppl 3:82–99. doi: 10.1080/10810730.2012.712621. http://europepmc.org/abstract/MED/23030563 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Stegeman CA, Zydney J. Effectiveness of multimedia instruction in health professions education compared to traditional instruction. J Dent Hyg. 2010;84(3):130–136. [PubMed] [Google Scholar]
- 60.US Department of Health and Human Services, Office of Disease Prevention and Health Promotion National Action Plan to Improve Health Literacy. [2018-03-16]. 2010 https://health.gov/communication/hlactionplan/pdf/Health_Literacy_Action_Plan.pdf .
- 61.Smith J, Forster A, House A, Knapp P, Wright J, Young J. Information provision for stroke patients and their caregivers. Cochrane Database Syst Rev. 2008 Apr;16(2):CD001919. doi: 10.1002/14651858.CD001919.pub2. [DOI] [PubMed] [Google Scholar]
- 62.Kim W, Kreps GL, Shin C. The role of social support and social networks in health information-seeking behavior among Korean Americans: A qualitative study. Int J Equity Health. 2015 Apr 28;14:40. doi: 10.1186/s12939-015-0169-8. https://equityhealthj.biomedcentral.com/articles/10.1186/s12939-015-0169-8 .10.1186/s12939-015-0169-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 63.Hwang SY, Zerwic JJ. Knowledge of stroke symptoms and risk factors among Korean immigrants in the United States. Res Nurs Health. 2006 Aug;29(4):337–344. doi: 10.1002/nur.20144. [DOI] [PubMed] [Google Scholar]
- 64.Jansen BJ, Spink A. How are we searching the World Wide Web? A comparison of nine search engine transaction logs. Inform Process Manag. 2006 Jan;42(1):248–263. doi: 10.1016/j.ipm.2004.10.007. [DOI] [Google Scholar]
- 65.Jacobson M. Leverage Marketing. [2018-03-18]. How far down the search engine results page will most people go? https://www.theleverageway.com/blog/how-far-down-the-search-engine-results-page-will-most-people-go/
- 66.Karaivanov D. Inet Solutions. 2016. Sep 08, [2018-03-18]. 25 Informative SEO Facts & Statistics You Probably Don’t Know About https://www.inetsolutions.org/25-informative-seo-facts-statistics-probably-dont-know/
- 67.Hodgdon M. Infront Webworks. 2015. Jul 17, [2018-03-16]. Value of Organic First-Page Results https://www.infront.com/blog/the-infront-blog/2015/06/17/value-of-first-page-google-results .