Health Literacy Analytics of Accessible Patient Resources in Cardiovascular Medicine: What are Patients Wanting to Know?

Abstract

Introduction

There remains an increasing utilization of internet-based resources as a first line of medical knowledge. Among patients with cardiovascular disease, these resources often are relied upon for numerous diagnostic and therapeutic modalities. However, the reliability of this information is not fully understood. The aim of this study was to provide a descriptive profile on the literacy quality, readability, and transparency of publicly available educational resources in cardiology.

Methods

The frequently asked questions and associated online educational articles on common cardiovascular diagnostic and therapeutic interventions were investigated using publicly available data from the Google RankBrain machine learning algorithm after applying inclusion and exclusion criteria. Independent raters evaluated questions for Rothwell’s Classification and readability calculations.

Results

Collectively, 520 questions and articles were evaluated across 13 cardiac interventions, resulting in 3,120 readability scores. The sources of articles were most frequently from academic institutions followed by commercial sources. Most questions were classified as “Fact” at 76.0% (n = 395), and questions regarding “Technical Details” of each intervention were the most common subclassification at 56.3% (n = 293).

Conclusions

Our data show that patients most often are using online search query programs to seek information regarding specific knowledge of each cardiovascular intervention rather than form an evaluation of the intervention. Additionally, these online patient educational resources continue to not meet grade-level reading recommendations.

INTRODUCTION

Cardiovascular disease continues to remain a leading cause of morbidity and mortality among individuals throughout the world.1 In the United States, healthcare expenditures on the management of this disease have increased by over 100 billion dollars in the past 20 years.2 As a result, there is a continued investment of resources in innovative healthcare systems and processes that continue to transform cardiovascular patient care, including facets such as decision-making models, diagnostic and therapeutic interventions, and patient understanding of these innovations.3 Regarding patient understanding, there remains an increasing utilization of internet-based resources as a first line of medical knowledge.4 With the advancement of technology and the wide-spread availability of internet access, patients have the convenience to access an abundance of healthcare educational information at their fingertips, and the global utilization of these resources continues to expand. However, the navigation of this abundance can be an obstacle given the overwhelming amount of information unregulated for content accuracy, quality, readability, and transparency.4–7 Therefore, it is vital to equip patients with the essential tools and education to actively participate in their informed decision-making process.

In cardiovascular medicine, patient education is pivotal in the management of diverse conditions such as hypertension, coronary artery disease, and heart failure. It also serves to inform patients about diagnostic and treatment modalities.8,9 A critical component of patient education is health literacy. This concept is an individual’s capacity to obtain healthcare information and interpret it in a manner that promotes the maintenance or enhancement of their health within a suitable context.10 The literature consistently demonstrates that a decline in healthcare literacy or the presence of poor healthcare literacy is correlated with an elevated risk of adverse health outcomes, such as increased hospital admissions and higher healthcare costs.11–14

With the wealth of Internet-based educational resources available, a patient’s online healthcare literacy is crucial in the realm of cardiovascular medicine. However, there remains a paucity of data evaluating the quality and content of the informational resources provided to patients in an online setting. The current literature on online cardiovascular disease education relatively has focused on disease education rather than diagnostic and therapeutic cardiac interventions which patients may be involved with during their disease management.15–18 This scarcity of information can hinder our comprehension of the existing landscape of online healthcare resources and diminish the focus on enhancing these resources if they fail to adequately educate patients on the subject matter. Therefore, the aim of this study is to describe and characterize the healthcare literacy profile of online patient educational materials on diagnostic and therapeutic cardiac interventions.

METHODS

To address the primary aim of this study, we performed a cross-sectional study in April 2023 to characterize components of healthcare literacy.19 This study did not require institutional review board approval given that it did not require human participants or animal subjects, and all the data utilized in this study were publicly available. The authors utilized a machine learning-based search engine algorithm, specifically RankBrain (Google, Mountain View, CA).20-23 In this public machine learning application, the most frequently asked questions were queried until the first unique 20 were extracted from the following categories: cardiac catheterization, percutaneous coronary intervention, balloon angioplasty, atherectomy, intra-aortic balloon pump, transcatheter aortic valve replacement, laser angioplasty, ventricular assist device, coronary artery calcium scan, echocardiogram, electrophysiology study, electrocardiogram, cardiac pacemaker. The choice of these categories was based on findings reported in previous literature.24–28 The first 40 questions and associated articles of each category were extracted based on these inclusion criteria: (1) the question and article are written in the English language, (2) the articles were publicly accessible without the creation of a subscription account or payment, (3) each article was at least 100 words, and (4) articles were extracted from the same search query.

After extracting each question and corresponding article, four independent reviewers evaluated each question per Rothwell’s Classification of Questions as demonstrated in previous literature.20,29–33 This classification was modeled after previous literature, as Rothwell’s Classification of Questions broadly categorizes the questions into the following categories: “Fact”, “Policy”, or “Value”.20,21,29–33 Additionally, these three categories were further subclassified by the raters into each respective group. This included questions regarding specific activities, length of time, restrictions, technical details, or costs within the “Fact” category. Questions within the “Policy” category were further subclassified as questions regarding either indications or complications. Questions within the “Value” category were further subclassified as questions regarding the evaluation of credibility or risk/benefit appraisal. 21,29–31,33

The corresponding educational articles, also addressed as an “educational resource” in this study, were further evaluated for content readability as shown in previous literature.33 Specifically, the literature content of the educational article was reformatted to plain text, Times New Roman, 12-point font, on Microsoft^® Word to allow raters to efficiently calculate readability scores for each resource. Additionally, resources were reformatted to remove author information, copyright disclaimers, figures, captions, legends, references, and web page navigation hyperlinks prior to the calculation of scores and standard deviation (SD) as outlined in previous literature. No content was revised or reviewed for source appraisal. Raters were then instructed to perform the following readability calculations after resource reformatting: Flesch Kincaid Grade Reading Level, Flesch Reading Ease, Gunning-Fog Index Readability, Coleman-Liau Index, Simple Measure of Gobbledygook (SMOG)Index, and Linsear Write Formula. The selection of these formulas was modeled on previous literature (Table 1).33,34 The use of readability calculations in this methodology was to serve as a modality to determine the quality and comprehension of these patient education resources as described in the aim of this study. Moreover, the use of these calculations can determine if the patient educational resources regarding each cardiac intervention meets reading level recommendations in the U.S. Educational resources which do not meet this recommendation can be considered more complex in terms of comprehension for patients. Regarding data tabulation, raters recorded all data using Microsoft^® Excel 2021 (Microsoft Corporation, Redmond, WA), and the date of the search queries also was tabulated to minimize potential ambiguity in data extraction.34–36 Inter-rater reliability was used to determine the degree of similarity among raters when quantifying Rothwell’s Classification of Questions. The use of this calculation would determine the degree of validity among the results.

Table 1.

Readability calculations.

Readability Calculation	Formula
Flesch Kincaid	0.39 x (words/sentences) + 11.8 x (syllables/words) − 15.59
Flesh Reading Ease	206.835 − 1.015 × (total words ÷ total sentences) − 84.6 × (total syllables ÷ total words)
Gunning Fog	0.40 [(words/sentences) + 100 (complex words/words)]
Coleman-Liau	0.0588 (average number of letters per 100 words) − 0.296 (average number of sentences per 100 words) − 15.8
Simple Measure of Gobbledygook (SMOG)	3 + √(number polysyllabic words)
Linsear Write	4.71(characters/words) = 0.5(words/sentences) − 21.43

RESULTS

There was a collective of 520 questions and associated articles across 13 cardiovascular interventions that were successfully extracted based on inclusion and exclusion criteria. Most articles (95.6%, n = 497) were from the U.S., followed by the United Kingdom (3.1%, n = 16). Nearly 31% (n = 161) of the articles were from academic institutions, 30.4% (n = 158) from commercial sources, 15.8% (n = 82) from government sources, 15.4% (n = 80) from medical practice, and 7.5% (n = 39) from media outlets. Table 2 contains further course classification by the cardiovascular intervention of interest. Precisely, no source classification succeeded in over 40% of each category among the 13 cardiac interventions.

Table 2.

Source frequency of educational materials on cardiac interventions.

Cardiac Intervention	Academic Institution	Commercial	Government Website	Media Outlet	Medical Practice
Atherectomy	10	10	11	2	7
Balloon Angioplasty	10	9	8	9	4
Cardiac Catheterization	15	11	8	2	4
Cardiac Pacemakers	8	9	10	6	7
Coronary Artery Calcium Scan	13	14	3	1	9
Echocardiogram	13	13	4	1	9
Electrocardiogram	12	16	3	1	8
Electrophysiology Study	15	12	4	0	9
Intra-aortic Balloon Pump	11	10	6	7	6
Laser Angioplasty	12	12	6	4	6
Percutaneous Coronary Intervention	10	10	7	5	8
Transcatheter Aortic Valve Replacement	16	16	8	0	0
Ventricular Assist Device	16	16	4	1	3

Regarding the 520 frequently asked questions extracted by the raters, there was an inter-rater reliability at 96.9% among Rothwell’s Classification of Questions. Most of the questions were classified as “Fact” at 76.0% (n = 395), followed by “Value” at 14.2% (n = 74), and “Policy” at 9.8% (n = 51; Table 3). Further subclassification of questions within each category demonstrated that questions regarding “Technical Details”- a subclassification of the “Fact” category - of each cardiac intervention were most common at 56.3% (n = 293), followed by the subclassification regarding “Evaluation of the Surgery” - a subclassification of the “Value” category - at 8.5% (n = 44).

Table 3.

Frequency table of Rothwell’s Classification of Questions on cardiac interventions.

Cardiac Intervention	Fact	Policy	Value
Atherectomy	26	10	4
Balloon Angioplasty	32	4	4
Cardiac Catheterization	28	3	9
Cardiac Pacemakers	34	3	3
Coronary Artery Calcium Scan	36	1	3
Echocardiogram	34	4	2
Electrocardiogram	30	4	6
Electrophysiology Study	35	5	0
Intra-aortic Balloon Pump	27	2	11
Laser Angioplasty	33	1	6
Percutaneous Coronary Intervention	30	1	9
Transcatheter Aortic Valve Replacement	23	6	11
Ventricular Assist Device	27	7	6

Regarding readability, a total of 3,120 readability scores were calculated from the 520 articles extracted in this study using six readability formulas (Table 4). Educational resources with content regarding an electrophysiology study had the lowest Flesh-Kincaid score at 10.2 (SD = 2.4), whereas resources regarding transcatheter aortic valve replacement had the highest Flesh-Kincaid score at 12.8 (SD = 6.8). Among Flesh Reading Ease calculations, resources regarding echocardiogram had the highest readability score at 67.4 (SD = 85.3), whereas resources regarding percutaneous coronary intervention had the lowest readability score at 43.4 (SD = 16.7; Figure 1). Among Gunning-Fog calculations, the average score was highest among resources regarding intra-aortic balloon pumps at 15.1 (SD = 5.9), and the lowest average score was among resources regarding echocardiograms at 12.6 (SD = 2.0). Among Coleman-Liau Index, the average score was highest among resources regarding percutaneous coronary intervention at 11.8 (SD = 2.6), and the lowest average score was the same between resources regarding laser angioplasty at 10.7 (SD = 1.7) and regarding echocardiograms at 10.7 (SD = 1.7).

Table 4.

Readability calculations of educational materials on cardiac interventions.

Cardiac Interventions	Flesch Kincaid Average SD	Flesch Reading Ease Average SD	Gunning-Fog Average SD	Coleman-Liau Index Average SD	SMOG Average SD	Linsear Write Average SD
Atherectomy	11.4	45.0	14.3	11.5	10.6	12.2
	3.2	17.6	3.2	2.5	2.4	3.9
Balloon Angioplasty	11.2	47.8	14.0	11.0	10.3	12.3
	2.5	13.3	2.5	1.6	1.7	3.4
Cardiac Catheterization	10.9	49.9	14.0	10.8	10.3	12.3
	2.7	16.0	3.0	2.3	2.1	3.2
Cardiac Pacemakers	10.4	51.8	13.4	10.9	9.9	11.4
	2.2	12.5	2.4	2.2	1.7	2.9
Coronary Artery Calcium Scan	11.3	47.0	14.2	10.9	10.6	12.5
	2.1	12.3	2.5	2.2	1.7	2.9
Echocardiogram	12.1	67.4	12.6	10.7	9.2	10.9
	2.9	85.3	2.0	1.7	1.3	2.8
Electrocardiogram	10.3	53.0	13.3	10.9	9.7	11.6
	3.1	17.1	3.3	2.5	2.4	3.8
Electrophysiology Study	10.2	51.8	13.5	10.9	9.8	10.8
	2.4	13.8	3.5	2.0	1.9	3.0
Intra-aortic Balloon Pump	11.5	44.0	15.1	11.3	10.7	12.4
	3.5	19.9	5.9	2.7	2.6	4.1
Laser Angioplasty	11.2	48.5	14.4	10.7	10.5	12.6
	3.0	13.2	3.2	1.7	2.3	5.0
Percutaneous Coronary Intervention	11.9	43.4	14.7	11.8	10.9	12.9
	2.8	16.7	2.9	2.6	2.1	3.3
Transcatheter Aortic Valve Replacement	12.8	44.3	14.6	11.1	10.7	13.2
	6.8	16.5	3.5	2.3	2.4	5.1
Ventricular Assist Device	11.5	50.5	14.0	10.9	10.4	13.2
	4.1	14.4	3.2	1.6	2.4	4.4

Figure 1.

Illustration of Flesch Reading Ease among cardiac interventions.

Among SMOG calculations, the average score was highest among resources regarding percutaneous coronary intervention at 10.9 (SD = 2.1) and the lowest average score was among resources regarding echocardiograms at 9.2 (SD = 1.3). Among Linsear-Write calculations, the average score was highest among resources regarding transcatheter aortic valve replacement at 13.2 (SD = 5.1) and ventricular left assist devices at 13.2 (SD = 4.4), and lowest average score was among resources regarding an electrophysiology study at 10.8 (SD = 3.0).

DISCUSSION

The rapid evolution of internet accessibility by the public, as well as the degree of oversight on fact-verification or resource credibility, creates a continuous need to evaluate and describe the climate of online health literacy.34–37 Similarly, information-seeking behavior among patients using these online resources, contributed by resource convenience, can play a role in the physical and mental health of a patient. Moreover, there is literature that suggests improper information-seeking behavior may be related to high-risk behaviors, including improper medication use, as well as reduced optimization in cardiovascular care.38–40 To address this current climate, this research investigation provided a descriptive profile of publicly available online educational resources and questions regarding clinical cardiovascular interventions.

Among all patient education resources included in this study, the source origin of most articles was from the U.S. at 95.6%, followed by the United Kingdom at 3.1%. These findings enable us to establish a geographical localization within our dataset.41 They suggest that the dataset of patient education materials used in this study would be most relevant for users seeking medical care within the U.S. healthcare infrastructure. Therefore, these findings may not be as informative for patients seeking information on these cardiac interventions in other countries, where guidelines, recommendations, and provisions may differ from those in the U.S.42,43 For example, the difference in insurance reimbursement structure schemes may vary among patients who enquire about a specific cardiac intervention (i.e., echocardiography) in the U.S. and in a different country.1,2,41,42 This can potentially account for a difference in the amount of applicable information, which can be relevant to the patient. Additionally, the most frequent source classification of articles was heavily composed of academic institutions at 31% and commercial sources at 30.4%. The prevalence of academic institutions in this dataset also implies a higher likelihood that the authors of each online education resource are affiliated with an academic setting rather than a privately employed practice. Additionally, educational materials from academic sources may have more interest to utilize literature using clinical research data compared to commercial sources which could rely on the potential utilization of aspects, such as physician credibility or medical brands.43–45

The results of Rothwell’s Classification of Questions in this study may indicate that patients are leaning toward seeking information regarding cardiac interventions rather than asking questions that evaluate the policies or perspectives of cardiac interventions. This finding could indicate that there is a curiosity present among patients and that may be due to a generalized paucity of knowledge regarding these cardiac interventions, or potentially due to a lack of content that appropriately answers the questions asked by patients in the U.S. Moreover, a majority of questions were focused on the technical details regarding each procedure compared to other subclassifications in the “Fact” category. This discovery is intriguing, particularly when contemplating the vital role that questions about costs or restrictions play in a patient’s education about cardiac interventions. It implies a scarcity of inquiries related to costs, indicating that patients may be directing such queries to alternative sources, such as their health insurance provider or physicians, rather than seeking information online.20,46–48 Future research should focus on comparing quality of primarily cost-based patient education resources on cardiac interventions.

The results of this readability analysis suggest that there is a generalized paucity of online patient education materials which meet institutional recommendations that the reading level of a resource should not be greater than the sixth-to-eighth-grade reading level.49–51 However, there are certain cardiac interventions that have readability scores that may require greater attention toward improvement.52,53 Specifically, patient education materials regarding percutaneous coronary interventions had the lowest Flesh-Reading Ease score at 43.4, which would indicate “difficult to read”. Similarly, percutaneous coronary interventions had the highest Coleman-Liau index and SMOG calculations among all cardiac interventions. This finding creates further validity to the greater need to improve the readability of materials on this intervention.53–55 Similarly, the readability scores on coronary artery calcium channel screening suggest that there remains a need to improve these materials, as readability scores have not improved since a similar analysis performed by Rodriguez et al.56 in 2020.

To the best of our knowledge, this is the first published study that utilizes Rothwell’s Classification of Questions and readability of online patient education materials on cardiac interventions in the U.S. A key strength of this study is the plan to analyze the Google RankBrain algorithm as the search query system comprises over 90% of the market share of the internet search query. Future studies ought to implement additional query programs to further increase the generalizability of these findings. Additionally, this methodology analyzed a sample size of online patient educational materials in cardiovascular medicine compared to previous literature.

Limitations

The study has limitations. Specifically, while this study employs health literacy assessment tools that are well-established in the literature, these tools do not assess the accuracy of content information among each educational resource material.20,21,57–60 Future studies ought to develop a measurement tool to assess for information transparency and accuracy in online educational articles. Similarly, the subjective nature of these assessment tools does not consider the potential overlap between categories when assessing questions. Although this study had a high interrater reliability of over 95%, there remains the potential to create the need for similar studies to validate these findings.

CONCLUSIONS

Overall, the findings of this study provide clinicians a direction toward improving the health literacy of patients in cardiovascular medicine. This study utilizes a machine learning algorithm to describe and assess questions commonly posed by the public audience. The findings of this study demonstrate that patients more often seek information regarding specific knowledge of each cardiovascular intervention rather than use it for value or policy-based questions or to seek advice. However, the current body of publicly available online educational literature does not meet grade-level reading recommendations across all interventions analyzed in this study. These findings encourage the need for improving literacy comprehension regarding these contemporary educational materials. Future studies ought to investigate efficacious solutions for creating general readability which can adapt to the continuously growing use of the internet.