Patient-Directed Vasectomy Information: How Readable Is It?

Abstract

Purpose

To assess the quality and readability of online health information on vasectomy using validated readability and quality assessment tools.

Materials and Methods

The top 50 search results for "vasectomy" on Google, Bing, and Yahoo were selected. Duplicate links, advertisements, blog posts, paid webpages, and information intended for healthcare providers were excluded. Flesch Reading Ease score, Flesch–Kincaid Grade level, Gunning Fog Index, and Simple Measure of Gobbledygook (SMOG) index were used to assess readability, with optimal readability level for online health information established as being at sixth grade reading level. DISCERN Instrument and JAMA Benchmark were used to assess the quality of selected webpages. Inter-assessment score correlation and results by webpage type were analyzed.

Results

We analyzed 44 webpages, including 16 academic, 5 hospital-affiliated, 6 commercial, 13 non-profit health advocacy, and 4 uncategorized sources. The average readability of the evaluated webpages was at a 10th grade reading level as measured by the Flesch Kincaid Assessment tool, and an undergraduate reading level per the SMOG and Gunning Fog indices. Non-profit health advocacy webpages had the best reading level but still was not at the recommended level of grade 6 to 7. The overall DISCERN quality of the webpages was “fair”, with non-profit health advocacy pages performing best.

Conclusions

The assessed webpages offer education on vasectomy in a language that is too complex for the general population to understand. Furthermore, several sources for online health information, such as non-profits, outperformed webpages by academic institutions. Increased healthcare collaboration and dedication to producing quality online patient resources is necessary to address these shortcomings and build trust among patients to increase utilization of vasectomy and decrease decisional regret.

INTRODUCTION

Vasectomy is the most effective and only long-term option for male contraception. Approximately 500,000 Americans undergo vasectomy each year [1]. It is a simple, safe, and effective method for sterilization. Routinely performed in the office under local anesthesia, vasectomy is very cost-effective and has a short recovery time [2]. Compared to female sterilization procedures, vasectomy is less invasive and less expensive [2]. Despite this, women undergo sterilization at rates of almost twelve times their male counterparts [3].

Poor health literacy may be contributing to the hesitancy surrounding vasectomy. Previous studies have pointed towards the effect of health illiteracy on preventative care and general health outcomes [4,5]. One Indian study specifically has looked at the impact of health literacy on acceptance of vasectomy [6]. They found that literate individuals accept vasectomies at a significantly higher rate than female sterilization. Additionally, a significant portion of men that consider undergoing the procedure do not feel well informed, leading some to regret their decision afterwards [2].

According to a survey conducted by California Health Care Foundation, an estimated 80% of internet users search for health information online [7]. In order for online health information to be most impactful, it must be both accurate and accessible. In fact, the National Institute of Health (NIH) and American Medical Association (AMA) recommend that patient-targeted health information be written at a 6th grade reading level or lower [8,9]. This is of particular importance in the realm of online-health information. Accessibleon-line health information may not only increase patient awareness of their treatment options, but also improve their ultimate treatment satisfaction. On the other hand, online health information that is accessible but not inaccurate and poor quality may contribute to a sense of mis- or disinformation, similar to the phenomenon seen in recent years with vaccine information [10]. Against this backdrop, we sought to better understand the online educational content on vasectomy by evaluating both its readability and quality.

MATERIALS AND METHODS

This study did not require access to any patient related health information, and therefore IRB approval was not required. Webpages with patient-oriented health education material about vasectomy were selected by interrogating the search term "vasectomy" on December 15, 2022, on three internet search engines: Google, Yahoo, and Bing. To minimize geographic bias and the influence of internet "cookies," a Google Chrome browser on "incognito mode" was used. The first 50 results from each search engine were collected to establish a list of 150 webpages, which were then culled through exclusion criteria for a final total of 44 websites (Fig. 1). Webpages were excluded if they were clearly advertisements, lay publications such as news features or magazine articles, blog posts, articles intended for health professionals, paywalled websites, and websites with information unrelated to vasectomy. The final list of web sites was then divided into the following categories: academic, hospital-affiliated, commercial, non-profit health advocacy, and other.

Fig. 1. Schematic depicting webpage selection.

1. Readability of online content

The content of each webpage was evaluated by one of our study authors, AJL, using four validated readability score systems: Flesch Reading Ease score, Flesch–Kincaid Grade level, Gunning Fog Index, and Simple Measure of Gobbledygook (SMOG) index [11,12,13]. The formulas for each of these tests are derived from text characteristics such as number of sentences, word count, and syllables per word. The Flesch Reading Ease score utilizes a scoring system from 0 to 100 with higher scores indicating easier reading. Scores of 75 to 80 generally equate to the recommended level of grade 6 to 7 for health material. The Flesch–Kincaid Grade level estimates the education level needed to fully understand a particular text and is calculated independently from the Flesch Reading Ease score. The Gunning Fog and the SMOG indices are two additional assessments that provide the education level required to comprehend a text on the first attempt. The target score for the Gunning Fog and SMOG indices is <6.9 to meet the validated recommendations for accessible comprehension. To obtain these scores, text extracted from each selected website was inputted into the Character Calculator website (www.charactercalculator.com).

2. Assessment of quality

Evaluation of the quality of online resources was done by our study author, AJL, using DISCERN and the JAMA Benchmark criteria. The DISCERN tool consists of 16 questions and three parts: reliability (Q1–8), quality of information on treatment choices (Q9–15), and overall score (Q16) [14]. Each question gets a score of 1-5 with a total possible score of 80. Previous literature suggests that scores between 16 to 26 are considered very poor, scores between 27 to 38 are poor, scores between 39 to 50 are fair, scores between 51 to 62 are good, and scores greater than 63 are excellent [15]. Furthermore, the JAMA Benchmarks, as first created by Silberg et al [16] were utilized to further assess the basic quality of selected internet information. The JAMA Benchmarks include four areas that a webpage needs to include to achieve higher quality: authorship, attribution, disclosures, and currency.

3. Statistical analysis

Results were analyzed using simple descriptive statistics. Pearson’s correlation coefficient was calculated in SPSS 26.0 (SPSS Inc.) to evaluate the correlation among mean scores obtained by both quality metrics, all readability metrics, and lastly, between the quality and readability metrics. One-sample t-test was used to compare Flesch Kincaid Readability Ease with the score corresponding to the AMA recommendations.

RESULTS

Forty-four of 150 screened webpages were included in our study. An example of one included website (https://www.cdc.gov/reproductivehealth/contraception/mmwr/spr/male_sterilization.html) describing the term "vasectomy" is included in Fig. 2. Only 32 (72.7%) of webpages mentioned the side effects often experienced post-vasectomy. Thirty-nine (88.6%) of webpages mentioned the potential for vasectomy reversal in case of regret or desire to have children in the future, but only 23 (52.3%) mentioned the complications of a vasectomy reversal.

Fig. 2. An example of health information on vasectomy.

1. Readability of online health information

The average readability score of included webpages as measured by the Flesch–Kincaid Grade level was equivalent to 10th grade education level, while the average Gunning Fog Index and the SMOG index were at the undergraduate level of education. The Flesch–Kincaid Grade level was significantly higher (i.e. more complex reading level) (p<0.0001) than the recommended 6th grade reading level. The Flesch Reading Scale and the average of Gunning Fog and SMOG indices had a strong negative correlation (r=-0.735; p<0.001).

2. Quality of online health information

Total DISCERN score of the 44 online sites was fair (average score of 47.5). Fig. 3 demonstrates the breakdown of DISCERN scores. Only 5 (11.4%) webpages received a score of excellent, while 11 (25.0%) sites received a score of poor or very poor. Only 11 (25.0%) webpages met all four JAMA Benchmark criteria, while 15 (34.1%) met only one criterion and 4 (9.1%) met none at all. The two quality assessment scores (DISCERN and JAMA Benchmark) were moderately positively correlated (r=0.451; p<0.01). There was a weak correlation between the measured quality of online health information for vasectomy when compared with the readability of the selected webpages. Readability and quality scores differed by category, with non-profit health advocacy webpages being the highest in both readability and quality (Table 1, 2).

Fig. 3. Breakdown of quality of online health information by DISCERN score.

Table 1. Average readability scores and grade level of online health information.

Categories (n)	Flesch readability ease scorea [scale 0–100, with 100 indicating the easiest comprehension level]	Gunning fog index scoreb [Target score for accessible comprehension: <6.9]	SMOG index scorec [Target score for accessible comprehension: <6.9]	Flesch–Kincaid grade levela [Recommended reading level: 6th grade]
Overall (44)	50.4±13.9	15.0±3.5	13.6±2.4	10th Grade
Academic (16)	49.1±10.1	15.4±3.6	14.0±2.5	10th Grade
Hospital-affiliated (5)	53.7±12.6	13.6±2.7	12.8±1.9	10th Grade
Commercial (6)	52.7±8.6	13.7±1.6	12.8±1.1	10th Grade
Non-profit health advocacy (13)	54.7±16.9	14.9±4.2	13.2±2.7	9th Grade
Other (4)	35.7±19.0	17.3±3.2	15.9±1.7	Undergraduate Level

Values are presented as number only or mean±standard deviation.

SMOG: Simple Measure of Gobbledygook.

^aFlesch Kincaid Readability Ease scores the reading complexity of a given text from 0–100 (higher score is less complex grade level). The Flesch–Kincaid Grade Level reports the education grade level corresponding to the readability ease score.

^bThe Gunning Fog index measures readability as a score from 6–17 with a higher score indicating a more complex grade level. Score of 6 corresponds to 6th grade and a score of 17 corresponds to college graduate level.

^cThe SMOG index measures readability as a score ≥3.12 with a higher score indicating a more complex grade level. Scores corresponding to grade level is broken down into: ≤4.9 (Elementary school), 5.0–8.9 (Middle school), 9.0–12.9 (High school), 13.0–16.9 (Undergraduate), and ≥17 (Graduate).

Table 2. Quality assessment of online health information.

Categories (n)	DISCERN instrument scorea	JAMA benchmark scoreb
Overall (44)	47.5±12.7	2.2±1.4
Academic (16)	46.9±11.8	1.8±1.0
Hospital-affiliated (5)	37.0±7.0	1.8±1.6
Commercial (6)	49.0±10.0	1.7±1.2
Non-profit health advocacy (13)	50.8±14.2	2.8±1.6
Other (4)	50.3±17.9	2.8±1.9

Values are presented as number only or mean±standard deviation.

^aDISCERN Instrument is a validated quality assessment tool compromised of 3 categories with a total of 16 questions, maximum score is 80. Descriptive cutoffs are further broken down into very poor (score 16–26), poor (score 27–38), fair (scores 39–50), good (scores 51–62), and excellent (scores >62).

^bJAMA Benchmark is a validated quality assessment tool compromised of 4 criteria (authorship, attribution, disclosure, and currency) that a high-quality resource should meet, maximum score is 4.

DISCUSSION

Vasectomies are significantly safer, more cost-effective, and require less recovery time compared to female sterilization procedures [2]. Yet, women seek more invasive procedures for permanent birth control at significantly higher rates [3]. According to the United Nations, an estimated 219 million women of reproductive age have undergone female sterilization, accounting for the most common (24%) form of birth control worldwide. In contrast, male sterilization through vasectomy was reported to be the least utilized form of birth control (2%) among partners of reproductive age [3]. While various cultural and sociological factors are at play, increasing the availability of high-quality online health information on vasectomy can help reduce knowledge gaps and increase utilization of male sterilization [2].

Sixty one percent of all adults look up health information online, and 49% accessed various webpages to receive information and learn more about specific diseases and treatment options [7]. Notably, google searches of vasectomy have significantly increased since the Dobbs vs. Jackson ruling in the United States [17]. Ease of reading is essential for the accessibility of health information, yet several studies have shown the current reading level of online health information to be too complex for an average person living in America [18,19,20,21]. Unfortunately, our study demonstrates this applies to online vasectomy content as well. The NIH and AMA suggest health information to be written for a maximum reading level of 6th grade [8,9]. The average readability of the evaluated webpages on vasectomies was at a 10th grade reading level as measured by the Flesch Kincaid Assessment tool, and an undergraduate reading level per the SMOG and Gunning Fog indices. Only one webpage received the appropriate score by the Flesch Kincaid Readability Ease and Grade Level tools. Despite the high correlation between assessment tools, no webpage achieved the recommended level across all readability scoring systems.

The quality of patient education material was also lacking. Utilizing the DISCERN Instrument for quality assessment, we identified 50% of the web sources to have fair or worse quality. Using the JAMA Benchmark criteria, we identified four webpages that did not meet any of the criteria of authorship, attribution, disclosures, and currency. Fifteen (34.1%) sources only met one criterion.

When broken down into categories, non-profit webpages had a slightly higher score in terms of both quality and readability. Academic webpages are usually regarded as trusted sources of information. However, only 44% of academic webpages were of fair quality by DISCERN score, and of those, only one was excellent quality. Over half of the academic webpages did not specify the author (nor their credentials), and 7 pages (44%) did not have a publication year. Similarly, many academic webpages failed to provide references for their published content. Such findings can contribute to the overall low credibility and reliability of online health information.

One way to improve the complexity of online health information is the potential utilization of modern technologies, such as artificial intelligence (AI). To illustrate this, we used ChatGPT, an open-source AI platform (https://chat.openai.com/auth/login), to "Write a vasectomy education handout with references at a 6th grade reading level". The resulting text is shown below in Fig. 4. Compared with the webpages included in this study, with an example shown in Fig. 2. The text generated by ChatGPT generally consists of fewer sentences in each paragraph, fewer syllables, fewer complex words, and a fewer number of total words. Next generation machine learning tools could prove to be vital in crafting health related patient information by meeting high quality guidelines and being reviewed by physician experts for accuracy.

Fig. 4. Illustration of patient education handout on vasectomy utilizing ChatGPT.

The major limitation of this study is that we only searched the term "vasectomy" in the English language using the top three search engines with the highest traffic. Men that undergo vasectomy in English speaking countries only make up a proportion of all vasectomies per year, and other search engines in different languages might be used more in other parts of the world. Furthermore, we only searched for the term "vasectomy" but other terms may be used by men interested in undergoing the procedure. Lastly, only one author performed the readability and quality assessments which could be considered a limitation. The readability of online health information, however, was done on an automated calculator with integrated readability formulas. The DISCERN Instrument and JAMA for quality assessment are also objective tools with detailed descriptions online regarding their proper use. Our author performed the quality assessment after comfortably understanding the rules of scoring. Despite these limitations, the use of different forms of readability and quality scoring systems allow for a reliable assessment of online, patient-focused vasectomy information. The easily available and relatively low-quality information presents an opportunity for physicians and specialty organizations to create high quality and easily readable materials for patient education.

CONCLUSIONS

Compared to female sterilization, vasectomies are performed at a considerably lower rate despite high efficacy and low risk. Given that over half of the American population learn about their medical conditions utilizing the internet, high-quality and easy-to-read information can help men and their partners to make well-informed decisions about choosing vasectomy and seeking out this type of reproductive care. This study highlights that the currently available online patient resources are of suboptimal quality and difficult to comprehend. Efforts aimed at improving the quality and readability of online health information for vasectomy may help improve quality and reduce disparities in reproductive care.