Online patient literature related to oesophageal surgery: the need for greater readability, accreditation and higher quality

Abstract

Introduction

Oesophageal cancer is the sixth most common cause of death worldwide but is treatable through surgery. As part of the consenting process, surgeons may guide patients towards online information leaflets to understand more about their condition and treatment. This review aimed to systematically analyse some of the current resources that can be accessed via the internet.

Methods

A stringent search criteria was used to select online patient information leaflets for oesophageal cancer surgery. Leaflets were scored based on the Flesch–Kincaid Reading Ease score, DISCERN score, Health on the Net Code of Conduct (HONcode) certification/Information Standard Certification and International Patient Decision Aid Standards (IPDAS) score.

Findings

Only five sites had achieved HONcode certification. Only three sources were deemed readable using the Flesch–Kincaid scoring system and no sources reached the recommended readability using IPDAS. No source reached a maximum score with DISCERN, with the mean overall quality being 2.98. There was no significant difference between accredited and unaccredited sources. From our sample, patient information sources on oesophageal cancer surgery have a low readability.

Conclusions

More research is required to ascertain patient behaviour with regards to accessing the literature. Patients and healthcare professionals should liaise with each other to produce more readable, high-quality patient information on oesophageal cancer surgery.

Introduction

Oesophageal cancer is the sixth most common cause of death in the world.¹ A cancer diagnosis often leads to fear and uncertainty in patients.² Conveying information to a patient in an understandable way is key to increasing patient satisfaction and informed decision-making, while reducing anxiety and uncertainty, and helping the patient to cope;^3–6 increasingly, these resources may come from the internet.⁷ Patients may value the ability to research a diagnosis online after a consultation, with reduced jargon and clearly signposted resources.⁸ With this in mind, many trusts and healthcare providers have produced patient information documents to steer patients away from the paternalistic consent and towards a more patient-centred consultation. Often, patients may have already accessed online information before a consultation, so it is of paramount importance that information is accurate, readable and accessible.

Guidance from the Royal College of Surgeons of England (RCS England) informs surgeons that use of a pre-printed pro forma outlining the risks of a procedure is no longer enough,⁹ but instead the information must be tailored to the patient and the surgeon must be satisfied that the patient has understood the information and can make an informed decision. Surgeons, therefore, may steer patients towards online resources to build their knowledge, and the RCS Eng guidance on shared decision-making emphasises the potential role of online resources in aiding this.

Previous studies have identified significant concerns regarding the quality and readability of online patient literature related to prolapse surgery and bariatric surgery.^10,11 Furthermore, prior to changes in the guidelines, Blencowe et al considered the quality of written information regarding oesophagectomy, direct from 41 of the cancer centres in England in Wales;^1,2 however, the current review considers online information, readily accessible through search engines.

The aim of this review was to assess the quality, accreditation and readability of online material on oesophageal surgery for patients to support a shared decision-making process.

Methods

This study systematically analysed the content and quality of online materials on oesophageal surgery available for patients to support a shared decision-making process. This involved looking at various parameters, including the readability of the material, accreditation, applicability and quality. We then scored the data using validated tools: the Flesch–Kincaid Reading Ease score,¹³ DISCERN score,¹⁴ Health on the Net Code of Conduct (HONcode) certification/Information Standard Certification (ISC)¹⁵ and the International Patient Decision Aid Standards (IPDAS) score.¹⁶

A systematic search for online patient websites on surgery for oesophageal cancer was completed using internet searches from the three most popular search engines: Google UK (www.google.co.uk), Yahoo UK (www.yahoo.co.uk) and Bing (www.bing.com). Searches were screened using the three key phrases oesophagectomy, oesophageal cancer and gullet cancer. These phrases were chosen to replicate searches by patients, substituting the anatomical term ‘oesophagus’ with a more lay term ‘gullet’.

Websites were screened for inclusion if they appeared on the first two pages of each search, based on previous studies claiming that 92% of internet users do not search beyond the first page.¹⁷ Search engines were used in an ‘incognito’ window to avoid any effects of personalisation based on the search history of the researchers. Finally, a Google Analytics search was performed across all the final extracted websites to check that the same search terms used above were used by others to arrive at the website domains.

Data were collected independently by two authors between 2 and 4 January 2021 and collated in an Excel 2013 (Microsoft, Redmond, WA, USA) spreadsheet. The authors then screened for duplicates and performed a full-text analysis against inclusion/exclusion criteria. Any discrepancy was resolved by referring to the IPDAS and DISCERN manuals.

Sources were included if they fulfilled the search criteria, were aimed at patients and were written in the English language. Exclusion criteria included advertisements for private institutions (information pages from private institutions were included), subscription sites, journalistic articles, academic papers and video sources. Figure 1 illustrates this process using a PRISMA diagram.

Figure 1 .

PRISMA flow chart for the study

A list of the sources provided from the search and application of inclusion and exclusion criteria is provided in Appendix 1. The country of origin, institution and URLs are listed.

Websites were then scored across four domains: the Flesch–Kincaid Reading Ease score, the DISCERN score, the HONcode certification/ISC and the IPDAS score.

The Flesch–Kincaid Reading Ease score is a formula created by Robert Flesch in 1948.¹³ It uses the average sentence length in words and the average word length in characters in the following formula:

This grades text with a numerical value between 0 and 100, with the number increasing as a source becomes more readable. Table 1 highlights the grading scale and target sources. Scores from 0 to 30 are considered to a university graduate level and scores above 80 are generally considered easy to read.

Table 1 .

Grading for the Flesch–Kincaid score

	Difficulty level	Target source home
0–30	Very difficult	Scientific
30–50	Difficult	Academic
50–60	Fairly difficult	Quality
60–70	Standard	Digests
70–80	Fairly easy	Slick-Fiction
80–90	Easy	Pulp-Fiction
90–100	Very easy	Comics

The DISCERN score was created in 1999 to assess the quality of written communications specific to patient health information. It is made up of 16 questions, split into three sections. Section 1 relates to the reliability of the publication and whether it is trustworthy as a source of information, section 2 discusses the quality of the information on the treatment choices and section 3 relates to an overall rating of the quality of the source.¹⁴ Each question is rated on a five-point Likert scale,¹⁸ with grading listed in Table 2. The questions are given in Table 3. Two authors scored the website by consensus, and ambiguity was resolved using the DISCERN manual.

Table 2 .

Table describing Likert analysis in DISCERN score

Likert score	1	2	3	4	5
Criterion	Not fulfilled	Partially fulfilled			Fulfilled

Table 3 .

Questions in the DISCERN survey

Question no.	Question
Section 1
1	Are the aims clear?
2	Does it achieve its aims?
3	Is it relevant?
4	Is it clear what sources of information were used to compile the publication?
5	Is it clear when the information used or reported in the publication was produced?
6	Is it balance and unbiased?
7	Does it provide details of additional sources of support and information?
8	Does it refer to areas of uncertainty?
Section 2
9	Does it describe how each treatment works?
10	Does it describe the benefits of each treatment?
11	Does it describe the risks of each treatment?
12	Does it describe what would happen if no treatment was used?
13	Does it describe how treatment choices affect overall quality of life?
14	Is it clear that there may be more than one possible treatment choice?
15	Does it provide support for shared decision-making?
Section 3
16	Based on the answers to all of the above questions, rate the overall quality of the publication as a source of information about treatment choices.

HONcode¹⁵ is a Swiss certification, used mostly by US websites, but also a few from the UK, to certify the reliability and credibility of information. Websites can be certified with this, indicating that the developer maintains basic ethical standards and to make sure that the readers understand the source and use of the information.

The ISC was also considered along with the HONcode. This is a certification provided by the National Health Service England (NHS England). It is formed from principles such as statements based on the quality of information production, evidence, user understanding, end product, feedback and review. Numerous organisations in the UK are certified and certification has to be renewed every 3 years.¹⁹

The final modality used was the IPDAS score, another measure of reliability and effectiveness of a patient decision aid. It was developed through a Delphi consensus process. A 12-item checklist based on the IPDAS v.4.0 score was developed, ratified by other studies¹⁶ and is listed in Table 4.

Table 4 .

IPDAS criteria and questions

No.	Criterion
1	Describes health condition
2	States treatment options clearly
3	Describes options
4	Describes positive features
5	Describes negative features
6	Describes the experience of consequences of options
7	Balanced an equal detail for all options
8	Citation to evidence
9	Publication date provided
10	Update policy provided
11	Information about levels of uncertainty around event
12	Funding source

Data were analysed using Excel 2013 and means were calculated based on the various scoring systems.

Findings

Accreditation

When sources were screened for HONcode or ISC accreditation, only five (ID numbers 4, 6, 18, 20 and 22) achieved this (n=5, 22.7%), three of which were HONcode certified. Only one NHS site (16.7%) had accreditation from the ISC.

Readability

Figure 2 shows the Flesch–Kincaid Reading Ease scores of the sources. Scores ranged from 16.7 to 91. The mean (s.d.) score was 58.5 (±12.7). Fourteen sources were within one standard deviation of the mean. This fits into the ‘fairly difficult’ category and when compared with the Flesch–Kincaid reading age, would place the sources as something to be understood by someone in the 10–12th grades, between the ages of 15 and 18. Research from Cotugna and colleagues recommends a score of above 80 for patient literature.¹⁹ Only sources 1, 9 and 14 met this criterion. No significant difference was found between accredited and unaccredited sources (p=0.47), with the mean (s.d.) score of 52.9 (±6.61) for accredited sources and 60.2 (±21) for unaccredited sources.

Figure 2 .

Flesch–Kincaid Reading Ease score of the sources

As shown in Figure 3, IPDAS scores ranged from 1 to 8, with a mean (s.d.) of 4.67 (±2.19). Furthermore, only one source provided an update policy. No sources achieved a score of 12 as recommended by IPDAS for the correct distribution of information. When accredited and unaccredited sources were compared, there was no significant difference (p=0.85) with a mean (s.d.) of 4.8 (±2.79) for the accredited sources and 4.59 (±1.91) for unaccredited sources.

Figure 3 .

IPDAS and DISCERN scores of the sources

When using the DISCERN modality, no sources achieved a perfect score. As shown in Figure 3, the mean overall quality of sources (question 16) was 2.98, fitting into the partially fulfilled criterion. When sources were compared with one another, the average DISCERN score ranged from 1.50 to 3.94. Sources scored highest on relevance (4.09) and lowest on describing what would happen if no treatment was given (1.18). When the mean quality of unaccredited and accredited sources were compared with a t-test, there was no difference between the two (p =0.81), with a mean (s.d.) of 2.99 (±0.43) for accredited sources and 2.93 (±0.56) for unaccredited sources.

A subgroup analysis of NHS and non-NHS sources revealed no difference between the two, however because the majority of sources were NHS, it was not statistically appropriate to compare them.

Discussion

Our research reports on the quality, readability and official accreditation of online patient resources related to oesophageal cancer surgery. The mean readability of our sources was 58.5, which is much more difficult than is recommended for patient literature. In terms of quality and reliability, the use of validated scores demonstrates that no source achieved the minimum score of 12 required by IPDAS. The average DISCERN score 2.98 for overall quality which corresponds to ‘partially fulfilled’ on the Likert scale. Only five sources achieved accreditation.

Limitations of our study include the restriction of search criteria to the first two pages of each search engine to capture the most visible data. This was done in keeping with research by Cotugna et al²⁰ who commented that 92% of users do not go past the first page of the search engine. Another limitation is that the majority of the sources were from the same overall provider, NHS, although most were authored by individual NHS Trusts. In addition, results were dependent on author scoring, which subjected them to a level of bias. Video material was not analysed, owing to the limitations of our validated scoring systems. The utility and impact of video material on patient decision-making is uncertain. Ferhatoglu et al looked at the quality of YouTube (www.youtube.com) videos and found that the lack of references and data sources in the videos and the lack of a review process to assess their reliability questioned the academic rigour of this medium. However, it should be noted that the focus of this review was videos for academic purposes rather than patient content, although DISCERN, a patient information tool was used to assess quality.²¹ In addition, another limitation was that we did not analyse social media providers such as Twitter, mobile apps and patient forums. The role of these in patient decision-making is uncertain.

Conclusions

Further research is needed to ascertain patient behaviour with regards to accessing and processing online patient literature. The peak incidence of oesophageal cancer is in the seventh decade²² and data has shown that only 48% of those aged 65–75 years used the internet in the past year and that the use of e-Health²³ declines with age. Greater signposting by clinicians and reintroducing the use of printed information may aid this. The authors recommend that the full strength of the multidisciplinary team and patients engage in producing better quality, more readable and accredited patient-related online literature with regards to oesophageal cancer surgery. This could take the form of a focus group with patients, clinicians and key stakeholders coming together under the umbrella of the National Institute for Health and Care Excellence or through the RCS Eng. Furthermore, online patient information leaflets could be prepared via a national pro forma to maintain uniformity, but allow for differences in provision dependent on NHS Trust. It is only through this that we can hope to appropriately utilise such material to aid decision-making.