The Accuracy of Information about Orthodontics Available on the Internet

Abstract

Objective

The aim of the present study was to evaluate the content of the informative websites related to orthodontic terms in Turkish and in English.

Methods

Five different orthodontic terms (“orthodontic appliances (ortodontik aygıtlar),” “orthodontic braces (ortodontik braketler),” “orthodontic elastics (ortodontik elastikler),” “orthognathic surgery (ortognatik cerrahi),” and “orthodontic treatment (ortodontik tedavi)”) both in Turkish and in English were searched using Google. There were 25 websites evaluated for each term. A total of 137 different websites were evaluated with three measurement tools: DISCERN (questionnaire, University of Oxford, 1999) (quality of information), LIDA (v1.2 Minervation, 2007) (accessibility, usability, and reliability), and AChecker (v0.1 ATutor, 2011) (accessibility).

Results

The mean overall score of the quality of information was “good” for terms in Turkish and in English. The LIDA score was classified as “moderate” for terms in Turkish and in English. More accessibility errors were found on the Turkish websites than on the English counterparts. Most of the statistical evaluations between Turkish and English terms were insignificant. However, intragroup evaluation of the terms mostly showed significant differences.

Conclusion

Accessibility, usability, and reliability; quality of information; and scores of access errors showed variations among Turkish and English sites. The collaboration of website designers and clinicians to increase the quality level of the websites is recommended.

INTRODUCTION

The Internet comes from the words “interconnected networks” or “international networks,” meaning “networks connected to each other.” This interconnectedness has given users the ability to do daily tasks quickly, making life easier and saving much time.

Health-related information is among the most frequently accessed information on the Internet. Today, approximately three-quarters of the population in developed countries use the Internet to receive information about health issues (1).

The Internet is also being used increasingly by patients to access dental information. Therefore, the number of dentistry-related websites on the Internet is increasing every day. However, the lack of any control over the Internet and the ease by which information is published and spread created the need for safe and reliable sites that provide health information. Promotional websites about dentistry or orthodontics often contain advertising information that pushes the limits of what treatments offer. Therefore, a number of ethical principles have been identified by national and international organizations to ensure that health websites are accurate and do not in any way mislead users with the information they provide (2). Even though the ethical rules are being determined, the audit system is weak.

Some validated tools, which evaluate the content of the websites from different perspectives, have been developed as a result of the preponderant access of the public to online health-related information. The DISCERN questionnaire, which was developed in 1999 (University of Oxford, UK), has been originally framed to analyze the quality of written information (3). The DISCERN was the first standardized and validated tool to evaluate the quality of the healthcare information spread through the web. The tool includes 16 questions. The first part evaluates the reliability of the written information, the second part assesses the presentation of alternative treatment options, and the final part analyzes the general quality rating question. All answers are scored from 1 to 5, and the total available maximum score is 80 versus the minimum score of 16. Som and Gunawardana (4) categorized the DISCERN scores as excellent (63–75), good (51–62), fair (39–50), poor (27–38), and very poor (15–26).

The LIDA (v1.2 Minervation, 2007) instrument is designed to evaluate the accessibility, usability, and reliability features of health websites (5). Accessibility indicates the compliance with the World Wide Web Consortium (W3C) accessibility guidelines. Usability is the ease of use of a website for an end-user, without requiring any specialized training. On the other hand, reliability refers to whether the information provided by the website is evidence-based, accurate, objective, or current. It is also expected to provide information on other treatment options and links to access additional sources on the websites. Deficiently designed websites create barriers to aggregate online information. Therefore, increasing the accessibility of the websites increases the ability of consumers to access information more efficiently. The accessibility score of the evaluated website is automatically generated by the tool. The usability score evaluates four main parameters: clarity, consistency, functionality, and engageability. The reliability score of the websites is defined by currency, conflicts of interest, and content production method. The questions are scored on a scale resulting in a total possible LIDA score of 144, with the maximum scores for accessibility (63), usability (54), and reliability (27). The LIDA scores are classified as high if the score is ≥90%, moderate if the score is between 50% and 90%, and low if the score is <50% according to Livas et al. (6).

As previously mentioned, web accessibility refers to targeting to inhibit obstacles that prevent interaction with or access to websites by people with disabilities. All users have equal access to information if websites are correctly designed. In addition to the LIDA instrument, there are a variety of instruments to detect access errors. AChecker (v0.1 ATutor, 2011) is a tool developed in 2009 by the Inclusive Design Research Centre of the University of Toronto. The tool provides evaluation options using the website’s URL, HTML source, or HTML file. The user can choose among the guidelines to evaluate it against, such as HTML Validator, BITV, Section 508, Stanca Act, Web Content Accessibility Guidelines (WCAG) 1.0, or WCAG 2.0. The WCAG 2.0 comprises a list of recommendations for making content accessible to a wider range of people with visual, cognitive, motor, auditive, or speech disabilities. The tool automatically generates a general and a detailed report of the accessibility problems based on the selected guidelines (7). The detailed report identifies three kinds of access problems:

1. Known problems: These accessibility hurdles are precisely defined. It is necessary to design the page from the beginning to be able to correct it.

2. Likely problems: These are probable barriers requiring a human being to decide.

3. Potential problems: These are problems that cannot be detected by the AChecker and require human judgment. It might be necessary to modify the webpage or just confirm that the problem is not present.

The aim of the present study was to evaluate the websites in English and in Turkish, providing orthodontic information in different perspectives. Three measurement tools have been used: DISCERN to evaluate the quality of information; LIDA to interpret accessibility, usability, and reliability; and AChecker to detect access errors.

All of these three tools were accessible via the Internet and were proven to be reliable, up-to-date, easy to use, and cost free.

METHODS

Internet access was established from Bezmialem Vakif University’s wireless connection. A list of possible popular keywords, which might be used by lay people to obtain information regarding orthodontics, was made in Turkish. After reviewing the list and excluding words that are too specific, five keywords were determined with the consensus of the two authors: “orthodontic appliances (ortodontik aygıtlar),” “orthodontic braces (ortodontik braketler),” “orthodontic elastics (ortodontik elastikler),” “orthognathic surgery (ortognatik cerrahi),” and “orthodontic treatment (ortodontik tedavi).” A literal translation was made for English terms.

Google (www.google.com) was used to display websites related to orthodontic terms from January to March 2017. Fifty (25 in Turkish and 25 in English) websites were evaluated for each term using three tools (DISCERN, LIDA, and AChecker). A total of 137 (72 in Turkish and 65 in English) different websites were included in the study. One operator scored all the websites.

The search was limited to the English and Turkish languages only. The results that were not related to the searched term or those that linked to other websites were excluded as well as advertisements and sponsored links. News, photos, videos, magazines, discussion or forum groups, and duplicate sites were also eliminated.

Statistical analysis was performed using the Statistical Package for the Social Sciences for Windows 22.0 (IBM Corp.; Armonk, NY, USA). The Mann-Whitney U test was used for comparison of two non-normally distributed groups. The Friedman test was applied for intragroup comparisons. The Wilcoxon rank test was utilized to determine the difference between the intragroup recurrent measurements. The results were evaluated at a 90% confidence interval. p<0.05 was considered as statistically significant.

No approval from the ethics committee was required in the present study since it involved the analysis of freely available online information.

RESULTS

The mean overall score of the information quality was classified as “good” (Turkish terms: 53.7±2.07 and English terms: 52.1±1.49), and the mean overall score of the LIDA tool was “moderate” (Turkish terms: 78.9±2.09 and English terms: 82.3±5.83). Accessibility errors detected with the AChecker in orthodontics-related Turkish websites (21.2±9.44) were higher than those English websites (18.4±1.66) (Table 1).

Table 1.

Mean scores related to terms in Turkish and in English

Terms in Turkish	“Ortodontik aygıtlar”	“Ortodontik braketler”	“Ortognatik cerrahi”	“Ortodontik elastikler”	“Ortodontik tedavi”
DISCERN	53.6+6.33	53.8+6.60	51.8+5.52	52.3+4.15	57.1+4.80
LIDA	78.1+5.88	78+7.73	81.8+7.17	76.5+8.94	80.3+10.83
AChecker	9+2.81	16.2+6.06	26.8+3.45	33.5+4.71	20.4+6.68
Terms in English	Orthodontic appliances	Orthodontic braces	Orthognathic surgery	Orthodontic elastics	Orthodontic treatment
DISCERN	53.2+3.82	53.7+6.16	52.5+8.14	50.9+5.15	50.2+7.74
LIDA	78.6+9.42	76+7.52	88+6.38	78.4+6.86	88.6+10.91
AChecker	46.1+4.98	26+6.19	29+7.36	20.6+2.27	24.7+3.64

The highest scores among the Turkish terms were reported for “orthodontic treatment” in terms of information quality. “Orthognathic surgery” had the higher scores related to LIDA, whereas the lowest access errors were encountered with “orthodontic appliances” (Table 1).

The highest scores among the English terms were reported for “orthodontic braces” in terms of information quality. The highest scores were recorded for “orthognathic treatment” for the evaluation using the LIDA tool, whereas the lowest access errors were encountered with “orthodontic elastics.”

A comparison of the Turkish and English websites revealed that statistically significant differences were present for “orthodontic braces” in terms of information quality (DISCERN, p=0.007) (Table 2), but no statistically significant differences were found for the LIDA and AChecker tools (Table 2). In-group evaluations for Turkish terms showed statistically significant differences in terms of information quality (p=0.000) and access errors (known problems, p=0.046 and likely problems, p=0.000) (Table 3). A statistically significant difference was determined in terms of information quality (p=0.000), information reliability (p=0.015), and access errors (likely problems, p=0.001) for English terms (Table 3).

Table 2.

Comparison of the DISCERN, LIDA and AChecker scores between the groups

DISCERN		Orthodontic appliances	Orthodontic treatment	Orthodontic elastics	Orthodontic braces	Orthognathic surgery
Terms in Turkish (n=25)	Median	37	38	49	44	56
	Minimum	19	24	43	25	40
	Maximum	64	60	58	63	76
Terms in English (n=25)	Median	37	44	49	49	53
	Minimum	27	30	43	30	41
	Maximum	52	55	62	72	72
	p	0.977	0.171	0.969	0.007*	0.741
LIDA
Terms in Turkish (n=25)	Median	103	107	93	107	102
	Minimum	65	61	77	60	75
	Maximum	127	134	132	131	134
Terms in English (n=25)	Median	104	100	109	96	96
	Minimum	77	76	54	50	72
	Maximum	125	135	137	129	133
	p	0.854	0.655	0.503	0.454	0.426
AChecker
Terms in Turkish (n=25)	Median	14	22	22	14	16
	Minimum	3	1	1	0	2
	Maximum	89	56	56	43	59
Terms in English (n=25)	Median	9	22	22	21	10
	Minimum	0	0	0	2	0
	Maximum	98	125	125	55	59
	p	0.072	0.93	0.655	0.091	0.641

Mann-Whitney U test, p<0,05 was considered as statistically significant

Table 3.

Intragroup comparison of DISCERN, AChecker and LIDA scores

		Information Quality (DISCERN)	Access Errors (AChecker)		LIDA
			Known problems	Likely problems	Accessibility	Usability	Reliability
Terms in Turkish (n=25)	p	0.000**	0.046*	0.000**	0.814	0.531	0.715
Terms in English (n=25)	p	0.000**	0.177	0.001**	0.425	0.271	0.015*

Friedman test, p<0,05 was considered as statistically significant

DISCUSSION

The Internet is an established portal of information. The number of websites providing healthcare information on the Internet has dramatically risen in the past years. Nowadays, there are no standards required for health information on the Internet (8). Taking advantage of this, some websites that appear to be educational are promotional in nature, whereas others may be inefficient, incomplete, not easy to understand, or may contain conflicting information (9, 10).

Although there are many engines that allow you to search the Internet, we listed the results derived from a single search engine (Google). Previous authors who evaluated online health-related information preferred using single or multiple search engines. We used Google for our search because it provides a larger number of results for keywords compared with other search engines. Moreover, it is the most popular search engine (10). Nevertheless, more than one search engine can be used in future studies.

A total of 137 websites in two languages (72 in Turkish and 65 in English) were evaluated in the present study. Based on the statement by Aldairy et al. (8) that patients would not visit more than the first 20 results in a regular search, 25 websites are chosen to be assessed for each term. A total of 50 webpages were evaluated for each term in both languages. Exclusion criteria included those web-based studies with similar aims (11). Previous studies that have focused on orthodontic terms have yielded 21 to 49 websites fitting the criteria for evaluation (12, 13).

One of the assessment tools used in the present study was DISCERN, which has been demonstrated as an effective tool to evaluate the information quality (14). The results of our study indicated that few websites showed high standards in information quality. Although DISCERN has been previously criticized for not analyzing insignificant detail when compared with other tools, the tool was proven to have good internal consistency (15). The mean total DISCERN score for the terms in both languages can be classified as “good,” with 52.9/80 (66.1%). The total DISCERN scores for Turkish and English websites were 53.7 and 52.1, respectively (Table 1). None of the websites was scored as excellent. These results indicate that when building practice websites, the designers can focus on the excellent level for the quality of information, and they can aim to ameliorate the information quality of the new websites.

The mean total LIDA score for all of the websites included in the study was 80.6/144 (55.9%), which could be classified as “moderate.” The total LIDA scores for Turkish and English websites were 78.9 and 82.3, respectively (Table 1). Other studies that evaluated websites providing information about orthognathic surgery, orthodontic pain, and oral hygiene instructions with fixed appliances using LIDA reported a similarly moderate level of the mean score (13, 16). In our study, the highest score of the LIDA tool was found for accessibility, and the lowest was found for reliability, similar to other studies (17).

The use of the Internet by lay people will hold a preponderant importance in the near future. Considering our results, even though the Internet is a powerful tool for people requiring orthodontic information, there are many issues that should be ameliorated in many perspectives. Table 4 shows some previous studies in the fields of dentistry and medicine evaluating online informative pages using the DISCERN and LIDA tools.

Table 4.

Former studies using the LIDA and/or DISCERN instruments

Author(s)	Release year	Search engine	Number of the evaluated websites	Measurement tools and scores	Searched terms
Berland and Elliott (19)	2001	10 search engines in English and 4 in Spanish	25 websites from each search engine	Fry Readability Graph. Method (86% high school level) and key words for information quality	Depression, obesity, breast cancer and childhood asthma (in English and in Spanish)
Patel and Cobourne (11)	2011	Google and Yahoo	50	LIDA (93)	Orthodontic extractions (in English)
Aldairy et al. (8)	2012	Google, Yahoo, and Ask.com	25	DISCERN (21–64)	Orthognathic surgery Jaw surgery (in English)
Som and Gunawardana (4)	2012	Google	10	DISCERN (56.1)	Chemotherapy information (in English)
Livas et al. (6).	2013	Google, Bing, Yahoo, Ask.com, and AOL	25	LIDA (16,9%–86,2%)	Orthodontic pain and braces pain (in English)
Patel and Cobourne (5)	2015	Google	100	LIDA (110) DISCERN (48)	Orthodontic braces (in English)
Shital Kiran et al. (19)	2015	Google	36	DISCERN (16–55)	Thumb sucking habit (in English)
Doğramacı and Rossi-Fedele (20)	2016	Google, Yahoo, Ask.com, Web Wombat, and Bing	200	LIDA (72%) DISCERN (47%)	Orthodontic retainers (in English)
McMorrow and Millet (17)	2016	Google, Yahoo and Bing	13	DISCERN (3,9/5) LIDA (115)	Adult orthodontics and adult braces (in English)
Canıgür Bavbek et al. (21)	2017	Google	25	DISCERN (28/75)	Orthognathic surgery (in Turkish)
Present study	2017	Google	137	LIDA (78,9/82,3) AChecker (21,8/18,4) DISCERN (53,7/52,1)	Orthodontic appliances, orthodontic braces, orthodontic elastics, orthognathic surgery and orthodontic treatment (in Turkish and in English)

In many countries, there are steps taken to ensure that differently abled citizens have the same rights as others. With the recognition and growth of the W3C, special laws regarding the use of the Internet for people with disabilities have been established. For example, in 2004, the government of Italy approved a new legislation designed to provide differently abled citizens access to online services.

There are a variety of tools to evaluate accessibility. However, most of them are used to evaluate images and screenshots (e.g., Cynthia Says and Accessibility Valet). Some of these tools require payment, whereas the tool used in the present study, AChecker, is free of charge and allows limitless use. Moreover, AChecker is easy to use and subjectively reports the accessibility errors in three domains (known, likely, and potential problems) since it automatically generates the reports. Even though there are studies evaluating general health websites accessibility, there is no previous study about the access errors of websites providing information about orthodontic terms. Therefore, our results regarding this tool are unique and cannot be compared.

Similar to previous studies, the results of the present study provide the “snapshot” of the websites at one point in time. Websites are constantly being updated, and their content can change over time, suggesting the dynamic feature of the Internet. This is the reason why we can find different results if the same search is conducted at a future date. Although there is a consensus from previous studies that improvements in online orthodontic information are required, no specific guidance on how to address these improvements has been documented (6, 8, 11). On the other hand, the American Medical Association has established guidelines that websites should follow (18). A similar international guideline could be developed regarding online dental or orthodontic online sources. Orthodontic societies should collaborate to create reliable online sources about commonly searched orthodontic topics. The Internet resources should be developed with reference to the validated tools (LIDA and DISCERN) and in conjunction with informatics experts.

Only one operator made all the evaluations in the present study. Even though the tool’s intra-operator reliability has been demonstrated to be good, we could include more examiners to provide a more objective evaluation.

Most similar studies evaluated terms in only one language (generally in English) (8, 17, 19–21). Only one study was performed with terms in two languages (English and Spanish) to evaluate terms such as depression, obesity, breast cancer, and childhood asthma (22). To our knowledge, this is the first bilinguistics study in the field of dentistry and, more specifically, in orthodontics to evaluate orthodontic information on the Internet. Furthermore, other terms in other languages can be incorporated into a future study to perform an international overview.

Based on these results, we suggest that websites can be checked for usability, quality of information, and access errors with suitable measurement tools before being made available to the public. Official organizations having no financial benefit can consider establishing norms for the websites providing health-related information, and an audit system supported by the government can be established. An alternative could be to create state-supported informative websites and question-and-answer forums. Public spots could direct patients to those perfectly designed, up-to-date websites to produce high-quality online services. Moreover, the awareness of accessibility issues among web developers and clinicians should be increased since problems could be easily solved if they are recognized.

CONCLUSION

Accessibility, usability, and reliability (LIDA), quality of information (DISCERN), and access errors’ scores showed variations among Turkish and English sites. The scores of English websites in information quality for “orthodontic braces” are significantly higher than those of Turkish websites. The average score for the information quality of the English and Turkish terms was reported as “good,” and the average LIDA score was found to be “moderate.”

People searching for information about orthodontics on the Internet should be aware that the information they find may not be appropriate or reliable. The collaboration of website designers and clinicians to increase the quality level of the websites is recommended.