The Quality of Pediatric Orthopaedic Information on the Internet

Abstract

Background

Many patients use the Internet for health information. However, there are few guarantees to the reliability and accuracy of this information. This study examined the quality and content of Internet webpages for ten common pediatric orthopedic diagnoses.

Methods

We identified ten common diagnoses in pediatric orthopedics: brachial plexus injury, cerebral palsy, clubfoot, developmental dysplasia of the hip, leg length discrepancy, osteochondroma, polydactyly, scoliosis, spina bifida, and syndactyly. We used two of the most popular search engines to identify the top ten websites for each disease. We evaluated the websites utilizing both the quality-based Health On the Net Foundation criteria and our own content-based grading sheets. The custom grading sheets focused on essential information about disease summary, pathogenesis, diagnosis, treatment, and prognosis.

Results

Three Orthopedic surgeons graded 98 academic, commercial, non-profit and physicians’ websites for ten diseases. Academic websites scored the highest in content (mean 60.8%±15.5), while commercial websites scored the lowest (mean 46.7%±22.2). Among the diagnoses, osteochondroma websites had the highest content scores (mean 75.8%±11.8), while polydactyly websites had the lowest content scores (mean 39.3%±15.7). In contrast, websites about developmental dysplasia of the hip had the highest HON scores (65.0±11.1), while those about brachial plexus birth palsy scored the lowest (42.6%±16.9). Among the content subgroups, scores were generally higher for disease summary and diagnostics and lower for prognosis.

Conclusions

The Internet websites reviewed demonstrated a wide range of content and information. We found that non-profit and academic websites were the most reliable sources while commercial and, surprisingly, physician-run websites were the least reliable. We advise physicians to talk with their patients about the information they get on Internet and how it dictates their expectations. We hope this study, combined with further understanding of how our patients use this information, can help improve Internet content.

Clinical Relevance

Physicians should know their patients may be receiving misleading information from the Internet and be able to discuss this with their patients.

Introduction

There is a wealth of health-related information available on the Internet and Internet use among adults has dramatically increased over the last decade. According to a Pew study conducted in 2010 (1), nearly 173 million Americans utilize the Internet to look for health information. The survey showed that adults rely on search engines to find relevant websites and the vast majority of adults think that the information found on these websites is trustworthy (2). Furthermore, up to 90% of people believe that the health information they find on the Internet is reliable (3). Patients use this health care information to investigate treatment options, to ask new questions, to get second opinions, and to influence their decision to visit a doctor (4), (5).

This trend is consistent for parents of pediatric orthopaedic outpatients. Seventy-four percent of parents find the Internet useful for medical information and would recommend it to others (6). While patients use their physician as a primary source of information, use of the Internet as a secondary source of information is increasing, and fewer patients are relying on friends/family, print media, the radio, or television (7).

The Internet is not actively policed or monitored by any agency for accuracy. Anyone can create a website and there are over 1 billion websites available to the American web user (8). Some initiatives have been taken to assure web users that information is credible and accurate, such as the non-profit Health On the Net Foundation (HON). While the HON seal does not guarantee that all the information is thorough and accurate, it is intended to identify websites that provide objective and transparent information from a qualified professional (9).

There have been numerous studies on the quality of healthcare information websites in the field of orthopedics. Some studies focus on specific conditions, such as cervical disk herniation (10) and knee arthroscopy (11), while others are in orthopedic fields, such as sports medicine (12), orthopedic oncology (13), and foot and ankle surgery (14). These studies found that the quality of information on the Internet is variable and generally poor. The goal of our study is to assess the accuracy and completeness of websites that families may use for information on ten common pediatric orthopedic diagnoses.

Materials and Methods

We used billing records to identify the 10 most common new diagnoses from the pediatric orthopedic service at our institution. These diagnoses are: brachial plexus injury, clubfoot, syndactyly, cerebral palsy, developmental dysplasia of the hip, osteochondroma, leg length discrepancy, scoliosis, polydactyly, and spina bifida.

We searched two of the most popular search engines, Google™ and Bing™ (2), and identified the top ten search results for each condition. Lay terms were used whenever possible. Both Google™ and Bing™ searches redirect misspelled words to the appropriate websites, so misspelled searches were not attempted. Three orthopedic surgeons individually graded 98 websites. We excluded websites that were duplicates on both search engines, required a user name/password, had videos as the primary source of information, contained information unrelated to the disease such as veterinary pages, and those that required multiple different web pages to find information. A total of 102 websites were excluded.

A custom content grading scale was created for each disorder. The grading scale was designed to assess the quality and completeness of information presented on each website, and was modeled after Smith et al (14). The scale focused on five weighted categories with a maximum score of 100 points: 30 points for summary of the disease, 20 points for pathogenesis and genetic influence, 10 points for diagnostic tests, 30 points for treatment and complications, and 10 points for prognosis and outcomes. We used the American Academy of Orthopaedic Surgeons Comprehensive Orthopaedic Review as a guide to create these grading scales (15).

We also evaluated each website according to the HON criteria for quality. This score has a maximum of 16 points: 5 points for transparency of information including accessibility and valid contact details, 5 points for stating the qualifications of the authors, 1 point for stating the privacy policy of user information, 1 point for stating when the information was updated, 3 points for accountability, and 1 point for accessibility (16). We converted the 16-point scale to a percentage score out of 100 for uniformity.

The types of websites were categorized as commercial, academic, non-profit, or physician, according to the HON code. The reviewers first categorized each website. There was one website that the reviewers did not initially classify the same, but the reviewers met and agreed upon classification. The reviewers then graded each website according to our custom content grading scale and the HON scale.

Statistical analysis

To investigate the quality and content of the websites available for our chosen diagnoses, we first examined the HON scores and content scores separately. A summary statistic for HON score was generated for each reviewer. An intraclass correlation coefficient (ICC) was conducted to measure the inter-rater variability of the HON score among the three reviewers. The mean and standard deviation for HON scores based on diagnosis, website and subgroup were calculated for each reviewer and overall, respectively. The content scores were analyzed in a similar manner.

We combined and averaged HON scores and content scores from the three reviewers, then analyzed their association using a Pearson product-moment correlation coefficient. All the statistical analyses were conducted in R version 2.10.1 for Windows with a significance level of 0.05 for hypothesis testing.

Results

A total of 98 unique websites were independently evaluated by three reviewers. There were 33 academic websites, 30 commercial websites, 31 non-profit websites and 4 physician websites.

HON Scores

The total HON scores had a maximum of 16 points and websites ranged from 1 to 15 points. Scores were converted to a percentage for easier interpretation. Comparing the HON score variability among the three reviewers, eighty-four percent of the websites had a score within 25 percentage points, and all but four websites had a score within 31 percentage points. An ICC of 0.725 with a 95% confidence interval (CI) of [0.584, 0.818] indicated strong agreement on grading HON scores across three reviewers. Mean HON scores from individual reviewers for each diagnosis ranged from 35.6 to 66.7. Developmental dysplasia had the highest HON scores across the three reviewers (65.0 ± 11.1), while brachial plexus birth palsy had the lowest HON scores across the three reviewers (42.6 ± 16.9). Out of the four types of websites, non-profit websites had the highest average HON score of 67.9 with standard deviation of 14.8, while physician websites had the lowest average HON scores of 41.1 with a standard deviation of 16.0.

Content Scores

Individual content scores ranged between 3 and 100 points. For the content score variability among the three reviewers, an ICC of 0.821 with a 95% CI of [0.751, 0.874] indicated very strong agreement. The average content scores based on diagnosis ranged from 31.0 to 78.8. Osteochondroma had the highest content scores across the three reviewers, while polydactyly had the lowest content scores. Scores of the five content subgroups (disease summary, pathogenesis, diagnostics, treatment and complications, and prognosis and outcomes) ranged from 32.1% to 66.3% of the maximum possible score for each subgroup and were consistent among reviewers. Scores were generally higher for disease summary and diagnostics, and lower for prognosis and outcomes, and pathogenesis.

Combining data by averaging scores of three reviewers, academic websites had the highest average content scores of 60.8 with standard deviation of 15.5, and commercial websites had the lowest average content scores of 46.7 with standard deviation of 22.2.

Comparison of HON and Content Scores

Only 15 websites had an HON score of 75 percentage points or higher. These “high-quality” websites had an average content score of 62.4, where the 83 “low quality” websites (those with an HON score of less than 75 points) had an average content score of 52.8. A Pearson correlation test was conducted to investigate the relationship between the averaged HON score and content score. A p-value of 0.0002 and a Pearson correlation coefficient of 0.367 indicated a weak positive correlation between the average HON score and content score.

Discussion

The Internet is a quick, convenient, anonymous source with an abundance of health information. However, identifying accurate and complete information is difficult. Patients often use commercial search engines to find information and are more likely to critique the credibility of a website based on its “design looks”, as opposed to the source of the information or the presence of advertising (17).

Our study evaluated 98 websites among ten common pediatric orthopedic diagnoses and judged them utilizing HON and content criteria. Like past studies that evaluated orthopedic information available on the Internet, we found that overall information quality was poor and variable among websites (8, 11, 12, 14, 18). We found numerous duplicate search results between both Google and Bing, indicating that despite different search methods, patients are likely to encounter the same poor-quality information. In addition, several websites contained text and figures identical to those we found on other websites. This is troubling for two reasons. First, these may represent plagiarism of one site by another. Second, duplication between sites may seem to legitimize incorrect or misleading information in the eyes of the patient who sees this information repeatedly.

The Health On the Net (HON) foundation was created to help guide people to high quality websites. Our finding that the HON score correlated with our content score indicates that the concept of the HON score is valuable in judging pediatric orthopaedic websites. However, we found that out of the 15 websites that were deemed high quality (receiving a 12 out of 16 on the HON rubric), none displayed the HON seal, making it difficult to recommend to patients that they look for the HON seal while searching the Internet. More troubling is that the physician-run website group had the lowest average HON score. This could represent a simple lack of knowledge on the part of the physicians who create websites about the elements of a reputable website (such as a privacy policy and a funding disclosure). Unfortunately, it is also possible that some physician websites might be intentionally opaque and may not even be authored by physicians.

We found that the commercial websites consistently scored the second lowest based on the HON criteria and the lowest based on content scores. We suspect that financial incentives associated with commercial websites could lead to the poor quality and content found on these websites. For example, the diagnoses of brachial plexus injury and leg length discrepancy had the lowest HON scores. Nearly half of the websites evaluated for this condition were commercial in nature.

Looking at the quality of specific types of information, the categories of diagnosis and disease summary scored the highest. Broad, non-specific website information quality was the best. The category of prognosis scored the worst, with an average score of 37.3 ± 30.3, possibly due to the patient-specific nature of prognosis. This misinformation could potentially place undue stress on parents about long-term prospects of disease management. Studies have shown that patient satisfaction with their surgical outcomes is related to expectations and preparedness (19),(20). With patients getting potentially faulty information from the Internet, a larger burden is placed on orthopaedic surgeons to properly inform their patients.

This study has implications on the physician-patient relationship. While more people are utilizing the Internet for information, 82% of patients either never or only sometimes speak with their physician about information they find on the Internet (3). Physicians must inquire where patients are receiving their information, be informed about common websites that patients are likely to use, and inform patients about reliable sources of web information. A poll of 1200 patients found that 70% of patients would like their physician to recommend website information while just 4% received this information (18).

There are several weaknesses to this study. We only graded the top ten websites for each search engine. A similar study by Green et al. found the quality of information on websites outside of the top ten found by a search engine to be worse (18). The implication for this study is that the content and HON scores reported here may overstate the quality of available websites. Furthermore, it is possible that patients use other search engines or media to find their information. Another potential weakness is that although we tried to use non-jargon medical terms, we only used one search term for each diagnosis. Patients could use alternative search terms that might lead to a different set of websites. A further study would be to investigate blogs and user contributed websites, such as forums or social media sites, as patients also use these as sources of information (3).

In conclusion, similar to other studies of orthopedic information on the Internet, we found that websites about common pediatric orthopaedic conditions had a wide variation of content and quality. Physicians should be open to discussing with patients the information they find on the Internet, and should be knowledgeable about websites containing quality information. While we cannot recommend any specific sites based on this study, in general physicians should direct patients to academic and non-profit websites.