Informational value and bias of videos related to orthodontics screened on a video-sharing Web site

Abstract

Objective:

To assess the informational value, intention, source, and bias of videos related to orthodontics screened by the video-sharing Internet platform YouTube.

Methods:

YouTube (www.youtube.com) was scanned in July 2010 for orthodontics-related videos using an adequately defined search term. Each of the first 30 search results of the scan was categorized with the system-generated sorts “by relevance” and “most viewed” (total: 60). These were rated independently by three assessors, who completed a questionnaire for each video. The data were analyzed statistically using Friedman's test for dependent samples, Kendall's tau, and Fleiss's kappa.

Results:

The YouTube scan produced 5140 results. There was a wide variety of information about orthodontics available on YouTube, and the highest proportion of videos was found to originate from orthodontic patients. These videos were also the most viewed ones. The informational content of most of the videos was generally judged to be low, with a rather poor to inadequate representation of the orthodontic profession, although a moderately pro-orthodontics stance prevailed. It was noticeable that the majority of contributions of orthodontists to YouTube constituted advertising. This tendency was not viewed positively by the majority of YouTube users, as was evident in the divergence in the proportions when sorting by “relevance” and “most viewed.”

Conclusion:

In the light of the very large number of people using the Internet as their primary source of information, orthodontists should recognize the importance of YouTube and similar social media Web sites in the opinion-forming process, especially in the case of adolescents.

INTRODUCTION

The Internet is a source of health-related information that is being used increasingly often by both health care professionals and nonexperts.1–7 Recent research has revealed that social networks and video-sharing Internet platforms are gaining importance and influence in relation to medical matters and opinion formation by laypersons,2 particularly adolescent users.8 Whereas previous analyses have established the value of such videos as a source of useful information regarding chronic diseases such as epilepsy3 or multiple sclerosis,9 another aspect of the value of the Internet is the speed with which information about outbreaks of epidemic diseases, such as H1N1 influenza,10 or contemporary issues and concerns, such as how human papilloma virus11 is spread.

For those with no access to research data or who are not familiar with such databases as PubMed (www.ncbi.nlm.nih.gov/pubmed), eg, adolescents, Web sites such as YouTube (www.youtube.com) may be of particular importance and have a substantial influence when adolescents seek health-related information.12 With regard to orthodontics, and especially since the majority of adolescents regularly use Internet social media for obtaining information,13 it is reasonable to suppose that those adolescents who are about to undergo orthodontic treatment make use of this popular Web site to obtain preliminary information. However, for some time now, it has not been only health care professionals who have generated health-related information on such Web sites. Social platforms enable literally anyone to share their knowledge, opinions, and experience, and there is an increasing proportion of patient-generated content as a result.9

In addition to the entertainment aspect associated with Internet video-sharing platforms such as YouTube, a further consideration is their role in advertising and providing patients and other laypersons with information,1,14,15 as well as peer-to-peer education.16–21 This is evident in the increasing number of journals and institutions, such as universities, that are independently running YouTube channels. Until now, this trend has not been assessed in relation to orthodontics. We therefore identified a need to consider this subject and chose YouTube as an example. We sampled the video-sharing Web site YouTube to ascertain how orthodontics is presented, with particular consideration to the informational value and bias of the videos screened.

MATERIALS AND METHODS

Video Material

YouTube (www.youtube.com) was scanned for orthodontics-related videos using the search term “orthodontist OR orthodontic OR orthodontics OR braces” on July 20^th and July 27^th, 2010. Search results with the category “all videos” were first sorted by relevance, then by number of views/most views by YouTube users. In both searches, the first 30 videos were evaluated (total n  =  60). The rationale for using only the first 30 videos was to obtain sufficient information about the material currently viewed by the majority of viewers, and the first 30 videos are those viewed several thousand times per day.3 To also detect those videos that are being viewed mostly by those searching for information on orthodontics on this Web site, we also analyzed the first 30 videos characterized as having the highest number of views. The total number of views and duration of each video was recorded. The mean duration of videos was 234.2 seconds and ranged from 58.0 to 624 seconds. The mean number of views (clicks) per video was 136,300 and ranged from 4244 to 810,100.

Content Assessment

Video content evaluations were independently carried out immediately following video clip retrieval by three assessors: an orthodontist, a nonorthodontist nondentist adult (ie, a person not directly involved in orthodontics, but with an adult point of view), and an adolescent aged 12 who had no history of orthodontic treatment and who served as a representative of the target groups of both YouTube and adolescent orthodontic patients. The assessors completed a questionnaire for each video (Table 1). Videos were watched at the same time and under the same conditions by all three assessors, and—to prevent communication between them—in different rooms. Video clips were classified as portraying orthodontics either positively or negatively. The other items in the questionnaire concerned the source of the video (patient, orthodontist, combination, or unclear); its information content (excellent, fair, poor); the intention of the video (entertainment, information, advertisement, or a combination) as judged by the assessor; and the presence of bias (pro-orthodontics, balanced, or anti-orthodontics). After they had viewed all the videos, the assessors were asked whether their attitude toward orthodontics or orthodontic treatment had changed.

Table 1.

Items to Be Rated for Each Video

Statistical Methods

The ordinal ratings of the assessors were compared using Friedman's test for dependent samples with significance levels of α  =  5%. In addition, pairwise correlations between each pair of the three assessors, taken in turn, were assessed using Kendall's correlation coefficient tau and Bonferroni-adjusted significance levels of .05/3  =  .017. Interassessor variability for the assessors' categorical answers was determined pairwise between the assessors using Fleiss's kappa statistic. All analyses were performed using the software R (version 2.8, www.r-project.org).

RESULTS

Selected Videos

The initial search identified 5140 videos. Some videos were excluded from further analysis after being identified as not relevant. By using the sorting term “relevance” (or “most viewed”), two videos in the first 30 (total: 60) were identified by consent of the three assessors to be unrelated to orthodontics and were eliminated from the sample and replaced by subsequent videos in the series.

Information Concerning Orthodontics

In general, the video content was rated as weak with regard to orthodontics. The mean ratings of the assessors ranged from 3.0 to 4.3 (Table 2). Ratings were not significantly different between the three assessors. The correlations between assessors' ratings were significant, although not high, and ranged from tau  =  0.49 (orthodontist versus nonorthodontist, sorting according to relevance) to tau  =  0.66 (orthodontist versus adolescent, sorting according to most viewed).

Table 2.

Comparison of Ordinal Ratings Between the Three Assessors^*

Attitudes Toward Orthodontics

Most of the videos were judged to be moderately pro-orthodontics. The mean ratings of the three assessors ranged from 2.8 to 5.3. However, when the “most viewed” sorting was used, four videos were consistently judged to be totally against orthodontics (rating  =  10) by all assessors. In the case of sorting by relevance, one video was judged to be between 8 and 10 by all three assessors. Moreover, again sorting by relevance, judgments were significantly different between the three assessors (Table 2, Figure 1), although, again, the ratings were significantly correlated (Table 3). The adolescent rated the videos significantly more often as being pro-orthodontics than the two adult assessors.

Figure 1.

Comparison of ratings for the question of whether the video appeared to be in favor of or against orthodontics.

Table 3.

Correlation (Kendall's Tau) Between Ordinal Ratings for Each Pair of Assessors^*

Credibility of the Content

Most videos were rated as credible. Mean ratings ranged from 3.2 to 4.4 (range from 1 indicating completely credible to 10 indicating not at all credible). While the ratings were again clearly and significantly correlated between the three assessors (Table 3), there was also a significant difference when using the “most viewed” sorting (Table 2, Figure 2). The nonorthodontist and the adolescent tended to judge videos to be believable more often than did the orthodontist.

Figure 2.

Comparison of ratings for the question of whether the content of the video appeared to be credible (left) and orthodontics was adequately represented (right).

Adequate Representation of Orthodontics

The question of whether orthodontics was adequately represented by the videos tended to be answered in the direction of “no.” The mean ratings ranged from 5.4 to 7.3 (range from 1 indicating completely adequate to 10 indicating completely inadequate). When videos were sorted according to the most viewed, the orthodontist rated the contents as significantly more inadequate in their representations of orthodontics than did the two other assessors (Table 2, Figure 2). Nevertheless, once more, there was a high correlation between assessors, ranging from tau  =  0.69 to 0.78 (Table 3, Figure 3).

Figure 3.

Pairwise correlation between ratings for the question of whether orthodontics was adequately represented by the video.

Sources of Videos

In the case of sorting by “most viewed,” the majority of the videos were assumed to be posted by patients, ie, 58% of all ratings assigned the origin of videos to patients. The second most frequently assumed origin was “unclear” (18%), followed by “orthodontist” (14%). Only 10% of videos were assumed to be posted by both patients and orthodontists. In the case of sorting by “relevance,” the distribution was not so skewed. Specifically, 34% of ratings assumed the videos were posted by an orthodontist, followed by 28% as “unclear,” 27% by patients, and 11% jointly by orthodontists and patients. In general, the level of agreement between the three assessors regarding the origin of the videos was higher than would be expected by chance (Table 4).

Table 4.

Interassessor Variability With Respect to the Sources of the Videos^*

Intention of the Videos

Multiple answers were permitted regarding the intent of the videos. Consequently, in the case of sorting according to “most viewed,” 46% of ratings included the answer “entertainment,” 36% “information,” 18% “advertising,” and 36% “unclear.” When sorted according to “relevance,” 36% of ratings included the answer “entertainment,” 54% “information,” 41% “advertising,” and 17% “unclear.” With a few exceptions, interassessor agreement was greater than that expected by chance (Table 5).

Table 5.

Interassessor Variability With Respect to the Intention of the Videos^*

The consistency of the assessors' attitudes over the course of the study was assessed. Whereas the orthodontist and adolescent assessor's opinions were found not to have changed in terms of their attitude to orthodontics, the nonorthodontist adult assessor said that his opinion had become slightly more positive toward orthodontics after the two viewing sessions.

DISCUSSION

The large numbers of both health professionals and laypeople searching the Internet for health-related information6 shows that this source of information must have a significant influence on the perception of orthodontics, especially those in their adolescence. The popularity of social platforms such as YouTube, MySpace, Facebook, or Twitter is, in part, a result of their participatory nature. Users are able to offer information while reaching a wide audience, to receive feedback, and to obtain information themselves. Information transfer is instant and easy. There are, however, drawbacks: the content may be questionable, there is a lack of information about the sources of information, and opinions may be construed as facts.11,14

The rationale for using only the first 30 videos in the categories “highest relevance” and “most viewed,” rather than screening all of the more than 5000 videos available related to orthodontics, was that the top 10 videos accessible on this Web site are viewed several thousand times a day.3 Therefore, the videos we analyzed provide an insight into what YouTube users view most often and what may influence their attitudes toward orthodontics, since it represents or even exceeds the volume that an average user of the social platform YouTube consumes. The duration of trial viewing sessions (without prior arrangements, query configuration, etc) was 2 × 4 hours, ie, a duration rarely reached by the average YouTube viewer, with a focus on only one topic and one social platform. The mean number of views of all videos screened in this study was 136,300 and ranged from 4244 to 810,100. This implies that we have considered videos seen by an audience that truly is interested in finding information about orthodontics using the Web site YouTube. Because a substantial proportion of YouTube users are pre-adult,13 orthodontists must recognize the fact that those social platforms are part of an opinion-making resource for this group, whether they like it or not. Because there has not been a similar survey concerning orthodontics to date, we have been able to provide new insight, although the results are a cause for concern. The informational content of most videos was rated as being low but moderately pro-orthodontics. However, this does not mean that there was an adequate representation of the orthodontic profession. In fact, analysis of the questionnaire revealed that this was not the case (ratings 5.4 to 7.3, Tables 1 and 3). Despite a high correlation between the assessors, there was a significant tendency for the two nonorthodontist assessors to feel that the videos more adequately represented orthodontics and were more believable (ratings 3.2 to 4.4) compared to the representative of the orthodontic profession (Tables 2 and 3, Figure 2). “Credible” in this context does not include a judgment of bias for or against orthodontics, ie, an adolescent who complains about his fixed appliances may or may not be judged as being credible, regardless of whether he or she speaks in favor of or against orthodontics. Generally speaking, the representative of the adolescent group tended to rate videos significantly more often as being pro-orthodontics than the two adult assessors. In this context, one should also take into account the fact that there was consent on the part of all assessors that almost 7% (4 of 60) of videos were biased totally against orthodontics, as was evident by a 10 rating in the pro/anti-orthodontics category.

If we now consider the sources of the videos, there was a bias depending on whether sorting was done by “most views” or “relevance.” The assumption that YouTube is used by consumers or adolescents receives support, since a high proportion (58% when sorting by “most viewed” and 27% when sorting by “relevance”) of the videos were judged to be posted by patients, while it was unclear in 18% (sorted according to “relevance,” 28%) or judged to be posted by an orthodontist, either as part of an educational measure (eg, how to clean teeth during straight-wire therapy) or for advertising purposes in 14% (sorted according to “relevance,” 34%) of videos. An additional 10% (sorted according to “relevance,” 11%) were judged to be posted by both patient and orthodontist, and these mostly contained an advertisement aspect. Generally speaking, those videos that were, based on ratings, assumed to be posted by patients were indeed most frequently viewed, whereas videos determined to be posted by orthodontists seemed to fit the search term best and therefore had a higher ranking when the sorting category “relevance” was used.

Another significant result of this study was that, while the entertainment aspect of this medium is certainly obvious (46% using the sorting criterion “most viewed,” 36% sorting by “relevance”), more than one third of the videos (36%) sorted by “most viewed” and more than half of the videos (54%) sorted by “relevance” were judged to have an informative intention. Eighteen percent of the videos sorted by “most viewed,” compared to 41% when sorting by “relevance,” were judged to have an advertising intention. When viewed together with the results for the item “originators of videos,” which revealed that there was a higher “advertisement proportion” when sorting by relevance, it may be concluded that advertising about orthodontics is present but is neither preferred nor liked to be viewed by a majority of viewers, as is evident by the distinctively smaller proportion found when sorting by most viewed.

Possible Implications and Suggestions

It is hardly possible to have a direct influence on what patients upload on YouTube; it is a positive feature of our society that each individual is free to speak his mind and to express an opinion. Overall, the videos screened here had a pro-orthodontic stance, and this reflects the fact that the orthodontic profession is perceived as beneficial by the public. However, in relation to the videos uploaded by orthodontists, it would be helpful with regard to public perception of the orthodontic profession if the same basic principles adopted for scientific publishing were to be applied here. This means that opinions should not be presented as facts, since this would not contribute to orthodontists' credibility in the long term. There should be no attempt to sell a specific appliance (eg, aligners, self-ligation brackets) as the unique appliance that can only be supplied by the office of the orthodontist that uploaded the video. It might be better, in terms of credibility, if those who upload videos were to keep to scientifically supported facts. This could, of course, be accompanied by advertising of a particular office or business (which is naturally the motivation of orthodontists in uploading videos), but it should be done in a serious manner and, ideally, also in an attempt to provide information with an educational content.

The author is convinced that, in general, the representation of a profession will undergo a change in its “image” on these types of Web sites when adolescents who are now growing up with social networks themselves make use of such networks as professionals. Also, the still small numbers of scientific journals currently independently running social media Web site channels themselves—and which thereby have an additional educational effect—are likely to increase.

Study Limitation

Although the large numbers of views hint at an elevated interest in YouTube videos with orthodontic content, it cannot be ascertained that viewers are watching those videos for reasons related to orthodontics. We assume that both the time during which a video is being watched and also the reasons for watching it depend very much on the specific content of the video and, in particular, on the educational background, mood, and specific situation (eg, whether the viewer is about to receive orthodontic treatment soon) of the person who is watching.

The study focused primarily on the determination of agreement between different assessors. To compare whether different groups of assessors (eg, orthodontists versus nonorthodontists) judge particular videos differently, a larger study with a larger number of assessors per group would be necessary.

CONCLUSIONS

• There is a sizeable audience that is interested in obtaining information about orthodontics using the participatory social platform YouTube. Orthodontists should recognize the importance of this and similar social media Web sites in relation to opinion-making processes, especially with regard to adolescents.

• A wide variety of information about orthodontics is available on YouTube. The greatest proportion of videos was found to be posted by orthodontic patients. These videos were also found to be the most viewed ones.

• The informational content of most videos was judged to be quite low, with a poor to inadequate representation of the orthodontic profession.

• There is a considerable amount of advertising by orthodontists on YouTube. However, this is not preferred by the majority of YouTube users, as was evident in the pronounced discrepancy in the proportions when sorting by “relevance” or “most viewed.”