Assessing the accuracy and quality of YouTube videos on early pregnancy loss

Abstract

Objective

To assess the accuracy and quality of YouTube videos pertaining to early pregnancy loss for use as a patient education tool.

Methods

A cross-sectional study was conducted via YouTube search using the keywords “miscarriage,” “spontaneous abortion,” “pregnancy loss,” and “pregnancy failure.” The first 20 results for each keyword search, sorted by both relevance and view count, were compiled into a list. Descriptive characteristics, including the numbers of views, likes, dislikes, video length, and duration of upload were collected. All videos were independently evaluated by two physician researchers using two unique assessment tools. The Currency, Relevance, Authority, Accuracy, and Purpose (CRAAP) test was used to measure the reliability of video content. The Miscarriage-Specific Question Score (MSQS) criterion was used to objectively assess video content specific to miscarriage. Inter-rater agreement was analyzed via kappa coefficient and Pearson correlation.

Results

160 videos were screened, among which 74 videos were included for analysis. The mean CRAAP score was 8.3 out of a total possible score of 15, demonstrating good quality sources, though not of academic level. Mean MSQS score was 8.1 out of a total possible score of 24, demonstrating “fair” accuracy and comprehensiveness. Pearson correlations were 0.87 and 0.86 for CRAAP and MSQS total scores, respectively, demonstrating excellent inter-rater reliability.

Conclusion

YouTube videos related to early pregnancy loss span a wide range of quality, accuracy, and purpose. While some videos provide effective content, mean rater scores demonstrate that YouTube is not a reliable source for patient education on early pregnancy loss.

Early pregnancy loss; Miscarriage; Spontaneous abortion; Patient education; Social media.

1. Introduction

Patients increasingly rely on the internet to obtain medical information [1, 2, 3]. A National Health Interview Survey of US adults showed that 44% of respondents use the internet to obtain health-related information, and that 8 out of 10 internet users accessed health information online [1]. A multicenter survey of contraceptive knowledge among adolescents in North America showed that over 50% of young adults used the internet as the primary source of contraceptive information [4].

YouTube is the second-most visited website on the internet [5] and is one of the largest, free sources of video content online. YouTube has over 2 billion active users each month, and every day people watch over a billion hours of video [5]. While YouTube has become a popular venue for patients to obtain medical information [6], videos on YouTube are not subject to quality control measures and, thus, may be of varied quality and integrity. Studies analyzing video quality, type, content, and accuracy, are needed to understand whether patients are at risk of receiving misinformation and making suboptimal medical decisions. Such studies will also be helpful for healthcare providers to better understand medical decisions made by patients to improve their quality of care.

Early pregnancy loss affects hundreds of thousands of women every year [7, 8]. Approximately one-third of pregnancies end in early pregnancy loss [8, 9], which can have significant physical and psychological consequences for women. Early pregnancy loss is frequently a distressing event for both women and their partners, which may result in depression, anxiety, and grief for those affected [10, 11].

Adequate patient comprehension of early pregnancy loss is essential for optimal patient care. No studies to date have analyzed YouTube videos specific to early pregnancy loss. The purpose of this study is to analyze YouTube videos specific to early pregnancy loss and evaluate their accuracy and quality as a patient education resource.

2. Methods

This was a cross-sectional study performed via search of YouTube videos. A video search on YouTube was performed on May 12th, 2020 by using the keywords “miscarriage,” “spontaneous abortion,” “pregnancy loss,” and “pregnancy failure.” The search was done using a cleared-cache web browser in incognito mode to prevent influence on search results from prior personal search histories. Videos were sorted by both relevance and view count in descending order. The first 20 videos by each sorting method, for each keyword were saved by compiling the hyperlinks on a spreadsheet for data collection. . Our search was limited to the first 20 videos per keyword search. At the time of data collection, YouTube displayed 20 search results per page, and internet users seldom go beyond the first page of search results to find health information [12].

Videos that met the following inclusion criteria were included in the study: (1) specific to first trimester pregnancy loss, and (2) in English. Videos were excluded if the following criteria applied: (1) not in English, (2) specific to second or third trimester pregnancy loss, (3) duplicate videos, (4) irrelevant videos, (5) videos over 45 min in length. Inclusion and exclusion criteria were determined prior to video search and data collection.

If a video did not explicitly state the gestational age of discussion, the video was included if it provided information reasonably pertinent to first trimester pregnancy loss. Videos were considered irrelevant if entirely off-topic from early pregnancy loss. Examples of irrelevant videos that were encountered during our search included “vasectomy failure” and “cardiac disease in pregnancy.” We excluded videos over 45 min as recent publications highlight the importance of short presentations to engage the audience [13, 14, 15].

The following characteristics were recorded for each video on the day of video collection: (1) number of views, (2) number of likes, (3) number of dislikes, (4) video length, and (5) duration of upload. Videos were categorized by authority: “medical provider,” “institution,” “patient,” “company,” or “unknown.” Videos were also categorized by purpose: “educational,” “news report,” “advertisement,” “testimonial,” or “other.”

All videos were independently evaluated by two OB/GYN physician researchers (HS and LM) who have clinical experience with early pregnancy loss. Both researchers reviewed each included video from beginning to end. Each researcher was blinded to the other researcher’s video scoring during the video review period.

The CRAAP test and MSQS criteria were used as assessment tools for each video. The CRAAP (Currency, Relevance, Authority, Accuracy, and Purpose) test, developed by Sarah Blakeslee and her team of librarians at California State University [16], is used to measure the reliability of video content. The CRAAP test has been used both as a teaching tool in academic settings [17, 18, 19], as well as an objective method to evaluate data sources in a variety of other medical studies [20, 21, 22]. For each of the five components of the CRAAP test (currency, relevance, authority, accuracy, and purpose), videos are scored 0–3 points according to the criteria shown in Table 1. Each category is weighed equally. The maximum total score possible is 15 points. A final score of 0–3 points indicate a questionable source of information and is likely not reliable; a score of 4–7 points indicate information may be reliable, but caution should be used; a score of 8–11 indicates a good source of information; and 12–15 points indicate an excellent source of information.

Table 1.

Currency, relevance, authority, accuracy, and purpose (CRAAP) test scoring Criteria.¹³.

Score	0	1	2	3
Currency	There is no indication of when the video was updated.	The video was uploaded over 5 years ago.	The video was uploaded over 1 year ago.	The video was uploaded within the last year.
Relevance	It does not address pertinent aspects of miscarriage.	It provides some information, but is not enough, or is not the right type of information.	It provides most of the pertinent information but is not comprehensive.	Provides all pertinent information.
Authority	The author is not an expert and has no relevant credentials. Or the organization is not known.	Author has ties to the topic but no relevant credentials. Or the organization is of questionable authority.	Author has credentials related to the topic but not expertise is field is not high. Or, the organization is well-known, but the degree of expertise is not high.	The author’s credentials are given and clearly indicate that he/she is an expert. Or the organization is well-known and highly credible.
Accuracy	Information is provided with no indication as to where it comes from.	Poor quality sources are cited, or information is entirely from personal experience.	Some information is obtained from quality sources.	All information is obtained from clearly stated, high quality sources.
Purpose	The purpose is to present a biased point of view, sell or promote an idea, service, or product. It is not factual or a balanced point of view. The opinion is either not backed up with facts of the facts are distorted.	The purpose of is to sell or promote something, but it also provides some quality information. Or expresses opinion that is somewhat logical with some evidence.	The purpose is to educate or offer mostly factual information. Expressed opinion is mostly logical and evidence based.	The purpose is to provide information of a scholarly, academic, or high-quality nature. Evidence for opinion is factual, presented as numbers in charts, graphs, tables, or statistics, or with adequate evidence for the opinion.

The Miscarriage Specific Question Score (MSQS) is a novel tool developed to objectively assess video content specifically on miscarriage. The MSQS tool was developed by the researchers of this study and reviewed by a panel of OB/GYN physicians with expertise in early pregnancy loss. The test consists of six questions that focus on the major medical aspects of miscarriage:

• 1.What are the common causes of miscarriage?
• 2.What are the symptoms of miscarriage?
• 3.When should medical care be sought for a miscarriage?
• 4.How is miscarriage evaluated and/or diagnosed?
• 5.What is the management/treatment of miscarriage?
• 6.What are the complications of miscarriage?

A score of 0–4 points was assigned to each question, as shown in Table 2. Each question was weighed equally with a maximum total score of 24 points per video. Scoring considers both the comprehensiveness of miscarriage content, as well as the accuracy of information. A final score of 0–6 points indicate poor performance, 7–12 points is fair, 13–18 points is good, and 19–24 points is excellent.

Table 2.

Miscarriage specific question score (MSQS) scoring criteria.

Score∗	Criteria
0	Not addressed
1	Incompletely addresses, has some incorrect information
2	Completely addresses, has some incorrect information
3	Incompletely addresses, has no incorrect information
4	Completely addresses, has no incorrect information

^∗For each video, a score was assigned for each of the six following questions according to the above criteria, for a total possible score of 24 points: 1. What are the common causes of miscarriage? 2. What are the symptoms of miscarriage? 3. When should medical care be sought for a miscarriage? 4. How is miscarriage evaluated and/or diagnosed? 5. What is the management/treatment of miscarriage? 6. What are the complications of miscarriage?

Data analysis for the study included the calculation of the kappa coefficient and the Pearson Product Moment correlation coefficient to assess the inter-rater agreement between two independent researchers. For descriptive statistics, the number of units (n), percent (%), mean values, and standard deviation (SD) were reported as applicable. P < 0.05 was considered statistically significant for all tests, and criteria from Landis [23] was used to evaluate agreement for kappa coefficients. All data analysis was conducted using SPSS version 28.0. This study was submitted to the Institutional Review Board (#1-1294984-1) and granted exemption due to the use of only publicly available data.

3. Results

160 videos were identified on preliminary screening. Of these, 86 videos were excluded because they were duplicates (n = 33), irrelevant (n = 47), or not in English (n = 6). The remaining 74 videos met inclusion criteria and were included for analysis as shown in Figure 1.

Figure 1.

Flowchart of video identification, inclusion and exclusion criteria. YouTube was searched using four search terms, then sorted by relevance and view count. Inclusion and exclusion criteria were applied to the 160 identified videos.

As shown in Table 3, videos were categorized by authority. Of the 74 videos, 25 were from companies (33.8%), 16 from institutions (21.6%), 15 from patients (20.3%), 15 from medical providers (20.3%), and 3 were unknown (4.1%). Videos were also classified by purpose. Most videos were educational (n = 51, 68.9%). 18 videos were testimonials (24.3%), 2 news reports (2.7%), 2 advertisements (2.7%), and 1 other (1.4%). Table 3 shows additional video characteristics. Mean video total view count was 406,010, with a mean of 8,086 likes and 232 dislikes. Mean video length was 546 s. Most videos had a duration of upload between one and five years (n = 45, 60.8%). Of the remaining videos, 16 were uploaded within the last year (21.6%), and 13 were over five years old (17.6%).

Table 3.

Video descriptive statistics and categorization by purpose and authority.

Characteristic	n
Mean views	406,010 ± 899,526
Mean likes	8,086 ± 25,002
Mean dislikes	232 ± 731
Mean duration (minutes)	9.1 ± 6.9
Upload duration∗
<1 year	16 (21.6%)
1–5 years	45 (60.8%)
>5 years	13 (17.6%)
Video Purpose
Educational	51 (68.9%)
Testimonial	18 (24.3%)
News Report	2 (2.7%)
Advertisement	2 (2.7%)
Other	1 (1.4%)
Video Authority
Company	25 (33.8%)
Institution	16 (21.6%)
Patient	15 (20.3%)
Medical Provider	15 (20.3%)
Unknown	3 (4.1%)

Data presented as n (%) or mean ± SD.

^∗Duration of time since video was uploaded to YouTube.

Mean CRAPP total score of the videos was 8.3 out of a total possible score of 15, which indicates a good quality source of information, though not of academic standards. Mean MSQS total score for the videos was 8.1 out of a total possible score of 24, which may be interpreted as “fair” regarding comprehensiveness and accuracy of miscarriage-specific content. Mean scoring subcomponents are shown in Table 4. Inter-rater reliability scoring showed a Pearson Correlation of 0.87 and 0.858 for the CRAAP and MSQ total scores, respectively. Per the commonly accepted Landis criteria, Pearson Correlation scores of >0.81 demonstrate excellent agreement between the two video raters [23].

Table 4.

CRAAP^∗ and MSQS^† mean scores with interrater reliability.

Test	Mean Score ±SD	Kappa	Pearson Correlation	P valuea
CRAAP Test
Currency	1.92 ± 0.60	0.74		<0.001
Relevance	1.26 ± 0.82	0.54		<0.001
Authority	1.59 ± 1.39	0.66		<0.001
Accuracy	1.68 ± 1.13	0.34		<0.001
Purpose	1.84 ± 1.22	0.45		<0.001
Total CRAAP score	8.30 ± 3.86		0.87	<0.001
MSQS Test
Cause	1.96 ± 1.74	0.60		<0.001
Symptoms	1.58 ± 1.77	0.57		<0.001
Medical care	1.12 ± 1.59	0.52		<0.001
Diagnosis	1.45 ± 1.67	0.51		<0.001
Treatment	1.30 ± 1.61	0.48		<0.001
Complications	0.70 ± 1.39	0.52		<0.001
Total MSQS score	8.11 ± 7.10		0.86	<0.001

^∗Currency, Relevance, Authority, Accuracy, and Purpose Test.

^†Miscarriage Specific Question Score.

^aP value reflects significance of interrater reliability score.

The lowest scoring subcomponent of the CRAAP score was “relevance” at 1.26, meaning that videos overall did not provide adequate depth of miscarriage-related information. The highest mean video scores were currency (1.92) and purpose (1.84). On further analysis of MSQS scores, videos on average scored higher for properly addressing the cause (1.96) and symptoms (1.58) of early pregnancy loss, while complications of miscarriage (0.7) and when to seek medical care for miscarriage (1.12) were less frequently and/or less thoroughly addressed as demonstrated by the lower scores in those categories.

Total CRAAP and MSQS scores were further analyzed by category, including the top 10 most liked videos, top 10 most viewed videos, and for all videos stratified by authority and purpose classifications as shown in Table 5. When analyzing videos by authority and purpose, videos from medical institutions and videos with an educational purpose had the highest total scores. Of note, several subcategories had a very small n (advertisement, n = 2; unknown, n = 3), and therefore the high scores in these categories may not be representative of a greater sample.

Table 5.

CRAAP^∗ and MSQS^† mean scores stratified by category.

Category	Total CRAAP Score	Total MSQS Score
Top 10 most liked	3.7 ± 0.92	5.6 ± 4.67
Top 10 most viewed	3.75 ± 0.91	5.55 ± 4.78
Video Authority
Medical Provider	10.67 ± 2.54	6.5 ± 6.84
Institution	11.41 ± 2.46	9.03 ± 7.32
Company	7.74 ± 3.7	9.12 ± 8.27
Patient	3.9 ± 1.06	6.57 ± 4.97
Unknown	6.17 ± 1.47	10.67 ± 3.08
Video Purpose
Educational	9.91 ± 3.19	8.84 ± 7.1
Advertisement	11.25 ± 1.5	22 ± 2.31
News Report	5.25 ± 2.63	1 ± 1.41
Testimonial	3.75 ± 1.18	5.67 ± 5.23
Other	7 ± 0	1.5 ± 2.12

Data presented as mean ± standard deviation.

^∗Currency, Relevance, Authority, Accuracy, and Purpose Test; maximum score 15.

^†Miscarriage Specific Question Score; maximum score 24.

4. Discussion

This study evaluated the accuracy and reliability of YouTube videos as a source for patient education on early pregnancy loss. 74 videos were reviewed, with a cumulative duration of 11.2 h and over 30 million views. YouTube has the potential to be an easily accessible source of patient education and information for many people. However, videos on YouTube are not subject to any quality control measures and, thus, have mixed quality. Our study demonstrated that though videos on early pregnancy loss were of good quality sources, they provided only a “fair” level of comprehensiveness and accuracy. Many videos failed to address complications of miscarriage, or when to seek medical care for a miscarriage. However, videos uploaded by medical institutions and those with an educational purpose had higher mean CRAAP and MSQS scores compared to the other videos, suggesting that this subset of videos offers the most reliable and accurate source of information among all miscarriage-related videos on YouTube. If patients choose to seek information on YouTube about early pregnancy loss, or if providers choose to recommend YouTube for patient education, our study demonstrates that videos from medical institutions and with an educational purpose provide the best reference.

YouTube video analysis has been undertaken in a variety of healthcare fields [24, 25, 26, 27, 28]. Though no published studies to date have analyzed YouTube videos specific to early pregnancy loss, YouTube video review has been carried out on contraceptive devices, which demonstrated significant misinformation rates [24, 25]. YouTube video analysis has also been reported on topics outside the scope of obstetrics and gynecology [26, 27, 28]. These studies found YouTube to be an unreliable source of information, with only a small number of videos addressing target concepts [26] and many videos of poor quality [27, 28].

Our study had several limitations. As a cross-sectional study that reviewed only a fraction of available videos on YouTube, greater sensitivity may have been obtained if more videos were included. However, since most YouTube users view only the first few available pages [12], the first 20 videos per keyword search and sorting method could provide us with an appropriate sample size that is reflective of the videos most likely to be accessed during a YouTube search A second limitation of our study is that videos are a snapshot in time and are always in flux. Reviewed videos were downloaded on a specific date and time and may not be the most current or relevant when information is sought and requested by patients.

The MSQS criteria is a novel scoring system developed by our team to assess the quality and comprehensiveness of miscarriage-specific content. Future studies could validate a more generalizable form of the MSQS criteria for use as a grading tool to assess the quality of educational contents for other diseases. For example, the same components and questions of the MSQS criteria could be studied in the context of videos on hypertension or diabetes.

In conclusion, our study demonstrated that YouTube videos related to early pregnancy loss span a wide range of quality, accuracy, and purpose. Videos from medical institutions and with an educational purpose provide the best reference. While some videos provide effective content, mean rater scores demonstrate that YouTube is not a reliable source for patient education on early pregnancy loss.

Online platforms are being more frequently utilized by the general public and learners of all fields. Data demonstrates that succinct, short presentations are better in engaging viewers [13, 14, 15], especially when utilizing online platforms for education. To minimize gaps in fact and misinformation, policymakers and healthcare administrators should consider adopting short educational materials with catchy titles using online platforms to meet the needs of the general public, as well as the needs of students of the scientific community [15].

Given the frequency of patients utilizing YouTube for healthcare information [6], it is important for healthcare providers to understand the characteristics of these videos and be aware of potential sources of misinformation. Similarly, if healthcare providers choose to recommend YouTube to patients, they should consider keeping a list of reliable videos on hand that may help patients make appropriate medical decisions.

Declarations

Author contribution statement

Lisa McNamee, MD: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Hannah Shakartzi, MD: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data.

Thomas Wasser, PhD, Med; Yan Li, PhD: Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data.

Chi-Son Kim, MD, MPH: Conceived and designed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Funding statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Data availability statement

Data will be made available on request.

Declaration of interest’s statement

The authors declare the following conflict of interests: Chi-Son Kim is a non-paid trainer for Merck Nexplanon®.

Additional information

No additional information is available for this paper.