Accuracy and readability of cardiovascular entries on Wikipedia: are they reliable learning resources for medical students?

Abstract

Objective

To evaluate accuracy of content and readability level of English Wikipedia articles on cardiovascular diseases, using quality and readability tools.

Methods

Wikipedia was searched on the 6 October 2013 for articles on cardiovascular diseases. Using a modified DISCERN (DISCERN is an instrument widely used in assessing online resources), articles were independently scored by three assessors. The readability was calculated using Flesch-Kincaid Grade Level. The inter-rater agreement between evaluators was calculated using the Fleiss κ scale.

Results

This study was based on 47 English Wikipedia entries on cardiovascular diseases. The DISCERN scores had a median=33 (IQR=6). Four articles (8.5%) were of good quality (DISCERN score 40–50), 39 (83%) moderate (DISCERN 30–39) and 4 (8.5%) were poor (DISCERN 10–29). Although the entries covered the aetiology and the clinical picture, there were deficiencies in the pathophysiology of diseases, signs and symptoms, diagnostic approaches and treatment. The number of references varied from 1 to 127 references; 25.9±29.4 (mean±SD). Several problems were identified in the list of references and citations made in the articles. The readability of articles was 14.3±1.7 (mean±SD); consistent with the readability level for college students. In comparison, Harrison’s Principles of Internal Medicine 18th edition had more tables, less references and no significant difference in number of graphs, images, illustrations or readability level. The overall agreement between the evaluators was good (Fleiss κ 0.718 (95% CI 0.57 to 0.83).

Conclusions

The Wikipedia entries are not aimed at a medical audience and should not be used as a substitute to recommended medical resources. Course designers and students should be aware that Wikipedia entries on cardiovascular diseases lack accuracy, predominantly due to errors of omission. Further improvement of the Wikipedia content of cardiovascular entries would be needed before they could be considered a supplementary resource.

Strengths and limitations of this study.

• Standardised quality and readability tools were used to answer the research questions.

• The study solely focused on 47 adult cardiovascular diseases and only those entries in the English language.

• The results cannot be generalised to other medically related topics on Wikipedia.

• The work was not blinded. Textbooks are not free from possible errors or differences in their contents.

• Wikipedia is not intended to be used as a textbook for medical students.

Introduction

With the introduction of integrated medical curricula and self-directed learning, students need to search for their learning issues using a wide range of learning resources.^{1 2} These changes aim at encouraging active learning and moving away from passive learning or relying on the teacher as the main source of information. Student-centred learning approaches are particularly recommended in medical schools because of the rapid proliferation of medical information.^{3 4} Therefore, medical students need to gain and update their information continuously, and master the skills on how to search for knowledge rather than limit their learning to lectures or the content of a particular textbook.¹

Owing to the increasing use of social web, also known as web 2.0, the way information is produced, shared and used has changed.⁵ Medical students are increasingly relying on the Internet and websites such as Google and Wikipedia in searching for information.² Recently, one study showed that one-third of college students use Wikipedia for academic learning.⁶ Guarino et al⁷ showed that Wikipedia significantly surpassed the peer-reviewed medical databases as their preferred learning resource. Another study highlighted the increasing use of Wikipedia among dental students² and surgical residents.⁸ This tendency may be related to the easy access. People most often start by searching Google; Wikipedia is usually among the top search results on Google.⁹ Wikipedia is also well known to most online users. Furthermore, Wikipedia is available at any time and from anywhere; users only need a computer or a smart phone connected to the Internet to search Wikipedia.⁵ Other minor advantages for the reliance of students on web-based resources such as Wikipedia may include: (1) information provided online usually address different aspects of an issue and use a range of resources to explain it such as images, videos, diagrams and entries,⁵ and (2) information provided online is more frequently updated than are textbooks.¹⁰ These factors may play a role in making students prefer online resources rather than textbooks.

Wikipedia, created in 2001, is a free multilingual online encyclopedia that provides entries about almost any topic, and is based on an openly editable model.^{9 11 12} There have been 4 880 039 entries published up to 28 May 2015 in the English language alone, with a total of 36 373 672 pages, and an increase of 197 138 articles during the proceeding year.^{13 14} It is interesting to note here that Wikipedia has created this number of entries without any budget or editorial team. The reason for the high number of Wikipedia entries is that anyone can write an entry (an article); it is entirely based on volunteers from different countries, who have worked cooperatively to create, edit and update articles.

Interestingly, searching for information by using a search engine such as Google or Yahoo has shown Wikipedia appearing in the first 10 results.⁹ As of May 2015, the number of views worldwide was over 453.10 million,¹⁵ and the number of medical and health-related articles was over 33 000 entries.^{13 14}

Although a good number of research publications examined the reliability of Wikipedia: (1) as an educational resource for patients,¹⁶ nursing students,¹⁷ clinical pharmacy education,¹⁸ drug information¹⁹ and veterinary medicine,²⁰ and (2) for gathering information on gastrointestinal and liver diseases,²¹ respiratory diseases²² and the nervous system,²³ there have been no studies examining the scientific accuracy and readability level of the Wikipedia entries on cardiovascular diseases, and whether the entries can be a reliable learning resource for medical students. Therefore, the aim of this study was to assess the scientific accuracy of the Wikipedia entries on cardiovascular diseases and whether they can be a suitable learning resource.

Methods

Study design

Five standardised medical textbooks were used as a reference for identifying the cardiovascular topics to be searched on Wikipedia. These textbooks (box 1) were selected because they are widely recommended in undergraduate medical courses in most medical schools and also recommended by regulatory authorities such as the Australian Medical Council in Australia (http://www.amc.org.au/publications#medicine), the General Medical Council in UK (http://www.gmc-uk.org/doctors/plab/23448.asp), and the Royal Colleges such as the College of Physicians and Surgeons of Canada (https://www.cpsbc.ca/content/exam-preparation), for examination preparation. They are written by experts and clinical teachers in different specialties in medicine, are regularly updated and reviewed, and were reviewed by scholars in prestigious medical journals such as the Journal of the American Medical Association, British Medical Journal and The New England Medical Journal.^24–27 The aims of using these textbooks were to: (1) identify the cardiovascular topics to be evaluated, and (2) use these textbooks as a reference resource during the assessment of the content of the Wikipedia entries. The strategy for searching the five cardiovascular chapters included: (1) identifying key topics and the content under each topic, and assessing the emphasis made, and (2) identifying topics that shared maximum emphasis across the five chapters. Each evaluator searched these chapters independently and the outcomes were discussed in a meeting. Topics shared among the researchers were further discussed and topics covering rare diseases or rare syndromes were not included. In fact, rare diseases included in Wikipedia were evaluated in an elegant study.⁹ The final list comprised 47 Wikipedia entries covering issues commonly included in undergraduate medical programmes. Another source for identifying the 47 topics was the author's long experience in medical school curricula at the University of Sydney and University of Melbourne in Australia as well as other detailed curricula for undergraduate medical schools such as curriculum verification reports (University of California—San Francisco School of Medicine; http://www.aamc.org/download/363632/data/sampleverifreprtucsf.pdf). Also, the recommendations of the report of Good Medical Practice (2013) issued by the General Medical Council (http://www.gmc-uk.org/guidance/good_medical_practice.asp) and the guidelines to medical schools issued by the Australian Medical Council (http://www.amc.org.au/accreditation/primary-medical-education).

Box 1. Medical textbooks used as a reference in evaluating Wikipedia articles.

Andreoli TE, Benjamin IJ, Griggs RC, et al. Andreoli and Carpenter's Cecil Essentials of Medicine, 8th Edition. Philadelphia: Saunders, 2010.

Colledge NR, Walker BR, Ralston SH. Davidson's Principles & Practice of Medicine, 21st Edition. Edinburgh: Elsevier, Churchill Livingstone, 2010.

Kumar P, Clark M. Kumar and Clark's Clinical Medicine, 8th Edition. Edinburgh: Elsevier, 2013.

Longo DI, Fauci AS, Kasper DL, et al. Harrison's Principles of Internal Medicine, 18th Edition. New York: McGraw-Hill, 2012.

McPhee SJ, Papadakis MA, Rabow MW. 2011 Current Medical Diagnosis & Treatment, 50th Anniversary Edition. New York: McGraw-Hill, Lange, 2011.

Searching Wikipedia

The topics identified were used as keywords in searching Wikipedia (http://www.wikipedia.com). The search was conducted on the 6 of October 2013 by three evaluators independently, and entries identified were printed out on the same day. The predetermined 47 topics on common cardiovascular disorders were all found to have a corresponding article on Wikipedia. Each researcher received a copy of the 47 Wikipedia entries for further evaluation. The reason for printing out copies was to ensure that all researchers evaluate the same version of a particular entry. This is important, since Wikipedia entries are constantly being updated and changing.¹⁴

Assessing the accuracy and depth of coverage

To assess the accuracy of the content of Wikipedia entries, a modified version of the DISCERN instrument was used. The original DISCERN is a standardised set of criteria for judging the quality of health information written for the public on treatment options.²⁸ The DISCERN instrument was created by the University of Oxford, and the project was funded by the British Library and the National Health Service (NHS) Research & Development Programme.²⁸ It consists of 15 questions plus an overall quality-rating question (http://www.discern.org.uk/discern.pdf). The instrument has been used to assess healthcare-related websites and online resources. For example, the quality of patient information on surgical treatment of haemorrhoids,²⁹ colorectal cancer information³⁰ and urology-related patient information.³¹

However, the original version of the DISCERN instrument was not suitable for evaluating online resources such as Wikipedia entries; it did not examine whether the information provided is scientifically correct and in agreement with current valid resources, and whether there were gaps or scientific errors in the given information. The original DISCERN instrument also lacked the evaluation of images, figures and tables that support information provided in the article and enhance understanding of the topic discussed. These deficiencies urged the need to use the modified DISCERN version, which has been tested and used in earlier publications.^21–23 The modified DISCERN comprised 10 questions. The questions covered the aims and objectives of the topic, the scientific accuracy of the content and whether the content was neutral (not based on personal views of authors); they also assessed whether the contents allocated to each part or each subtitle were distributed in a balanced way, and examined the clarity of content, frequency of updating and the quality of images, illustrations and tables, and whether they add value and enhance understanding of the topic. The final question was on the overall rating of the article (entry) (see online supplementary appendix S1). Each question was rated on a 5-point scale. One corresponds to ‘NO, if absent or not addressed’, three corresponds to ‘partially addressed, addressed but not in an adequate or satisfactory level’, and five corresponds to ‘Yes, adequately addressed’. For question 10, a score of 1 was given if the answer was “extensive shortcomings, for example, the article was not completed and/or lacked key issues, and/or had several scientific errors or contradictory statements’, 3 was given if the answer was “potentially important but not extensive shortcomings, for example, the article was completed but had one major deficiency that may limit its educational value’ and 5 was given if the answer was “minimal shortcomings, for example, the article had minor shortcomings that will not interfere with its overall educational value’. The maximum score for the modified DISCERN is 50 and the minimum score is 10. In order to make the results of total scores meaningful, cut-off points of the total score have been used where a total score of 40 or above is described as Good, a score of 30 to 39 is described as Moderate and a score less than 30 is described as Poor.

Piloting the work

Before using the modified DISCERN to assess the 47 Wikipedia entries, the work was piloted. The aims of piloting were: (1i) familiarise the evaluators with the scoring system of the DISCERN instrument, (2) discover sources of disagreement among evaluators and discuss ways to minimise disagreement and (3) enable the evaluators to master the use of the instrument. To conduct the pilot phase, it was decided to choose 10 entries comparable to, but other than, the 47 entries included in the study. Each evaluator was asked to apply the criteria independently. Once the evaluation was completed, the outcomes were discussed in a meeting. Wikipedia entries that were scored differently by different evaluators were discussed until an agreement was reached. Another 10 entries were re-evaluated independently. After two rounds of piloting the study, the agreement among evaluators was in the range of 80–90%, indicating readiness to assess the articles included in the study.

Conducting the study

Along the same methodological approach, the 47 Wikipedia entries were evaluated independently using the modified DISCERN instrument. Scores were then placed on an Excel sheet (Microsoft Excel for Mac 2011, V.14.4.1, Microsoft Corp, Redmond, Washington, USA). The agreement between the evaluators was measured using the Fleiss κ scale.^32–34

Assessing the references

Since one of the parameters of assessing an academic and scholarly work rests on assessing the references that were used in creating the work, it was decided to assess the references of each Wikipedia entry. The assessment included the following: (1) total number of references, (2) number of peer-reviewed journals, (3) number of peer-reviewed journals in the past 5 years, (4) educational/procedures guidelines, (5) textbooks, (6) professional websites, (7) general websites and (8) news and media. Evaluators conducted the evaluation of references independently.

Frequency of updating Wikipedia entries

Owing to the continuing changes and discoveries in medical and scientific fields, it is indispensable to assess the rate of updating of cardiovascular articles. The frequency of updating each article was assessed through a revision of the ‘history page’ on Wikipedia. The page includes detailed history such as the date of creation of the article, the number of authors and the frequency of updates, and the changes made to the article since its creation. The aim was to assess the number of updates/revisions made up to the date of printing out the copies (6 October 2013) and calculate the number of updates in the 12 months prior to the printing. Assessing the outcomes of such revisions and editing was examined in light of a few examples of Wikipedia articles, regarding their adequacy of knowledge and scientific accuracy.

Assessing readability

To test the hypothesis of whether the articles were written at the college student's readability or the public level, it was decided to calculate the readability level of each article. Using the free online readability instrument provided by ‘Readability Formulas’ (http://www.readabilityformulas.com/), the Flesch-Kincaid Grade Level tool was used to calculate readability;³⁵ the measurement has been widely used in a number of studies to calculate reading levels.^{16 21–23 30 35–37}

The formula for the Flesch-Kincaid Grade Level score is:

Where: ASL is the average sentence length (the number of words divided by the number of sentences) and ASW is the average number of syllables per word (the number of syllables divided by the number of words). This tool evaluates text based on US school grade levels. For instance, a score of 10 indicates the reading level of a 10th grade student, and scores in the range of 14 to 16 indicate the reading level of college students. To calculate the readability score, each evaluator randomly selected and copied a sample of text between 150 and 600 words from the beginning, middle and end of each article. The text was placed into the online calculator and the readability score obtained for each input. Headings, external links and images were excluded. The mean readability score and SD were calculated for each article.

Assessing medical textbooks

The aim was not to compare textbooks with Wikipedia articles because Wikipedia is not written for medical students and is not aimed to be an academic resource. However, it was decided to assess one textbook, Harrison's Principles of Internal Medicine, 18th edition, in regard to the number of images, illustrations, graphs, tables, references as well as its readability, using the Flesch-Kincaid Grade Level. This assessment is limited to the corresponding 47 cardiovascular articles. The DISCERN instrument was not applied to the textbooks because textbooks are used in this study as a resource reference for comparison. Furthermore, the only reference to test the scientific accuracy of textbooks is the medical literature; a task outside the scope of this study. While most textbooks in their paper format are regularly reviewed and new editions released every 4–5 years, most textbooks have introduced online resources to accompany the textbooks. For example, the CD, Access Medicine (Accessmedicine.com), is provided with the Harrison Textbook, along with online resources, images, tables and figures. The other four books are linked with Expert Consult Online, which provides users with updates and additional resources. These online resources are regularly updated with indications of when they were updated.

Statistical analysis

The final scores for accuracy and readability of each article were collected from each evaluator and placed on an Excel spreadsheet. Analyses for the median, IQR, mean and SDs as well as correlation studies were performed using SPSS (V.22.0 for Mac OS, SPSS Inc, Chicago, Illinois, USA). The agreement between evaluators was calculated using the Fleiss κ scale.

Results

Overall evaluation of Wikipedia entries

In the template created by Wikipedia, each article has a table of contents to guide the reader. In addition, an ‘infobox’ displays the International Classification of Diseases (ICD-10, ICD-9), Medical Subject Headings (MeSH) codes, Online Mendelian Inheritance in Man (OMIM) entry and links to the topic on MedlinePlus, eMedicine and Diseases Database. Also, many of the articles have external links for further reading, and images to support and elaborate on the information provided. It is worth mentioning that the pericardial effusion article has a two-dimensional transthoracic echocardiogram animation of pericardial effusion, and the article on palpitations has two audio recordings demonstrating the difference between normal and abnormal heart rhythm. Such audiovisual aids aim at enhancing the understanding of the written description.

The articles covered key points such as disease aetiology, clinical picture and treatment. However, in most articles, there were deficiencies regarding the pathophysiology, mechanisms, diagnostic approach and management plan. Also, significant variations in covering the appropriate subtitles and key concepts were noted in some articles. For example, while the article on atrial fibrillation mentioned the definition of the disease, its classification, causes, clinical signs and symptoms and the diagnosis of a patient with atrial fibrillation as well as its pathophysiology, management, prognosis and epidemiology, the article on pulmonic regurgitation failed to address diagnosis, pathology, pathophysiology, epidemiology and management.

Although articles (entries) used images, including illustrations and tables to further explain the content, not all articles had images, illustrations or tables. For example, the articles on acute pericarditis, angina pectoris, chest pain, pulmonic regurgitation, pulmonic stenosis, tricuspid regurgitation and tricuspid stenosis had no images or tables. The total number of images and tables in the 47 Wikipedia entries was only 179; 3.8±3.1 (mean±SD).

Scientific accuracy and depth of coverage

Table 1 shows the median and the IQR for the accuracy scores of each article. The highest score was 45 for the article on deep vein thrombosis while the lowest was 28 for the article on acute pericarditis. The 47 Wikipedia articles had a median score=33, IQR=6—the highest possible score was 50. Out of the 47 entries, 4 (8.5%) articles scored 40 or higher, described as Good as per our cut-off system, 39 (83%) articles scored 30 to 39, described as Moderate as per our cut-off system and 4 (8.5%) scored less than 30, described as Poor as per our cut-off system. We did not observe vandalism of the 47 entries during the conduction of the study (raw data covering DISCERN scores made by three evaluators for each Wikipedia entry and the calculation of readability of the Wikipedia entries are shown in online supplementary material 1)

Table 1.

Summarises the accuracy, the DISCERN scores (median and IQR), number of images, illustrations, tables and readability score of Wikipedia cardiovascular articles

Topics	Number of pages	Accuracy DISCERN Score Median (IQR)*	Number of images	Number of illustrations	Number of graphs	Number of tables	Total	Readability Flesch-Kincaid Grade Level (Mean±SD)	Further reading/external links
Acute pericarditis	5	28 (1)	0	0	0	0	0	12.6±0.8	0 external links
Acute pulmonary oedema	6	38 (0)	1 CT 2 X-ray	0	0	0	3	15.5±1.4	1 external link
Angina pectoris	12	29 (1)	0	0	0	0	0	13.6±1.3	6 external links
Aortic aneurysm	7	33 (2)	1 MRI 2 CT reconstruction	0	0	0	3	11.9±0.6	2 external links
Aortic dissection	13	41 (3)	1 Chest X-ray 1 CT 1 Histopathological slide 2 Echocardiogram 2 MRI	5	1	0	13	12.2±2.0	1 external link
Aortic regurgitation	8	31 (1)	1 Histopathological slide	1	1	0	3	16.3±1.6	0 external links
Aortic stenosis	10	33 (3)	1 Electron micrograph 1 Photograph	1	1	1	5	16.4±6.1	3 external links
Arterial hypertension	19	38 (1)	1 Photograph	4	2	1	8	14.1±0.8	1 external link
Atherosclerosis	24	37 (1)	1 Doppler US 1 Photograph 2 Micrograph	2	0	0	6	16.3±2.0	3 external links
Atrial fibrillation	17	39 (2)	1 Photograph	4	1	0	6	14.6±2.3	1 external link
Atrial flutter	5	33 (1)	1 Photograph	1	0	0	2	14.5±1.9	0 external links
Atrial septal defect	13	33 (1)	1 Ultrasound	4	0	0	5	11.7±1.1	4 external links
Bradyarrhythmias	5	30 (3)	0	2	0	0	2	10.6±1.1	0 external links
Cardiac tamponade	5	33 (3)	1 Ultrasound	0	0	0	1	14.0±1.5	0 external links
Cardiomyopathies	5	32 (2)	1 Histopathological slide 1 Photograph	0	0	0	2	16.4±1.5	3 external links
Chest pain	5	32 (2)	0	0	0	0	0	20.6±8.3	0 external links
Coarctation of the aorta	8	31 (4)	2 CT reconstruction	2	1	0	5	12.5±0.8	13 external links: (7 websites and 6 research papers)
Complete transposition of the great arteries	10	32 (0)	4 Photographs	1	0	0	5	16.7±0.7	9 external links
Cyanosis	4	37 (2)	3 Photograph	0	0	0	3	13.3±1.9	1 external link
Deep vein thrombosis	21	45 (1)	1 CT scan 1 Ultrasound 1 Venogram 7 Photographs	3	0	1	14	12.2±1.4	4 external links
Dyspnoea	8	33 (1)	0	0	1	0	1	14.2±1.3	2 external links
Eisenmenger syndrome	4	35 (2)	1 Photographs	1	0	0	2	15.5±0.9	3 external links
Endocarditis	3	33 (1)	1 Microbiology stain	0	0	0	1	18.3±6.6	2 external links
Heart failure	20	42 (1)	1 Photographs 1 X-ray	1	1	0	4	13.6±1.7	2 external links
Infective endocarditis	11	33 (2)	1 Echocardiogram 1 Photographs	0	0	0	2	16.8±3.4	0 external links
Long QT syndrome	10	38 (1)	0	1	1	0	2	11.4±1.1	1 external links
Mitral regurgitation	9	34 (1)	1 TOE	2	3	0	6	15.1±1.4	3 external links
Mitral stenosis	7	33 (2)	2 Photographs	4	1	0	7	13.9±0.9	3 external links
Mitral valve prolapse	8	38 (0)	1 Painting 1 TOE	3	0	0	5	14.2±1.0	2 external links 5 further readings (research papers)
Myocardial infarction	20	39 (1)	0	6	0	0	6	15.6±0.4	2 external links
Palpitations	4	30 (1)	0	2 Audio recordings	0	0	2	14.8±1.6	6 external links
Patent ductus arteriosus	6	34 (1)	2 Echocardiogram	3	0	0	5	11.4±1.3	9 external links
Pericardial effusion	4	35 (3)	1 CT scan 1 ECG 1Echocardiogram 2 Ultrasound 2 X-ray	1	0	0	8	15.2±1.6	4 external links
Peripheral arterial disease	9	31 (3)	1 Photograph	0	0	0	1	16.4±3.4	1 external link
Prosthetic heart valves	10	28 (0)	5 Photographs	0	0	0	5	11.5±1.8	2 external links
Pulmonary arterial hypertension	11	34 (1)	1 Micrograph	3	0	0	4	13.9±08	8 external links
Pulmonary embolism	14	35 (1)	1 Angiogram 1 Nuclear imaging 2 CT 1 X-ray 4 Photographs	0	0	0	9	11.5±0.7	0 external links
Pulmonic regurgitation	2	29 (1)	0	0	0	0	0	17.7±1.2	0 external links
Pulmonic stenosis	2	30 (2)	0	0	0	0	0	14.4±3.5	1 external link
Rheumatic fever	8	37 (4)	1 Micrograph 1 Photograph	3	0	0	5	14.4±1.2	3 external links
Syncope	9	33 (1)	1 Painting	0	0	0	1	12.9±1.0	4 external links
Tetralogy of Fallot	9	36 (1)	1 Photograph 1 X-ray	3	0	0	5	14.3±1.0	7 external links
Tricuspid regurgitation	3	31 (0)	0	0	0	0	0	15.3±1.0	3 external links
Tricuspid stenosis	2	31 (1)	0	0	0	0	0	14.6±0.1	1 external link
Ventricular septal defect	7	40 (2)	1 Echocardiogram 1 Photograph	2	0	0	4	12.6±2.0	16 external links
Ventricular tachycardia	6	32 (1)	2 Photographs	0	0	0	2	14.1±1.8	0 external links
Wolff-Parkinson-White syndrome	8	38 (1)	2 Photographs	4	0	0	6	14.4±0.2	3 external links

*The modified DISCERN instrument was used in calculating accuracy. The results represent median and IQR of the scores. The results represent English Wikipedia cardiovascular articles searched on 6 October 2013.

CT, computed tomography; MRI, magnetic resonance imaging; TOE, transoesophageal echocardiogram; US, Ultrasound.

Assessing the references

Table 2 summarises the mean and SD of the references included in the 47 Wikipedia cardiovascular entries. The total number of references was 1218, and the number of references in each article varied from 1 to 127. Of the 1218 references, 229 were from the 1990s and 6 references were from the 1980s. Thirty-five references did not mention the year of publication. The majority of these 1218 references were from peer-reviewed medical journals, making a total of 790 references (65%). Other references were professional medical websites (10%), educational/procedural guidelines (6%), textbooks (10%), general websites (5%), news and media (2%), and others (2%). Several errors were identified in citing the references: (1) a total of 182 (14.9%) were incomplete references, missing information about the author's name, title of article, year, date of retrieval of online websites and the book publisher, (2) 21 entries included a statement such as ‘citation needed’ after some sentences or paragraphs, which indicates the need to proof the origin of statements made, (3) inconsistency among entries or within the same entry in citing references was observed (n=29, 61.7%); some references were cited using the American Psychological Association (APA) style, others using Harvard style, and (4) URL links to several references were found to be broken or inaccessible (n=46, 3.8 %). It was also noticed that the quality of references varied substantially from peer-review journal articles to blogs reviewing episodes of comedy TV shows. On average, only 73.7% of references per article (entry) were accurate in all respects.

Table 2.

Summarises the mean and SD of the references of the 47 English Wikipedia entries on cardiovascular diseases included in the study

Number of Cardiovascular Entries	Peer-reviewed journals (Mean±SD)	Guidelines (Mean±SD)	Textbooks (Mean±SD)	Professional websites (Mean±SD)	General websites (Mean±SD)	News media (Mean±SD)	Others (Mean±SD)	All references (Mean±SD)
47	16.8±21.56	1.53±5.63	2.68±2.35	2.53±2.77	1.23±1.86	0.53±1.41	0.51±1.38	25.91±29.4

Frequency of updating the topics

Among the 47 Wikipedia entries, the entry on bradyarrhythmias was the earliest article, created in 2001. Total number of updates for that article was 350, with 40 updates made in the past 12 months. On the other hand, the most recent article, created in 2008, was on pulmonic regurgitation, with 22 total updates and 4 updates in the past 12 months. The total number of updates of the 47 Wikipedia entries was in the range of 22 to 3700; 624.9±747.4 (mean±SD). While the total number of updates for the 47 entries in the past 12 months was in the range of 2 for the article on tricuspid stenosis to 215 for the article on myocardial infarction; 46.2±43.3 (mean±SD). Regardless of the year of creation, the frequency of updates varied significantly between articles. For example, the articles on arterial hypertension and endocarditis were created in 2002. During the past 11 years, arterial hypertension had 3700 edits and 100 edits in the past 12 months, making a yearly average of 336 edits, while endocarditis had 369 total edits over 11 years and 8 in the past 12 months, with a yearly average of 33 edits. Figure 1 shows a cumulative frequency graph (Ogive) for updates of the 47 Wikipedia entries on cardiovascular diseases since the date of their publication.

Figure 1.

Cumulative frequency graph (Ogive) for updates of the 47 Wikipedia entries on cardiovascular diseases.

Table 3 summarises a few examples of knowledge deficiencies and scientific inaccuracies in four Wikipedia entries on cardiovascular diseases included in the study. The examples show that, despite the relatively good number of updates of each of these articles (conducted since the time of their publications and over the past 12 months), the articles were not free from knowledge deficiencies and inaccuracies. Suggestions for improvement were provided for each item identified.

Table 3.

Examples of knowledge deficiencies and scientific inaccuracies in the English cardiovascular Wikipedia entries and suggestions for improvements

Wikipedia topic (entry)	Number of updates (edits in past 12 months)	Areas of deficiency/inaccuracy	Suggestions for improvements
Angina pectoris	Edited 979 times (150 times in past 12 months)	There is no mention of staging of angina. The pathophysiology is incomplete and not scientifically correct. Although it was mentioned that, “some cases may have associated symptoms such as breathlessness, sweating, and nausea”, there were no indications that these symptoms were due to sympathetic stimulation triggered by pain. Another paragraph stated, “some patients experience autonomic symptoms” without linking the symptoms, stated earlier, to the autonomic system and giving an explanation The statements about Prinzmetal's angina are incomplete and not scientifically correct and should be amended in light of the current literature. Also, there is no mention under diagnosis of key investigations	The Canadian Cardiovascular Society Grading Scale and or the New York Heart Association Classification should be added An understanding of the pathophysiology of angina starting from coronary atherosclerosis, endothelial dysfunction, cholesterol accumulation and key molecular and biochemical changes should be added It should be mentioned that the diagnosis of Prinzmetal's angina is based on the documentation of transient ST segment elevation during an episode of chest pain in the absence of severe, fixed coronary stenosis. Associated Raynaud phenomenon and migraine headache have been described in some of these patients, suggesting that the syndrome is due to a more generalised vasospasm disorder. Although the pain is always at rest, one-third of patients experience pain during exercise It should be stated that the evaluation of chest pain depends on a careful history and physical examination. The assessment of a patient with stable ischaemic heart disease includes 12-lead ECG, measurement of biochemical and inflammatory markers, and non-invasive diagnostic tests. The first goal is to assess the patient's probability of ischaemia so that emergency treatment can be expedited. Investigations such as non-invasive stress testing with a standard electrocardiographic treadmill or bicycle exercise, radionuclide imaging, stress echocardiography, cardiac magnetic resonance and positron emission tomography, need to be added.
Chest pain	Edited 357 times (23 times in past 12 months)	There is no mention of the signs and symptoms of patients presenting with chest pain Although several causes of chest pain are listed under differential diagnosis, some important causes were not stated such as aortic stenosis, aortic dissection, oesophageal spasms and oesophageal rupture	It is important to add key clinical features of a patient presenting with chest pain and how the medical history is important in defining the possible cause. Key issues in the history that should be added are: (1) site of pain, (2) onset, when did start? (3) character of pain, type of pain, description of pain, (4) radiation, does the pain go anywhere else? (5) associated features, (6) time/pattern. Is it any worse at a particular time of the day? In relation to activity, cough or movement? (7) exacerbating and relieving factors, and (8) severity on a scale from 1 to 10 (pain intensity, interference with sleep). The differential diagnosis should be organised under common causes and uncommon causes. It can also be grouped under emergency causes and non-emergency causes
Pulmonic regurgitation	Edited 22 times (4 times in past 12 months)	Although several causes of pulmonic regurgitation are mentioned, it may be important to state that pulmonic regurgitation is often secondary to an underlying pulmonary hypertension or dilated cardiomyopathy rather than due to primary valvular defect. There are no links provided for this topic	Pathologically, three possible mechanisms may result in pulmonic regurgitation: (1) dilation of pulmonic valve ring, (2) acquired change in pulmonic valve leaflets and (3) congenital defects in valve leaflets. Important links from the American Heart Association (also has two videos): http://www.heart.org/HEARTORG/Conditions/More/HeartValveProblemsandDisease/Problem-Pulmonary-Valve-Regurgitation_UCM_450884_Article.jsp should be added
Pulmonic stenosis	Edited 26 times (10 times in past 12 months)	Common causes of pulmonic stenosis are not clearly stated The symptoms and signs of pulmonic stenosis are not mentioned Pathophysiology: the statement that “if right ventricular failure develops, right atrial pressure will increase and this may result in a persistent opening of the foramen ovale, shunting of unoxygenated blood from the right atrium into the left atrium”, is not scientifically accurate	Common causes of pulmonic stenosis are: (1) isolated valvular stenosis (90% of causes). This forms about 10–12% of all congenital heart diseases, (2) subvalvular obstruction: obstruction or narrowing of the subinfandibular region and normal valve, and (3) supravalvular obstruction at the level of pulmonary artery bifurcation and/or more peripherally Symptoms and signs should be stated. The stenosis ranges from mild (without symptoms) to severe (which usually worsens) and requires surgery. Main symptoms are: shortness of breath, chest pain, loss of consciousness and fatigue Usually, persistent opening of the foramen ovale is related to other congenital anomalies in the heart occurring in these patients An external link such as Mayo Clinic [http://www.mayoclinic.org/diseases-conditions/pulmonary-valve-stenosis/basics/definition/con-20013659] can be added

Correlation of the DISCERN score and other parameters

A correlation was found between the DISCERN score and the total number of peer-reviewed journals included in the references (R²=0.234, p<0.001) and between the DISCERN score and the number of peer-reviewed references in the past 5 years (R²=0.243, p<0.001). There was also a correlation between the DISCERN score and the total number of article updates (R²=0.333, p<0.001; figure 2) and a strong correlation between the number of pages and the total number of updates (R²=0.647, p<0.001; figure 3).

Figure 2.

Correlation between the DISCERN score and total number of article updates for Wikipedia cardiovascular articles.

Figure 3.

Correlation between number of article pages and the number of article updates for Wikipedia cardiovascular articles.

Calculating readability

Table 1 shows the readability scores of each Wikipedia entry measured by the Flesch-Kincaid Grade Level. The minimum score was 10.6±1.1 for the entry on bradyarrhythmias, while the maximum score was 20.6±8.3 for the entry on chest pain. The mean readability score was 14.3±1.7 (mean±SD). The results are consistent with a college reading level.

Comparing Wikipedia entries with respective articles in the Longo et al (2012) textbook

Table 4 summarises a comparison between the Wikipedia entries and the respective cardiovascular topics in the textbook by Longo et al (2012), Harrison’s Principles of Internal Medicine 18th Edition. The table shows that the textbook provided more images, graphs and tables than the Wikipedia articles did. The differences were not significant for the graphs (p=0.055) and images (p=0.998), but were for the tables (p=0.004). Although Wikipedia articles had more illustrations, the difference was not significant (p=0.148). However, the number of references was significantly less in the textbook compared to the Wikipedia articles (p=0.002). No significant differences were found regarding readability level (p=0.524). It is interesting to note that the new edition (the 19th Edition) of Harrison’s Principles of Internal Medicine, which appeared in the market after the submission of this paper, has an access to outstanding multimedia resources including practical videos demonstrating essential bedside procedures, physical examination techniques, endoscopic and cardiovascular findings. (Raw data covering Longo et al's (2012) number of images, illustrations, graphs, tables, references and readability are shown in online supplementary material 2).

Table 4.

Comparing the 47 Wikipedia entries on cardiovascular diseases and corresponding topics in Longo et al (2012)

Item	Wikipedia entries*	Longo et al (2012)†	p Value‡
Images	93	150	0.998
Illustrations	69	51	0.148
Graphs	14	41	0.055
Tables	03	85	0.004
References	1218	582	0.002
Readability§	14.3±1.7	16.7±1.5	0.524

CD provided with the textbook (AccessMedicine Resources). Longo DI, Fauci AS, Kasper DL, Hauser SL, Jameson JL, Loscalzo J Harrison's Principles of Internal Medicine, 18^th Edition. New York: McGraw-Hill, 2012.

*47 entries on cardiovascular disorders, from the English Wikipedia.

†The assessment covers corresponding topics in the textbook plus accompanying extra-resources online, Access Medicine, provided with the book.

‡p Value calculated using χ² test, considered significance if p<0.05.

§Readability was calculated for each article and presented as (mean±SD).

Agreement among evaluators

The inter-rater agreement for the modified DISCERN instrument was calculated using the Fleiss κ scale; the overall agreement was good (Fleiss κ 0.718 (95% CI 0.57 to 0.83).

Discussion

The aim of this study was to assess the accuracy and readability of Wikipedia entries on cardiovascular diseases and whether they are suitable as a learning resource for medical students. Although the Wikipedia entries covered the aetiology, clinical picture of the disease and its treatment, there were some deficiencies in the pathophysiology of diseases, signs and symptoms, diagnostic approach and treatment. Some entries were incomplete and treatment options available were not well covered. To assess the accuracy of contents of Wikipedia entries, the evaluators used a modified DISCERN instrument, because the original instrument: (1) has been criticised for not analysing the quality of the information in significant detail,³⁸ (2) does not analyse tables, images or illustrations as part of its evaluation parameters and (3) was originally designed to analyse websites on patient's education and not academic work. The modified DISCERN instrument was used in an earlier publication^21–23 and was piloted before its use in assessing the 47 entries. The piloting of the work familiarised the evaluators with applying the instrument and enabled them to master its use with minimal disagreement. The agreement between the evaluators was an acceptable value (Fleiss κ 0.718 (95% CI 0.57 to 0.83).^32–34

The articles covered key points such as disease aetiology, clinical picture and treatment. However, there were deficiencies in most articles, regarding the pathophysiology of diseases, diagnostic approach and treatment. Also, significant variations in covering the appropriate subtitles and key concepts were noted in some articles.

In addition, key areas in some articles were not properly addressed and needed further details. For example, the diagnosis section in the article on pericardial effusion was only based on images of investigations such as radiology and ECG, with a brief explanation of the findings without addressing approaches needed to diagnose a patient with pericardial effusion. On the contrary, some articles contained information that was too detailed, such as the types and subtypes of heart valves, mentioned in the article on artificial heart valves. In addition, there was a variation in the length of the paragraphs within each article and the length of the articles.

The results from this study showed that the total number of references was 1218, and of these, 790 were from peer-reviewed journals. The number of references varied from 1 to 127 references; 25.9±29.3 (mean±SD). The references comprised a range of sources including peer-reviewed journals, educational/procedural guidelines, textbooks, professional websites, general websites, as well as news and media. Although the percentage of references from peer-reviewed journals comprised 65% of total references, peer-reviewed journals from the past 5 years comprised only 18% of the total references. These peer-reviewed journals included journals of high impact factors such as Hypertension, Circulation, the American Journal of Cardiology, the Lancet, the Journal of the American Medical Association and the New England Journal of Medicine. The correlation between the DISCERN scores and the total number of peer-reviewed journals included in the references (p<0.001), and between the DISCERN score and the number of peer-reviewed journals in the past 5 years (p<0.001), indicates the importance of peer-reviewed references to Wikipedia entries in enhancing and making their content more accurate.

However, the Wikipedia entries were deficient in citing educational guidelines produced by cardiology societies and associations such as the American Heart Association, the European Society of Cardiology and the British Hypertension Society. Only 6% of the references were guidelines. In addition, several problems were observed in the way the references were presented and cited. These findings are in agreement with Haigh,¹⁷ who concluded that Wikipedia citations should be treated with some caution.

The study also showed that Wikipedia entries are regularly updated. Yet, the frequency of updating an entry varied significantly (figure 1). The reason for this variability may be due to a number of factors: (1) the lack of an editorial team to distribute work and ensure regular revision of each Wikipedia entry, (2) the review process being dependent on the interest of those Wikipedians and viewers willing to contribute and (3) topics that are popular and frequently referred to being more likely to receive attention and most likely to attract the attention of Wikipedians. Collaborators from different backgrounds and different areas of interest usually conduct the update of Wikipedia entries. As per the system created by Wikipedia, anyone can update or start a new Wikipedia entry without even registering. There are no assigned editors or topic coeditors appointed to review articles and ensure that they are scientifically correct, up-to-date, organised and serve an educational purpose. This has resulted in variability in the quality of articles, their length and the accuracy of their content.

It is obvious from this study that some entries need images, illustrations or tables to enrich their educational value. Our findings showed that the mean number for images, illustrations and tables was 3.8±3.1, indicating the need for Wikipedians to consider this area when updating articles. Also, the inclusion of audio recordings, video clips and multimedia to articles is recommended. The comparison between the Wikipedia entries and the Textbook (table 4) shows that Harrison’s Principles of Internal Medicine 18th Edition had more tables, less references and no significant difference in number of graphs, images, illustrations or readability level.

There has been an extensive debate about whether health information, such as from Wikipedia, on the Internet should be controlled, rated or validated in some way by experts/professionals. Suggestions about possible means for achieving such goal to improve the quality of Wikipedia articles have been recently discussed.¹² These suggestions included collaborating with health organisations and the involvement of experts to improve the content of medical articles on Wikipedia. Medical schools could challenge their students to analyse Wikipedia entries critically and contribute to editing articles.¹⁵ Also, medical and health-related societies/organisations could encourage their members or task groups to be involved in such projects and even provide a statement of approval on articles.³⁹

Wikipedia has implemented several safety measures and quality surveillance mechanisms to identify and repair inaccurate information, and to improve the quality of its articles. For example, the initiation of a protection policy that limits contributors from editing or updating an article.⁴⁰ However, until now, most of the articles are unprotected.⁴¹ Also, Wikipedia launched WikiProject, with the aim to improve its articles. The English Wikipedia comprises about 2000 WikiProjects and one of them is the WikiProject Medicine.⁴² The latter project aims to improve existing medical/health-related articles, enhance the quality of references and add additional pictures to the articles.¹⁵ Wikipedia also has a self-rating system of stubs, start-up to good and featured articles, indicating that Wikipedia aims at conveying not only content but also tries to lay an emphasis on the quality of entries. In addition, Wikipedia has an electronic citation template but its use is not obligatory. A suggestion to improve the writing of references is to control the referencing style by using an electronic template that all authors have to use, thus limiting inconsistencies when citing information. Other suggestions that can help in improving the quality of the cardiovascular articles may include adding audio clips for the sounds of murmurs, and video clips on cardiovascular diseases, labelling all diagrams and pictures, adding motion echocardiograms and linking embryology illustrations to congenital heart diseases. These improvements would enhance the quality of articles and their educational value.

This study is not free from limitations. First, Wikipedia articles on cardiovascular diseases may have improved since this research was conducted. Second, this research focused on adult cardiovascular diseases in the English language only and did not study articles on paediatric cardiovascular diseases or articles in other languages, so this research cannot be generalised to other medically related topics on Wikipedia. Third, despite piloting the use of the modified DISCERN instrument and the good inter-rater agreement between assessors, the modified DISCERN instrument may not enable the identification of each error or deficiency in the Wikipedia entry. Also textbooks were used as a resource reference and there may be some differences in their content. Fourth, the study focused on only 47 entries on common cardiovascular diseases and did not examine other entries such as pathology, anatomy, physiology, toxicology, microbiology, pharmacology and public health-related to the cardiovascular system.

Conclusions

Despite the enormous number of volunteers (Wikipedians) who have contributed to the Wikipedia cardiovascular entries, a number of deficiencies in the content, particularly in pathophysiology of diseases, signs and symptoms, diagnostic approaches and treatment, were noted. The use of images, illustrations and tables was not optimum and most entries did not include audio recordings, video clips or multimedia to enrich the educational purpose of these entries. Although peer-reviewed articles were cited in the references, several problems were encountered with the references. The references did not refer to educational guidelines and position statements produced by major international cardiovascular societies and associations. The entries lacked accuracy, predominantly due to errors of omission. However, Wikipedia is not intended to be a textbook for medical students. Wikipedia is deliberately not intended to meet the needs of a medical audience. It is written for general readers. Therefore, the deficiencies in Wikipedia entries may be considered when readers want to use these entries in place of proper medical resources, which is not the purpose of Wikipedia. However, improving the quality of Wikipedia cardiovascular articles by scholars carries the potential to improve the educational usefulness of this online resource.