Wikipedia in Health Professional Schools: from an Opponent to an Ally

Abstract

As an online encyclopedia, Wikipedia is the world’s largest reference Web site, with 1.7 billion visits per month. Given how easy it is to access and read, students use Wikipedia globally, despite most faculty members’ admonitions. Since 2013, health professional schools worldwide have incorporated Wiki-editing into their formal curricula. These courses impact students by (1) strengthening their ability to evaluate evidence-based content and (2) multiplying their contributions to society through improvements to Wikipedia articles accessed by millions. We showcase several models of incorporating Wikipedia-editing assignments into health professions education worldwide. These successful initiatives can be replicated everywhere.

Introduction

Wikipedia is a collaborative and multilingual online encyclopedia launched in 2001 and is continually created and updated based on a freely editable content model. Its name comes from merging “wiki,” which means “quick” in Hawaiian, and encyclopedia. As of November 2020, Wikipedia is the world’s largest reference Web site and attracts a total of 1.7 billion monthly visits [1]. Wikipedia is the most visited Web site for medical information in English by the general population, receiving more than 4.8 billion views in 2013 [2]. Among the reasons for this success are easy accessibility, no space limitations for articles, and quick updates [1].

As anyone can edit Wikipedia’s content, a serious concern is that any article may contain misinformation [1]. Regarding reliability, in 2005, the journal Nature invited blinded expert reviewers to read and look for errors in general articles from both Wikipedia and Encyclopaedia Britannica. Surprisingly, the number of serious errors was identical, and the average inaccuracy was very similar—3 per entry in Britannica Encyclopaedia, and 4 per entry in Wikipedia [3]. A volunteer community also receives automatic notifications when Wikipedia medical content is updated—thus minimizing vandalism [4]. Most health-related changes on Wikipedia articles are assessed by editors in the Wiki community within 1 day [5].

Wikipedia’s medical content comprises more than 155,000 articles worldwide. This content is supported by more than 950,000 references, most of which are recently published in the best known medical journals, such as The New England Journal of Medicine, Cochrane Database of Systematic Reviews, Lancet, and Nature [6, 7]. Wikipedia articles are graded in quality on a scale from Stub (lowest quality), though Start, C, B, Good, and Featured (highest quality) article. Peer review is required to achieve a quality rating of Good or Featured Article. As of August 2020, of Wikiproject Medicine’s 97 top-importance Wikipedia articles, 83% were at B-class or above [8]. Of note, Wikipedia health-related article quality is much higher than other domains of Wikipedia. In 2017, compared to 25% Wikipedia-wide, only < 1% of high-importance and top-importance medical articles were Stub quality [9]. As of April 2019, the 34,324 English articles maintained by Wikiproject Medicine were four times more visited than other Wikipedia articles, averaging 5,875,470 daily visits for all articles [10].

Health professional students find Wikipedia-editing courses enjoyable and effective in the goal of preparing students to contribute to Wikipedia. Students also broaden their sense of responsibility in the era of digital communication [11, 12]. Given Wikipedia’s tremendous impact on society, we sought to showcase several models of incorporating Wikipedia-editing assignments into health professions education across the globe.

Wikipedia: How Often Do In-training Health Professionals Use It? Is it a Matter of Concern?

Most faculty members orient their students not to use Wikipedia for academic purposes. However, data has already shown that, despite this recommendation, most medical students and physicians use Wikipedia as an information source during their training. An assessment of 1658 Australian undergraduate students revealed that 87.5% considered Wikipedia to be part of their University studies [13]. Of 1365 medical students at different training stages in different European countries, 67% disclosed using Wikipedia in a high or moderate manner for medical information. Notably, this use for medical purposes was correlated with general use, meaning that those who use Wikipedia for other reasons tend to use this tool more often for medical education. Furthermore, 97% of students affirmed they found inaccurate entries on Wikipedia and, of those, 65% did not know how to correct this content [14]. Since overall Wikipedia’s visits are increasing over the years, we speculate that health professional students’ use of this platform is also increasing, despite faculty advice in the opposite direction.

A 2014 Harvard Medical School survey of 62 psychiatry residents asked what resources were used for educational purposes. Among the 57 respondents, only PubMed (97%) and UpToDate (91%) were used more often than Wikipedia (88%). While acknowledging that it is not as trustworthy as PubMed or UpToDate, residents considered Wikipedia the easiest-to-use platform [15].

The fact that Wikipedia is so easy to use makes it very attractive for health professional students, who often have to get quick concepts to fill in their training gaps. In a randomized controlled trial designed to assess quick learning, a group from Canada assessed knowledge acquisition from 3 different resources: UpToDate, Wikipedia, and a digital textbook. A total of 116 medical students completed a multiple-choice pretest with 25 questions, and they were allowed to take notes from questions they would like to further research. After the pretest, they had 30 min to research using only the resource they were randomly assigned. Students then had a post-test to assess short-term knowledge acquisition. As expected, there was no difference in scores between the three groups in the pretest. Posttest, Wikipedia’s group performed better than the digital textbook group (P < 0.001). Although it did not achieve the classic statistical significant threshold, there was a trend for Wikipedia’s group to perform better than the UpToDate group (P: 0.08) [16].

Wikipedia as an Ally in Health Professional Schools—an Emerging Global Community Since 2013

USA—University of California, San Francisco

Given the widespread use of Wikipedia as a health information source for medical students, in 2013, the University of California, San Francisco (UCSF) School of Medicine started a formal Wikipedia-editing course integrated into the curriculum as an elective course for 4th-year medical students. The goal was to transform medical education into a Wikipedia ally, enabling students to contribute to society while studying to edit and enhance health-related articles quality.

The course is a 4-week elective offered 1–3 times each academic year and is generally co-led by a faculty member and a librarian. Student enrollment has ranged from 3 to 37 students, all of whom are enrolled exclusively in this full-time course (e.g., they are not concurrently enrolled in other courses). Each 4-week course begins with a 1–2 day immersive “boot camp” experience to provide foundational preparatory guidance. Students then select a single Wikipedia article to improve throughout the duration of the course. Students can work alone or in teams of up to 4 students. Weekly hour-long work-in-progress (WIP) sessions are offered synchronously, either in-person or remotely via video-conferencing, and students are expected to attend a minimum of 75% of those sessions. These sessions allow students to troubleshoot technical challenges (e.g., “How do I insert the DOI to create a citation?”, “Where do I log into the UCSF Library website to gain access to our journal subscriptions?”, etc.) as well as intellectual challenges (e.g., “Do you think I should list this rare side-effect?”, “Should we include links here to other Wikipedia pages vs. a table or a bulleted list?”, “How should I respond to this Wikipedian’s comments on my work on the talk page?”, etc.). The course includes a required peer-review of a classmate’s work in the penultimate week, which allows students to respond to peer-review in the final course week. The peer-reviews are posted publicly on Wikipedia’s associated talk pages, thereby allowing other Wikipedians to address recommended improvements long after the UCSF student completes the course. The final day of the course incorporates a synchronous activity that includes student reflection on (1) recommendations for course improvement and (2) evolving identities as physicians in the digitally interconnected world. Table 1 summarizes the 15 cycles of the UCSF course between 2013 and 2020.

Table 1.

Summary of UCSF School of Medicine course 2013–2020

Month and year	Number of enrolled students	Articles edited	Total edits	Words added	References added	Images added	Article views during active course dates
Nov–Dec 2020	4	8	155	9070	65	0	124,000
March 2020	7	7	287	12,700	122	0	175,000
Dec 2019	7	8	301	17,300	195	6	583,000
Nov 2019	2	5	79	4250	27	0	67,700
Mar 2019	10	13	261	18,300	264	7	301,000
Nov 2018	20	21	628	41,700	571	4	685,000
Mar 2018	13	12	701	34,300	381	10	994,000
Nov–Dec 2017	37	36	1,420	61,200	705	14	4,830,000
Mar 2017	8	7	236	12,000	182	0	3,260,000
Nov–Dec 2016	22	21	843	35,200	541	5	5,030,000
Mar 2016	10	1	350	21,900	480	2	2,760,000
Oct–Nov 2015	18	17	145	4210	− 11 (e.g., net overall deleted lower-quality references)	4	7,680,000
Nov 2014	16	16	841	Standardized assessment tools/metrics were not yet developed at this time		0	230,000
Mar 2014	7	8	546			0	285,000
Nov–Dec 2013	5	5	662			0	459,000

Wikipedia-editing courses began spreading to other US medical schools in the mid-2010s. In the 2017–2018 academic year, there were 3 US medical schools that incorporated Wikipedia-editing into formal credit-bearing courses. In total, 78 students made 3368 edits to 71 Wikipedia pages, adding 155,100 words [17]. Subsequently, the movement to incorporate Wikipedia-editing into health professional schools spread worldwide. A scoping review of Wikipedia’s use as a health information source was recently published [6]. Consequently, rather than replicating that study, we deliberately chose to highlight several complementary global exemplars of Wikipedia-editing within both formal and informal health professional education contexts.

Israel—Tel Aviv University

The School of Medicine at Tel Aviv University (TAU) has been offering its Israeli Medical Program students a for-credit elective course focusing on Wikipedia since 2013 [18]. The semester-long course is available at TAU once a year and spreads over 13 weeks, in which students learn to contribute medical-related content to Wikipedia and its sister projects (mainly WikiCommons and Wikidata). The course was designed with 5 main goals: (1) that students contribute medical-related content to Wikipedia and its sister projects, thus contributing to the Open Educational Resources available to both future learners and the general public; (2) that students improve their academic skills, critical thinking, digital literacy and data literacy, work on their collaborative skills and medical communications skills, as well as gain life-long learning competencies; (3) that the course addresses topics such as knowledge gaps, and specifically the gender gap, and focuses on knowledge equity, including accommodating students from varying backgrounds (such as different mother tongues, social-economic background, and digital skills); (4) that the learning process is engaging and positive, focusing on active learning; and (5) that the course model is adaptable, scalable, and could be reused.

The course structure included 2 introductory sessions, 9 core sessions teaching skills for contributing to Wikipedia, WikiCommons, and Wikidata, and 2 sessions of students’ presentations, reflecting on learning processes and helping instructors improve future interactions. During the course each student created 2 new articles. Each article submission included a process of initial submission, a peer evaluation process, corrections and final submission. Assessment was based on 5 parameters (out of 100% of the total grade): first article grade (40%); second article/Wikidata project grade (40%); peer assessment (10%); active participation and making deadlines (5%); reflection presentation (5%). Students were expected to attend 80% of classes. The course used to be offered as a live, synchronous course, with a weekly meeting of 1.5 h each (2 academic hours). During COVID-19, the course was transformed to an online course format using a hybrid model, offering 8 synchronous sessions, and 5 asynchronous ones.

Since its launch 8 years ago, 298 students have participated in it, including students with varying backgrounds and mother tongues, among them Hebrew, Arabic, and Russian. Together, the students wrote 397 new medical articles, including roughly 13% of all medical content in Hebrew Wikipedia, which have been viewed over 7 million times. They have also edited additional 1150 articles (some expanded significantly from a stub level), uploaded free-licensed media files to WikiCommons and contributed structured data to Wikidata. Since 2015, at least 50% of the articles created by each class have focused on Women’s Health, a category in Hebrew Wikipedia that did not exist before the course, and which has been populated with articles ever since.

Overall, students reported a unique and positive learning experience, which included gaining and improving skills, as well as a lasting impact beyond the course [18]. Additionally, the course model has been adapted to other disciplines and countries: in 2015, the course model was adapted to accommodate other disciplines, and a second course was created and approved by the University’s Rector, and has since been offered on campus to all undergraduates, from all disciplines. The course model has also inspired similar courses and trainings, such as one developed in Armenia’s Medical School in Yerevan.

Australia—University of Notre Dame

Beginning in 2016, every first-year student enrolled in the School of Medicine at the University of Notre Dame Australia (Fremantle) has learned how to edit a WikiProject Medicine page in their orientation week. The Australian Medical Council accreditation guidelines require that medical graduates access and critically appraise, interpret, and apply evidence from the medical and scientific literature. The Wikipedia training task is the first step in a research training pathway undertaken by all students over their 4 years in graduate medical school. The orientation task requires students to create an account and complete the Wiki-editing training tutorials. In groups of three and guided by an academic mentor, they select a Wiki page from the quality and importance tables. Students then plan a minimum of four simple edits (e.g., improving grammar, broken links, adding an image). Many students complete the modifications in the 6 h allocated to this task. A semester later, at the commencement of the next step in the research training pathway, students report on the activity related to their Wiki page, noting the page statistics and the fate of their edits. This task introduces medical students to Wikipedia’s rules and guidelines that govern content specifically related to medical and health topics. Specifically, the aim is for students to understand scientific writing principles, copyright laws and levels of verifying evidence specific to medical and health-related issues. Also, they are informed of the social contribution that participation in Wikipedia editing at any level affords.

UK—A Consortium of Dental Schools

The Wikipedia Collaboration of Dental Schools, led by the University of Dundee, is an initiative that invites dental students to edit Wikipedia on an extra-curricular basis. Since 2016, when the group was formed, 9 dental schools across four continents have voluntarily contributed to the Project. At the commencement of the academic year, students attend a training session that equips them with necessary evidence-based dentistry knowledge as well as editing skills to be able to improve Wikipedia pages. Subsequently, each dental school divides its editors into subgroups, ranging from 3 to 6 students. On an annual basis, approximately 90 students across all schools participate. Any number of Wikipedia articles can be edited so long as the pages are selected in advance and a list is shared with all schools. Editors work to improve pages throughout the academic year, with input and support from faculty and staff, particularly regarding the references cited. At the conclusion of the academic year, a Conference is held annually in Dundee where each dental school shares a summary of their edits. The summary can be presented either in person or through a poster presentation, engendering discussions about evidence, dental topics, and Wikipedia articles.

Canada—Queen’s University

In this course, the Wikipedia Project is longitudinal, includes approximately 100 students per year, and lasts 1–2 months. Queen’s faculty experts start by identifying a list of Wikipedia articles that need to be improved. Except in certain circumstances where the evidence is evolving quickly (for example, the concussion Wikipedia article), experts are encouraged to choose articles rated at “B” level of quality to ensure that there are sufficient gaps in the content for the students to build on. To allow the students the opportunity to evaluate and assess the information already shared in the article and build on the existing content, Stub or Start quality articles are avoided for this assignment. Through collaboration with medical librarians and a local Wikipedia expert, the medical students receive training and complete an assignment on evidence-based medicine and how to assess appropriate secondary sources for Wikipedia. The medical students then work together in small groups with their assigned content expert to identify gaps in the article and find appropriate sources to improve the Wikipedia articles. Students receive a practical 50-min Wikipedia Skills session workshop and practice editing in their “sandboxes,” a practice area on Wikipedia. Working with the Wikipedia expert, students then upload concise details about their proposed improvements to the article talk pages before editing the Wikipedia article. This gives the students an additional opportunity to practice their editing prior to editing the main article, work with the existing community of volunteer Wikipedia editors, and have the faculty content expert and the Wikipedia expert moderate their work. After approximately one week of monitoring the talk page and responding to feedback, the students add their suggested improvements to the current article. Finally, the Wikipedia expert reviews all final Wikipedia submissions to ensure that the contribution to the live Wikipedia article is high quality and follows Wikipedia's Guideline for Reliable Sources in Medicine and the Medical Manual of Style.

Sweden—Uppsala University

All Ph.D. students in medicine (about 40 each semester) undergo a 5-week introductory course in research studies encompassing Research ethics, Statistics, Research methodology, and Scientific presentation where editing Wikipedia is a part of the theme Scientific presentation. The Wikipedia part consists of two lectures. The first one takes place at the beginning of the course and focuses on how Wikipedia works and how it can be used for science outreach. Then, the Ph.D. students have an assignment on their own where they are asked to make at least two edits on Wikipedia within their research area. The research areas edited cover all aspects of medicine such as vision in insect models, protein–protein interactions, psychology, maternal health, and surgery. Either students should make two edits in English or one in English and one in their mother-tongue. Each edit should contain a reference to a published work with a strong preference for secondary sources. Before the course, all Ph.D. students have been “consumers” of Wikipedia, but very few have been “producers.” At the end of the course, when they all have edited Wikipedia, the second lecture consists of examples of edits that the Ph.D. students had made during the course and highlights good examples and examples when the Ph.D. students encountered problems. Since all examples are from the Ph.D. students in the classroom/lecture hall and they are asked to comment on their edits, the second lecture becomes very interactive.

Canada—University of Manitoba

The College of Pharmacy at the University of Manitoba requires students to participate in 25 h of service-learning programs in the first and second years of the program. The second year of service-learning started in 2020 and requires students to work on a project for a site that focuses on health promotion and advocacy. The pandemic of 2020 required the program to step “outside the box” to find opportunities for students to complete hours in a safe manner. The Wikipedia editing project met program Learning Objectives and provided students the ability to work remotely and to be involved in a global initiative. The second-year service-learning course started in September 2020 and ends in May of 2021. As of March 2021, the Manitoba Wiki project had 12 pharmacy students working on articles of their own choice that are all health related. Students are required to complete the training sessions before choosing an article to work on and then edit that article in their “sandbox.” Students are provided 4 h per week for service-learning but are encouraged to work on their projects at any time. The coordinator and students met several times during the process to discuss any challenges that occurred and to keep students motivated. Anecdotally, students reported that the training sessions were excellent and something that was valuable prior to searching for an article that they wished to work on. The students recognize the value of their contribution to improving the articles they chose and indicated that they are excited and proud to be contributing to something that provides information that is evidence based. This pilot Wiki project has worked well and will continue to be offered to students as part of their service-learning program in the future.

Another Global Model—International League Against Epilepsy (ILAE)

The ILAE tries to motivate epilepsy professionals worldwide to contribute to Wikipedia to enhance the quality of scientific content of the Wikipedia articles, so that people across the world can have access to quality information on the subject. Members of the Young Epilepsy Section (YES) of the ILAE usually play an active role in the project. The existing articles within the subject’s purview are identified and reviewed by reviewers or subject matter experts, and a to-do list is generated for the given article. The authors/editors thereafter make the necessary changes, and the article is reassessed. Experienced Wikipedians oversee the entire process. If applicable, the article is submitted for a Good Article or Featured Article review process. The authors/editors' contributions are acknowledged by both the ILAE and other forms of Wikipedian recognition.

The Wiki Education Foundation

Since 2013, the Wiki Education Foundation (Wiki Ed) has promoted inclusion of Wikipedia-editing assignments into formal courses at universities in the USA and Canada. The non-profit has created a dashboard tool that simplifies the process of assigning and tracking students’ work contributing to Wikipedia. Summarizing the impact since 2010: over 91,000 students in over 4400 courses across 500 universities have worked on at least 102,000 Wikipedia articles. Through their work in these formal academic courses, students have added over 76 million words to Wikipedia [19–21]. More narrowly within the health professions space, WikiEd aggregates courses into a “Students in the health professions” campaign. As of July 5, 2021, WikiEd listed 100 courses within that campaign: 1560 students added 1.35 million words, 16,800 references, and 153 images to 973 health-related Wikipedia pages. Students also created 11 new pages. In total, these 973 health-related Wikipedia pages were viewed 67 million times during only the active dates of the respective courses [22].

Additionally, since 2018, WikiEd has been training scientists and scholars across non-university institutions (e.g., foundations, museums, academic associations, and organizations) to contribute their scholarly expertise to Wikipedia. The “Scholars and Scientists” campaign lists 27 programs of 349 contributors. They have added 808,000 words and 8640 references to 1760 Wikipedia articles. Within this campaign, editors have created 153 new Wikipedia pages and added 239 new images [23].

Using the WikiEd dashboard tool, tutors and faculty can track students’ progress by easily assessing data such as which article each student is editing, number of words changed, references added, and images inserted. This allows all faculty-of-record to subjectively evaluate their own students’ quality improvement efforts. It is also possible to track how many times the article was accessed during the assignment. Because each school designs their course structure differently, we do not have a systematic approach to evaluate the degree of quality improvement of all students’ efforts. In the UCSF initial courses offered between 2013 and 2015, from 28 articles edited, faculty considered that 26 improved and 2 did not improve. Regarding the Wikipedia quality scale, 11 improved their grade and 17 maintained the previous grade [11]. Anecdotally, the faculty course directors of all the other courses listed in our summaries believe that their students have improved the quality of their respective Wikipedia pages, in some cases with dramatically significant improvements. Additionally, it is possible to request that Wikipedians “regrade” articles after students have made significant contributions to any Wikipedia article. While this regrading process is subjective, since it is conducted by longtime and active Wikipedia editors, it may be more objectively robust than the subjective interpretations of faculty members’ impressions of student contributions to Wikipedia.

Students’ Perspectives

Students’ perspectives of these Wikipedia editing experiences varied due to several factors, including type and level of learner, the different course structures, the location within students’ other curricular obligations, and how much academic credit students were receiving for these efforts. Nevertheless, here are several perspectives in students’ own words:

• “Editing Wikipedia pushed me to think of the vast amount of lay persons that could benefit from expert medical information that may otherwise be unobtainable for them. It gave me a sobering appreciation for the knowledge I am able to share with others through this universally accessible portal.”

• “I realized how easy it was to change Wikipedia.”

• “As Physicians we have a responsibility to fix egregious errors in medical knowledge.”

• “It’s really hard to write simply….It took a while to get out of that mindset of talking like a doctor or talking like a medical professional.”

Limitations

The examples we have compiled above are not meant to be a systematic review nor a comprehensive summary of existing health professionals that embrace Wikipedia as an ally. It is quite possible—indeed likely—that we have omitted other innovative educational initiatives that align health professions education with contributions to Wikipedia’s health-related content. Given the diversity of examples we showcased, we have also not attempted to evaluate or compare the outcomes of these courses, campaigns, and initiatives. Other open questions include all of the following:

1. Do students who contribute during their health professions’ training contribute to Wikipedia later in their professional careers?

• 2.As a result of these courses, do students adapt/evolve/broaden their sense of professional identities and/or scope of duties as future health professionals?

• 3.What are the most effective or optimal ways to integrate Wikipedia-editing into the formal curricular structure of various health professions training programs?

• 4.What is the impact of Wikipedia-editing courses on the faculty who design and deliver the courses?

• 5.In the wake of the COVID impact on health professions education writ large, how might these courses/assignments evolve?

Conclusion

Wiki-editing courses have spread over many institutions worldwide. The diversity and breadth of the examples we summarize demonstrate an emerging “ally-ship” between health professions education and Wikipedia. Teaching students to contribute to Wikipedia provides excellent opportunities for students to develop evidence-based medicine skills [24–26]. Notably, there are a myriad of ways to deliver these courses to the audience, including live sessions, videos, and online interactive platforms, any/all of which can be delivered both in person and remotely [25]. The target audience varies from 1st-year students to professionals, and different faculty adapt the methodology to their local institutional environment, as shown in Table 2. Especially in the context of our global COVID-19 pandemic, we believe that broadening this Wikipedia-editing movement to more health professional schools will help fulfill what we paraphrase from Wikipedia’s vision: providing free access to the sum of all human health-related knowledge.

Table 2.

Summary of different health-related Wiki-editing courses

Institution	Starting year	Type	Duration	Enrolled students	Assessment
UCSF, USA	2013	Elective	4 weeks	4th year medical students	•Peer assessment of Wikipedia pages •Course is pass-fail for the enrolled students
Tel Aviv University, Israel	2013	Elective	13 weeks (semester long)	1st year medical students	•1st article grade •2nd article grade •Peer assessment •Active Participation •Reflection on learning process presented
University of Notre Dame, Australia	2016	Required	1 week in orientation	1st year medical students	•Non graded pass for satisfactory completion •Students complete and submit a form requiring identification of the edits made, the number remaining after 6 months, and the number of page views in that period
UK Consortium of Dental Schools	2016	Extra-curricular	9 months	Undergraduate Dental students	•Literature review and identification of sources suitable for citation on dental articles is carried out by students then reviewed by a faculty member •Peer review of content takes place before publication of material onto Wikipedia •Participants are rewarded proportional to the quantity of their contributions to Wikipedia articles
Queens' University, Canada	2017	Required	8 weeks	1st year medical students	•Critique of a Wikipedia article •Literature review and selection of sources appropriate for Wikipedia •Preparing the final article improvement and sharing it on the Wikipedia talk page •The final contribution on Wikipedia is moderated by the course coordinator but marked based on completion of the activity
Uppsala University, Sweden	2018	Required	4–5 h over a 5-week course	PhD students in medicine	•Teacher evaluates edits
University of Manitoba, Canada	2020	Required	25 h (over academic year—2 terms)	2nd year pharmacy students	•This is a pass/fail service-learning course with a focus on health promotion •Peer assessment recently introduced •Course coordinator reviews material regularly

Declarations

Ethics Approval

Not applicable.

Conflict of Interest

The authors declare no competing interests.