Situating Wikipedia as a health information resource in various contexts: A scoping review

Denise A Smith

doi:10.1371/journal.pone.0228786

. 2020 Feb 18;15(2):e0228786. doi: 10.1371/journal.pone.0228786

Situating Wikipedia as a health information resource in various contexts: A scoping review

Denise A Smith ^1,^2,^*

Editor: Stefano Triberti³

PMCID: PMC7028268 PMID: 32069322

Abstract

Background

Wikipedia’s health content is the most frequently visited resource for health information on the internet. While the literature provides strong evidence for its high usage, a comprehensive literature review of Wikipedia’s role within the health context has not yet been reported.

Objective

To conduct a comprehensive review of peer-reviewed, published literature to learn what the existing body of literature says about Wikipedia as a health information resource and what publication trends exist, if any.

Methods

A comprehensive literature search in OVID Medline, OVID Embase, CINAHL, LISTA, Wilson’s Web, AMED, and Web of Science was performed. Through a two-stage screening process, records were excluded if: Wikipedia was not a major or exclusive focus of the article; Wikipedia was not discussed within the context of a health or medical topic; the article was not available in English, the article was irretrievable, or; the article was a letter, commentary, editorial, or popular media article.

Results

89 articles and conference proceedings were selected for inclusion in the review. Four categories of literature emerged: 1) studies that situate Wikipedia as a health information resource; 2) investigations into the quality of Wikipedia, 3) explorations of the utility of Wikipedia in education, and 4) studies that demonstrate the utility of Wikipedia in research.

Conclusion

The literature positions Wikipedia as a prominent health information resource in various contexts for the public, patients, students, and practitioners seeking health information online. Wikipedia’s health content is accessed frequently, and its pages regularly rank highly in Google search results. While Wikipedia itself is well into its second decade, the academic discourse around Wikipedia within the context of health is still young and the academic literature is limited when attempts are made to understand Wikipedia as a health information resource. Possibilities for future research will be discussed.

Introduction

“Imagine a world in which every single person has free access to the sum of all medical knowledge” [1]. This is what WikiProject Medicine, a community of editors who develop the health care content in Wikipedia, is doing and it hasn’t gone unnoticed. Wikipedia is the most frequently accessed resource for health information: its English language medical content has received more unique pageviews than any other health information resource online [2]. Of course, any discussion of Wikipedia would be incomplete without acknowledgement of the stigma attached to it. It is common knowledge that there is a general wariness around using Wikipedia [3]. However, a paradigm-shifting study conducted by Nature found that Wikipedia and Encyclopedia Britannica were comparable in the number of errors found in a sample of entries from each resource [4]. This finding has not been challenged since its publication and was vital in the ongoing effort to de-stigmatize Wikipedia.

Further, an exploration of what might motivate editors to contribute to health-related articles on Wikipedia found that editing Wikipedia is influenced predominantly by an editor’s inherent values–a belief that all knowledge should be free; and a sense of obligation that they have a duty to fix Wikipedia as health workers [5]. This single study offers unique insight into the motivations of those who voluntarily edit existing content, translate studies and reviews into plain language, or translate content to languages other than English. While the findings of this study are not generalizable, they imply that behind the pages of Wikipedia is a community of (mostly) altruistic editors motivated by a shared core value that information should be available and accessible to all communities.

Today Wikipedia is the 7^th most frequently accessed web site in the United States and 8^th in Canada [6,7]. For perspective, English Wikipedia alone averages approximately 830 million unique visits per month globally [8]. Its content is regularly one of the top results in Google searches, with about 93% of clicks on Wikipedia directed from Google [9]. However, efforts to understand, in depth, Wikipedia’s role in the landscape of health information resources are limited. While systematic reviews of the scholarly literature covering Wikipedia have been conducted [10,11] a comprehensive literature review that focuses specifically on the treatment of Wikipedia in the academic health literature has not yet been reported. This scoping reviews asks:

What does the existing body of literature say about Wikipedia as a health information resource? and,
What publication trends exist, if any, among health-related articles about Wikipedia?

Methods

Search strategy

A scoping review is a comprehensive and robust literature review that diverges from the systematic review in its aim to “summarize and disseminate [existing] research findings”, “identify research gaps in the existing literature”, and may not incorporate strict criteria, such as study design, as a requirement for inclusion in the review [12]. The literature search and retrieval process for a scoping review can be iterative in its execution and relies heavily on mapping concepts or themes within the body of literature. As a result, scoping reviews may place an emphasis on producing a highly sensitive search rather than a search with more balance between specificity and sensitivity, as is done in systematic literature reviews. To maximize the sensitivity of her literature search, the author developed and executed a search strategy (S1 Appendix) in June 2019 to retrieve any article where the term “Wikipedia” appears in either the title or the abstract of any article indexed in OVID Medline, OVID Embase, EBSCO CINAHL, and OVID AMED. As these three databases are clinically focused, it is likely that the term “Wikipedia” will be retrieved within a health context. The author also searched for the term “Wikipedia” in the titles or abstracts of all articles indexed in EBSCO Library and Information Science and Technology Abstracts (LISTA) and EBSCO Library Literature and Information Science Full Text (Wilson Web).

A literature search in Web of Science presented a special conundrum due to the limitations of its advanced search interface where it is not possible to perform keyword searches for terms appearing in the abstract. As a result, the author chose to conduct a sweeping search for the term “Wikipedia” in any Web of Science record, which she followed up with a manual screening of all results to remove irrelevant content. Content was considered irrelevant if the term Wikipedia did not appear in the title or abstract or if the title or abstract indicated the article was not health related.

The author’s exclusion of search terms related to health and medicine was intentional. While one may be inclined to use health-related terms in the search strategy, the breadth of terms that would need to be included in order to capture all health-related concepts would be prohibitively time-consuming and prone to human error. In order to get a sense of how and in what health fields Wikipedia is discussed in the literature, the author decided an attempt to create an exhaustive search string to retrieve anything health-related posed too great a risk. Instead, the author elected to conduct a broad search and use exclusion criteria to manually eliminate irrelevant records not related to health and medicine. A protocol for this review was not published or registered.

Study selection

All records were exported to a free citation manager (Zotero) and imported into systematic review (SR) software (DistillerSR). Using the SR software, database records were screened first by title and abstract. A second round of review screened the full text of all records. Records were excluded if English Wikipedia was not a major or exclusive focus of the article; English Wikipedia was not discussed within the context of a health or medical topic; the article was not available in English (no funds were available to support translation for this review); or the article was a letter, commentary, editorial, or popular media article.

In some cases, a conference proceeding record and a published article record were retrieved for the same study. In these instances, the conference proceeding was excluded. Some articles that met the criteria requiring a discussion of Wikipedia’s health or medical content were excluded because Wikipedia was not a major focus of the paper and the treatment of Wikipedia's health content was insufficient to merit inclusion.

8,030 records were retrieved in the literature search. After manually removing irrelevant studies from the Web of Science search results (n = 4,400) and removing duplicates (n = 1,417), the literature search retrieved 2,213 unique results (Fig 1) that were subsequently screened for relevance based on title and abstract content. 1,815 articles were excluded in the first round of screening. After a review of the full-text using the exclusion criteria outlined above an additional 309 articles were excluded. As a result, 89 articles and conference proceedings were selected for inclusion in this scoping review.

Thematic analysis

The objective of the author’s thematic analysis was to inductively analyse and identify any patterns, themes, or trends, that emerged through a reading of the full-text of all 89 studies included in the review. The author chose an inductive approach to remain open to any themes that may emerge from the literature as she read.

The first step in her thematic analysis included the generation of an annotated bibliography. The exercise of engaging with each article and annotating its methods, findings, and research questions, allowed patterns to naturally emerge. The thematic analysis was an organic process that unfolded as the author read each paper. As patterns emerged, the author maintained a table of themes and patterns. Upon completion of the annotated bibliography, the author organized each emergent theme into one of four larger categories in which the articles were placed:

Articles with a general focus on situating Wikipedia as a health information resource (S2 Appendix);
Assessments of Wikipedia’s quality (S3 Appendix);
Wikipedia’s utility for education (S4 Appendix), and;
Wikipedia’s utility for research (S5 Appendix).

Results

Situating Wikipedia as a health information resource

Approximately 1% (n = 9) of the 89 included articles focus on generally situating Wikipedia as a health information resource (Fig 2) by measuring its readership in comparison to other online health information sources that are already validated by the health or information community. Despite the small relative size of this category of the literature, these studies (S2 Appendix, Table 1) position Wikipedia as a heavily used health information resource. They do not focus on any particular context or user population.

Publications that position Wikipedia as a health information resource demonstrate this by referencing its frequency of use. Some leverage this data to call on the medical community to contribute to Wikipedia’s health and medical content. The Pew Research Centre collected evidence that establishes the high-use of Wikipedia for health information among the public, physicians, and medical students [13] and Heilman, et al. question the relevance of traditional peer-reviewed articles when Wikipedia’s health content can be updated in real-time, has a global audience, and is written in plain, accessible language [13].

A more recent investigation finds that Wikipedia’s medical content is now consistently superior to its non-medical content with 83% of its articles rated as B-Class, the highest quality rating an article can receive without undergoing internal peer-review [14]. Overall, the literature within this theme attempts to destigmatize Wikipedia by addressing its limitations as a health information resource and whether solutions for these challenges are a possibility [15].

Situating Wikipedia as a health information resource is also verified by its relative ubiquity in online searches. Although it is based on data from 2008, a 2011 study finds that Wikipedia is the first result for most online generic drug name searches [16]. Further, its general capacity is also helpful in positioning Wikipedia as a health information resource. At the end of 2015, Wikipedia had 155,085 medical articles, approximately 18% of which were in English Wikipedia. This content is supported by more than 950,000 references and the most commonly cited resources are top-tier medical journals such as, New England Journal of Medicine, The Lancet, Nature, and Cochrane Systematic Reviews [15]. A single study places Wikipedia as one of the most heavily accessed sources of online health information based on aggregated pageview data of Wikipedia articles under the umbrella of WikiProject Medicine [2]. It finds Wikipedia to be equal to that of the National Institutes of Health web site and surpasses WebMD, another dominant health information web site.

Other investigations into the validity of Wikipedia as a health information source include explorations of the opportunity Wikipedia’s health content offers globally [13]. For example, during the 2014 Ebola outbreak in West Africa, Wikipedia’s Ebola content was drastically updated, translated to more than 100 different languages, and was viewed more than 89 million times that year [14]. However, the most influential and widely cited study in the literature places Wikipedia at the forefront of online health information resources [17]. Laurent and Vickers (2009) find a statistically significant difference, in which Wikipedia takes prominence, between the pageviews of Medline Plus and Wikipedia for the same topics [17]. This seminal study positions Wikipedia as prominent among the existing complement of online health information resources.

Quality of Wikipedia’s health content

Assessments of Wikipedia’s quality are categorized by the population of concern to the study. That is, whether the quality assessment is conducted with a general information consumer or patient in mind, a health professional, a student, or whether a user of Wikipedia is not clearly identified in the study. The focus on assessments of the quality of Wikipedia’s health content within the literature is overwhelming. Despite efforts to affirm Wikipedia’s value as a source for health information, questions about its quality dominate the conversation. 53% (n = 47) of the articles included in this study assess the quality of Wikipedia’s health content (Fig 2). In this review, studies of Wikipedia’s quality include assessments of readability and reliability, wherein reliability may include any of the following measures: accuracy, comprehensiveness, completeness of coverage, number or quality of references, and currency.

For consumers

It is well established that the public uses the internet, and by extension Wikipedia, to find and consume health information. Of the articles included in this review, 19% (n = 17) were concerned with the suitability of Wikipedia’s health content for patients or consumers. These studies (S3 Appendix, Table 2) assess readability, reliability, and accuracy or completeness, but specifically discuss their findings in relation to the public consumer or patient. Of those that include results–some conference proceedings do not–they generally agree that Wikipedia is suitable for patients [18–20]. Further, a 2010 study claims that while Wikipedia is not the superior resource, its findings suggest that it is the preferred resource [21]. Therefore, it could be suggested that enquiries into Wikipedia’s suitability for patients may have limited impact if patients are likely to prefer Wikipedia anyhow. Unfortunately, these assessments of suitability for patients are also unlikely to be relevant given that their publication dates range between 2008 and 2013.

Some unique studies that examine the quality of Wikipedia include Skancke and Svendsen’s (2017) investigation into who edits Wikipedia’s content on medicines. Skancke and Svendsen explore the impact and potential implications associated with pharmaceutical representatives editing Wikipedia [22]. Another study investigates the impact of the contributor’s experience on the quality of Wikipedia articles through an evaluation of edits [23] and Cozza, Petrocchi and Spognardi use natural language processing software to automate the assessment of Wikipedia articles [24]. However, most studies focus on the information in Wikipedia about a specific health speciality. These include gastroenterology [18], nephrology [19], autoimmune disorders [25], medicinal drugs or herbal supplements [26–31], pathology informatics [32], surgery [33,34], toxicology [35,36], nutrition [37–39], complementary and alternative medicine [40], cancer [20,21,41,42], hearing loss [43], and mental health or the brain [44,45].

Two distinctive studies describe leveraging Wikipedia’s content in order to improve patient care. Martin-Carreras & Kahn (2019) describe an online patient-oriented information resource in radiology that is now integrated with Wikipedia to pull radiology images from the Wikimedia Commons into their database [46]. Another study describes how it used Wikipedia to mine consumer health vocabulary by finding synonyms for medical terminology in order to improve communication between practitioners and patients [47]. While the content is not necessarily evaluated for consumption, both of these studies are examples of how Wikipedia’s health content could be used in creative ways to complement existing resources maintained by experts and improve access to information or patient care.

For students

Less than 1% (n = 8) focused on the suitability of quality of Wikipedia for student use (S3 Appendix, Table 3). Herbert, et al (2015) present evidence that suggests most medical students use Wikipedia at a moderate or high rate (67%), but this investigation reports a response rate of 21% so the findings cannot be generalized [48]. Interestingly this study also finds that of the participants, 65% did not know how to revise Wikipedia when they encountered an error. This begs the question of whether faculty should follow the lead of Queen’s University [49] and USCF [50] and instruct medical students to edit Wikipedia so that they can revise errors when they inevitably use the resource. In a more methodologically sound study, Judd and Kennedy (2011) find that medical students used Google in 69% of biomedical sessions in a computer lab and Wikipedia in 51% of those same sessions [51]. While the study also notes an interesting trajectory in which students’ reliance on Wikipedia decreases each year from first year to third year, actual Wikipedia usage remains prominent throughout students’ progression through the curriculum.

Assessments of Wikipedia’s suitability as a learning tool [52–59] range in topic and methods and again, encounter similar challenges to the quality assessment studies discussed earlier. Information sources used as a point of comparison and methodologies are not necessarily appropriate, Wikipedia’s dynamic nature reduces the reliability of the studies, and the findings vary from one study to another. Some conclude Wikipedia is suitable for medical students [53,57], but most conclude it is not [53–56,58,59].

For professionals

A small set of studies (n = 5) framed evaluations of Wikipedia’s health content within the context of its suitability for use by health care workers (S3 Appendix, Table 4). Park, Masupe, Joseph, et al (2016) report that Botswanan health care workers’ perceptions of Wikipedia’s quality is divisive at best. In this study, Wikipedia’s value was based more on its consistent availability, through the now defunct Wikipedia Zero project [60], than the quality of its medical content [61].

Within specific medical contexts, the findings are inconsistent. For example, as a surgical reference Wikipedia is found to be accurate and appropriate resource, albeit incomplete [33]. Conversely, the drug information on Wikipedia is deemed variable in comparison to Micromedex and therefore considered an inappropriate drug reference for professionals [31]. Within the professional context, the literature also discusses efforts to improve the quality of Wikipedia. For example, Morata and Lum (2018) describe a partnership between the U.S. National Institute for Occupational Safety and Health (NIOSH) and Wikipedia with the goal to improve Wikipedia’s occupational safety and health content [62]. NIOSH developed and continues to manage WikiProject Occupational Safety and Health and NIOSH also participations in a program that encourages students to contribute to Wikipedia. This partnership could be interpreted as recognition of Wikipedia’s influence and broad audience.

Wikipedia’s prevalence in scholarly communication has also caught the attention of researchers. In a unique departure from assessing Wikipedia’s citations, Bould, et al. (2014) examine the prevalence of Wikipedia citations in peer-reviewed health sciences journals indexed in PubMed, Medline, or Embase [63]. The study finds a small but growing frequency of Wikipedia citations in peer-reviewed health sciences literature and that most of these citations occur after 2010. The majority of these citations are used for descriptions or definitions. This growth in referencing Wikipedia in peer-reviewed literature could be an indication of the growing reliability of Wikipedia’s health content.

General assessments of quality

General assessments of the quality of Wikipedia’s health information, those that do not take into consideration the impact of its quality on a specific population, comprise approximately 18% (n = 16) of the articles included in this review (S3 Appendix, Table 5). Despite claims that English Wikipedia’s health content is written in plain, accessible language, assessments of its quality, regardless of context, share a common finding: while Wikipedia does well to remain current [26,41,44], its medical content uses technical terms that result in readability levels too low to accommodate the very people it stands to benefit most. The readability of the easiest articles has been reported to be around ninth grade [25,30,40]. Some studies find that Wikipedia’s medical content requires at least a college reading level or is considered highly difficult [34,43]. Other studies simply report Wikipedia’s readability as higher than that of other health information sources such as WebMD, MedlinePlus, or MayoClinic [44,64].

Utility in education

Wikipedia’s utility in education is also considered within the context of different populations, ranging from its general use in educational settings to its deliberate inclusion in course or program curricula, specifically as an educational tool for evidence-based practice or information literacy, or both. The term “education” here applies to students enrolled in a formal medical or health education program, professionals enrolled in a continuing education program, workshop, or course, and professionals who use Wikipedia to inform their everyday practice. Approximately 18% (n = 16) of the literature in this review addresses the utility of Wikipedia in health education within the above three contexts.

Explorations into the various roles of Wikipedia in health and medical education have a strong presence within the literature (S4 Appendix, Table 6), which includes editing Wikipedia as an educational tool in the classroom [49,50,52,65–68], measuring student interactions with Wikipedia [48,51,53,69], and its utility in knowledge sharing between professionals or for daily reference in everyday practice [52,70–73].

The first documented for-credit course at a medical school for editing Wikipedia was offered by University of California, San Francisco in 2013 [50]. The first reported use of Wikipedia in a medical course, as a single assignment, is from 2011 and it required students to edit neuroscience stub articles [66]. Since then, the inclusion of editing Wikipedia in health education courses has crossed a number of health disciplines. Pharmacy students have been asked to write content in Wikipedia as an alternative to composing pharmaceutical drug monographs [65] and a gerontology instructor identified gaps in Wikipedia’s content on aging and assigned students the task of using their term papers to edit it [67].

A more recent example comes from Queen’s University where editing Wikipedia is integrated into the first-year evidence-based medicine curriculum [49]. It is a strong example of the utility of Wikipedia to enforce critical skills in medicine, where students are instructed in evidence-based medicine and are subsequently asked to apply those skills in a project that has real-world impact with relatively low risk. Finally, in a survey of first year students from the same program [69], the authors find that students recognize the utility of Wikipedia after editing it and developed a greater appreciation for it after completing their evidence-based medicine projects. Interestingly, the majority of survey participants also indicated that they would be likely to use Wikipedia during medical school and their subsequent residencies but would not necessarily recommend it to patients.

One distinguishing study is a randomized control trial from 2017 that compares Wikipedia to UpToDate and a digital textbook. It tests for short-term knowledge acquisition among a sample of first and second-year medical students from Canadian medical schools [74]. This two-year parallel arm RCT finds a statistically significant difference in knowledge acquisition between the textbook group and the Wikipedia group and an observed trend of better performance from the Wikipedia group compared to the UpToDate group, albeit the difference between the two groups is not found to be statistically significant. This study is unique because it employs traditional scientific rigour, a rarity here. The authors hypothesize that Wikipedia’s content may impose a smaller cognitive load than the digital textbook, which could have been a contributor to a significantly better performance on post-tests in the Wikipedia group than in the digital textbook group. It presents a strong case against the traditional discouragement of Wikipedia use in medical education and instead presents evidence to support its potential utility in the education of medical students.

With respect to adult education, continuing education, or professional practice (S4 Appendix, Table 7), Lieberthal and Leon (2015) report on having adult health economics students post their written assignments to Wikipedia [52]. Additionally, genomics experts use and contribute to the GeneWiki, a collection of Wikipedia entries where each entry is related to a specific human gene, which aims to comprehensively annotate gene functions for each known human gene [70]. The GeneWiki has been integrated into the larger body of Wikipedia articles [70] and is presented as a solution to the previous problem biomedical researchers faced in producing review articles for each of the 20,000 human genes [71]. Further, the GeneWiki itself has been mined for its gene annotations in an effort to generate a gene ontology and a disease ontology [72]. Finally, a study from New Zealand, while citing a low participation rate, suggests that Wikipedia use by interns, residents, and consultants is common to aid with memory recall or to provide a fast overview [73].

Utility in research

Wikipedia’s role in health or medical research is also discussed within the literature within two different contexts: in human information practices research, as a data source for bibliometric and epidemiological research, and in epidemiology research. It’s utility in epidemiological research constitutes half (n = 8) of the 18% (n = 16) of articles that focus on Wikipedia’s utility in health-related research.

Bibliometric and information-seeking behaviour studies

As with assessments of quality and suitability for medical students or patients, the small contingent of infodemiology research (S5 Appendix, Table 8) is also heavily reliant upon subject specific Wikipedia content. These include movement disorders [75], neurological disorders [76,77], hemophilia [78], and critical care [73]. While this is not a robust field of literature some findings are worth noting. Namely, that pageviews in Wikipedia spiked after a celebrity’s announcement of diagnosis, which suggests Wikipedia might play an important role in the public consumption of health information [77]. A single study of Wikipedia’s dominance in search engine results suggests that while specialist web sites may have more depth, they may be at higher risk than Wikipedia of being missed in more general searches due to Wikipedia’s prominence in search engine retrievals [78].

Bibliometric studies (S5 Appendix, Table 9) have been used to assess whether Wikipedia is cited in academic literature and its influence on the popularity of, or access to, high-impact journals, or to measure article impact. Wikipedia is positioned as a key alternative resource for the measurement of journal or article impact [79], to identify influential medical journals [80], or to act as a “gateway” to traditionally published health research, given that approximately 69% of the articles cited in Wikipedia’s health content are reviews, a highly valued source of evidence [81,82]. These studies also assert that a major value of Wikipedia is its curated list of published articles, many of which are cited in Wikipedia within days, but at most months, of publication [79,80].

Epidemiological studies

Finally, a little under half of the articles that address Wikipedia’s utility within the context of research focused on Wikipedia’s potential as a data source in epidemiological research (S5 Appendix, Table 10). Generous, et al (2014) propose that Wikipedia has the ability to respond to the major challenges that currently face disease monitoring–openness, breadth, transferability, and forecasting [83]. There is evidence to support the feasibility of Wikipedia in disease monitoring in relation to seasonal changes in mood [84] and seasonal fluctuations in influenza[85–87]. However, while the work on the development of a plausible means to use Wikipedia’s access log data to monitor and forecast disease continues [88], one study actually proposes Google data could be more useful [89] and another study struggles to prove the utility of Wikipedia for this purpose [90], although this is likely due to the fact that the researchers used a sample of 1,633 diseases rather than focus on a single disease as other studies have done.

Publication trends

In 2018, a study based on a sample of 100,000 articles indexed in CrossRef, a database that indexes all articles that have a DOI, estimated that approximately 28% (95% CI [27.6–28.2%]) of all literature is published using an open access publication model (OA) [91]. Comparatively, a quantitative analysis of the articles and conference proceedings included in this review (Fig 3) indicates that 40.4% (n = 36) are OA. The complement of OA literature within this review is 143% greater than the complement of OA literature indexed within CrossRef, which may indicate that the authors of these studies, like Wikipedia, value openness and the broad dissemination of information. It is encouraging to see evidence of similar values between those investigating Wikipedia and Wikipedia itself.

Another observable pattern is the overall upward trend in the production of health literature about Wikipedia, which is illustrated by a noticeable peak in 2014 (Fig 4), the same year the only two known systematic reviews of Wikipedia scholarship were published [10,11]. The growth of publications in this field is a promising indicator of the growth in the acceptance of Wikipedia within the academic health community.

Discussion

While there are consistent findings with respect to the readability and currency of Wikipedia’s health content, the results of assessments of reliability (accuracy, comprehensiveness, currency, and breadth or depth of coverage) are varied at best. These studies vary not only in their results, but also in their methodologies. Studies with dubious methodological approaches assess Wikipedia in comparison to a resource with which significant differences should be expected, such as clinical tools designed for practitioners or experts [27,30,35,36], or websites managed by leading organizations on the topic [41]. More appropriate comparisons are drawn between other open, free tertiary resources [26]. Alternatively, a validated instrument, such as DISCERN, can be used to quantitatively assess Wikipedia articles [33,34,43,92]. Other measures of quality take into consideration the number and quality of references and since there are close to one million references supporting Wikipedia’s medical content, this could be a useful approach for the assessment of reliability.

Sample selection methods in studies concerning quality range between using terms derived from course resources [32], semi-structured interviews [43], other tertiary sources [38,39], or national statistics [30] to establish a list of Wikipedia articles to evaluate. One study outright acknowledges that their sample of articles may not be representative of the field in question [93] while another neglects to acknowledge a severe mismatch between the sample studied and the results of the study [94]. In addition, some studies are just too ambitious to possibly be reliable [64,94,95] and of the 47 articles concerned about quality, only four discuss or propose methods for improving or adding to Wikipedia’s health and medicine content [62,96–98] in response to their findings. Regardless of methodology or focus, the findings of investigations into Wikipedia’s quality are varied with respect to the accuracy, coverage, comprehensiveness of Wikipedia’s medical content.

Attempts to generalize findings from a single study to generate an evaluation of the quality of the entire corpus of medical content in Wikipedia are futile at best. It is impossible to generalize that its medical content, generally, is of either good or poor quality. Wikipedia’s articles are individual pieces of a larger whole, creating a mosaic of information where each piece contributes to the summarization of medical knowledge but where some pieces are more complete than others. While it is prudent to describe Wikipedia as a tertiary information resource, it is not equivalent to what is considered a traditional encyclopedia. That is, its content is not clandestinely produced by an elite group of experts and only made available once it is deemed a complete publishable encyclopedia. Wikipedia is fluid. Not only is it incomplete, it is dynamic: evolving, and expanding; thus, rendering the practice of evaluating Wikipedia’s content, in any context, extraneous. Any evaluation will remain a snapshot in time that may or may not be applicable by the time the study hits the press. Unless these studies are used to establish a plan to improve Wikipedia’s content, it is a challenge to understand their value. Studies that measure the quality of Wikipedia’s health information seem to be asking the wrong question. A more appropriate question might be “how can Wikipedia’s health information be improved?” Further, recommendations for or against the use of Wikipedia in any context based on quality do not take Wikipedia’s utility in education or research under consideration.

Overall the literature tells us that Wikipedia is used in educational and professional environments, it is an engaging and impactful way to teach evidence-based medicine skills or information literacy, that its utility is demonstrated by the knowledge acquisition of students who use it, and that it is difficult to determine whether its quality makes it suitable for student use, despite students being aware of its utility and accessing it frequently. The value of student engagement with Wikipedia also trends in this small body of literature. Students have the time, resources, and capacity to identify gaps in Wikipedia’s health content and make the contributions required to improve it. It appears that this approach in health and medical education continues to gather momentum. An interesting contradiction in the literature however, is that some studies find Wikipedia is appropriate for public consumption, while others deem it unsuitable for student consumption This raises the question: what makes Wikipedia suitable for patient use, but not medical student use? If the Wikipedia community aims to create a Wikipedia that is suitable for both medical students and patients, what problems may arise?

Finally, there is promise that Wikipedia has potential to benefit the health research community in various contexts (bibliometrics, infodemiology, and epidemiology or disease monitoring), but more needs to be done to develop a better understanding of its utility in this regard. Wikipedia’s health content could, potentially, be leveraged to understand the public perception of a specific health topic. An evaluation of online information about genome editing finds that Wikipedia and the top seven web sites retrieved by Google focus primarily on explaining the technicalities of genome editing and do not necessarily focus on other hotly debated conversations around the topic, such as ethics [9]. These findings can be useful in determining what pieces of health information might be missing from public view.

Limitations

The most significant limitation of this review is its exclusion of literature in which Wikipedia is mentioned in articles that focus on online health information seeking but do not explicitly discuss or address Wikipedia as a major topic. These studies may provide richer insight into the role of Wikipedia in online health information seeking, its prevalence in search results, why it may be selected as a source of health information in online searching, and if, or how, Wikipedia is utilized in conjunction with other online information sources. These are explorations that merit further investigation into existing literature in the future and will be the subject of forthcoming research.

Conclusion

The literature provides strong evidence to position Wikipedia as a prominent health information resource for the public, patients, students, and practitioners seeking health information online. Wikipedia’s health content is accessed frequently, as much as or more than other free online health information sources, such as Medline Plus, and its pages regularly rank highly in search engine results, thus increasing the possibility that a user may select Wikipedia as a health information source. Trends in education where Wikipedia is employed as a tool for education in the classroom, particularly in medical curricula, have emerged but there is no obvious consensus in this domain. Outlier topics such as the utility of Wikipedia in health research or the study of disease outbreaks require more research to produce sufficient evidence to support the value of Wikipedia in research.

Much of the attention on Wikipedia is based on assessments of the quality of its medical content. However, the dynamic nature of Wikipedia and the sheer volume of content that would need to be assessed in order to properly evaluate its medical content holistically often renders these types of evaluations irrelevant. Investigations into accuracy and completeness are particularly difficult to assign value to, but a consistent finding brings to light the low readability of much of Wikipedia’s medical content. Rather than rate the quality of Wikipedia within a specific domain and recommend its use or discontinuance of use as an information resource, these findings could instead be utilized as evidence to support advocacy for expert participation in the production of Wikipedia’s content. Shafee et al (2017) believe that “Wikipedia is set to retain its position as a key public health information resource” [14] and they issue a call for participation to the medical community, as do their peers [13,25,33], “to ensure that medical content is accurate in the world’s most consulted encyclopedia” [14].

While Wikipedia itself is well into its second decade, the academic study of Wikipedia within the context of health is still young, however its use in various contexts is apparent in the existing literature. It can be a patient or consumer health information resource, a resource for medical or health students, a resource for health and medical professionals, and a data source for researchers.

The literature does not investigate to what capacity Wikipedia is used. While page view data provides evidence that Wikipedia is accessed frequently, this data alone cannot provide insight into what usage looks like. Tom Wilson, a prominent contributor to the field of information behaviour and information needs research is credited for the general notion that “the acquiring of information is not… an end in itself. An identification of the use to which [the information resource] is to be put… is essential to any form of information seeking” [99]. That is, it is not sufficient to know that Wikipedia is being used for health information. We must also seek to learn to what end it is used. Questions that remain to be answered are: Does the information on Wikipedia influence users’ decisions about their own health or health care, or the health care of those within their care, such as children or aging parents? Does Wikipedia’s health content, when consumed by patients or the public, supplement, support, or interfere with health care workers’ delivery of care to patients? Why is Wikipedia used as a health information resource compared to other free online health information by either patients or health workers? Is it used in conjunction with other online or offline health information resources? What makes a user trust or distrust Wikipedia? What can Wikipedia offer as a health information resource that traditionally accepted online health information resources, such as Medline Plus, cannot? Can the currency of Wikipedia complement the quality of more authoritative health information resources online, and if so, what might this look like?

The next step to contribute to this field that is ripe for deeper investigation will include a comprehensive scoping literature review of online health information seeking behaviour and the role of Wikipedia in the retrieval and use of online health information among patients and health care workers and any differences therein. This review will ask “what is the role of Wikipedia in online health information seeking?” Following the review, an exploratory study into the application of online medical information, with a focus on Wikipedia, will be initiated.

Supporting information

S1 Appendix. Search strategy and results.

(DOCX)

Click here for additional data file.^{(14.3KB, docx)}

S2 Appendix. General articles that aim to situate Wikipedia as a health information resource.

(DOCX)

Click here for additional data file.^{(13.8KB, docx)}

S3 Appendix. Studies of Wikipedia content quality categorized by population of concern.

(DOCX)

Click here for additional data file.^{(21.6KB, docx)}

S4 Appendix. Studies of Wikipedia’s utility in health education, categorized by education type.

(DOCX)

Click here for additional data file.^{(15.8KB, docx)}

S5 Appendix. Studies of Wikipedia’s utility in health-related research, categorized by field.

(DOCX)

Click here for additional data file.^{(19.5KB, docx)}

Abbreviations

JMIR: Journal of Medical Internet Research
SR: Systematic review(s)
OA: Open Access

Data Availability

All relevant data are within the manuscript in Appendices B through E, inclusive.

Funding Statement

The Wikimedia Foundation (awarded to DS) https://meta.wikimedia.org/wiki/Grants:Project/Rapid/Mcbrarian/Situating_Wikipedia_as_a_health_information_resource_in_various_contexts:_A_systematic_review_of_the_literature This grant was awarded to DS after the work was complete and the manuscript had been submitted. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

References

1.Wiki Project Med—Meta [Internet]. [cited 2019 Sep 25]. Available from: https://meta.wikimedia.org/wiki/Wiki_Project_Med
2.Heilman JM, West AG. Wikipedia and Medicine: Quantifying Readership, Editors, and the Significance of Natural Language. J Med Internet Res. 2015;17(3):e62 10.2196/jmir.4069 [DOI] [PMC free article] [PubMed] [Google Scholar]
3.Hodson R. Wikipedians reach out to academics. Nat News [Internet]. [cited 2019 Jul 29]; Available from: http://www.nature.com/news/wikipedians-reach-out-to-academics-1.18313 [Google Scholar]
4.Giles J. Internet encyclopaedias go head to head [Internet]. Nature. 2005. [cited 2019 Jul 29]. Available from: https://www.nature.com/articles/438900a [DOI] [PubMed] [Google Scholar]
5.Faric N, Potts HWW. Motivations for contributing to health-related articles on Wikipedia: an interview study. J Med Internet Res. 2014;16(12):e260 10.2196/jmir.3569 [DOI] [PMC free article] [PubMed] [Google Scholar]
6.Alexa—Top Sites in Canada—Alexa [Internet]. [cited 2019 Jul 29]. Available from: https://www.alexa.com/topsites/countries/CA
7.Alexa—Top Sites in United States—Alexa [Internet]. [cited 2020 Jan 8]. Available from: https://www.alexa.com/topsites/countries/US
8.Wikimedia Statistics—English Wikipedia—Unique devices [Internet]. [cited 2019 Jul 29]. Available from: https://stats.wikimedia.org/v2/#/en.wikipedia.org/reading/unique-devices/normal|line|1-year|~total|monthly
9.Calabrese C, Anderton BN, Barnett GA. Online Representations of “Genome Editing” Uncover Opportunities for Encouraging Engagement: A Semantic Network Analysis. Sci Commun. 2019. April;41(2):222–42. [Google Scholar]
10.Okoli C., Mehdi M, Mesgari M., Nielson F.A., Lanamäki A. Wikipedia in the eyes of its beholders: A systematic review of scholarly research on Wikipedia readers and readership. J Assoc Inf Sci Technol. 2014. December;(12):2381. [Google Scholar]
11.Mesgari M, Okoli C, Mehdi M, Nielsen FÅ, Lanamäki A. “The sum of all human knowledge”: A systematic review of scholarly research on the content of Wikipedia. J Assoc Inf Sci Technol. 2015. February;66(2):219–45. [Google Scholar]
12.Scoping studies: towards a methodological framework: International Journal of Social Research Methodology: Vol 8, No 1 [Internet]. [cited 2020 Jan 6]. Available from: https://www.tandfonline.com/doi/abs/10.1080/1364557032000119616
13.Heilman JM, Kemmann E, Bonert M, Chatterjee A, Ragar B, Beards GM, et al. Wikipedia: a key tool for global public health promotion. J Med Internet Res. 2011;13(1):e14 10.2196/jmir.1589 [DOI] [PMC free article] [PubMed] [Google Scholar]
14.Shafee T, Masukume G, Kipersztok L, Das D, Haggstrom M, Heilman J. Evolution of Wikipedia’s medical content: past, present and future. J Epidemiol Community Health. 2017;71(11):1122–9. 10.1136/jech-2016-208601 [DOI] [PMC free article] [PubMed] [Google Scholar]
15.James R. WikiProject Medicine: Creating Credibility in Consumer Health. J Hosp Librariansh. 2016. October;16(4):344–51. [Google Scholar]
16.Law MR, Mintzes B, Morgan SG. The Sources and Popularity of Online Drug Information: An Analysis of Top Search Engine Results and Web Page Views. Ann Pharmacother. 2011. March;45(3):350–6. 10.1345/aph.1P572 [DOI] [PubMed] [Google Scholar]
17.Laurent MR, Vickers TJ. Seeking Health Information Online: Does Wikipedia Matter? J Am Med Inform Assoc. 2009. August;16(4):471–9. 10.1197/jamia.M3059 [DOI] [PMC free article] [PubMed] [Google Scholar]
18.Czarnecka-Kujawa K, Abdalian R, Grover SC. The quality of open access and open source internet material in gastroenterology: Is wikipedia appropriate for knowledge transfer to patients? GASTROENTEROLOGY. 2008. April;134(4, 1):A325–6. [Google Scholar]
19.Thomas GR, Eng L, de Wolff JF, Grover SC. An Evaluation of Wikipedia as a Resource for Patient Education in Nephrology. Semin Dial. 2013. April;26(2):159–63. 10.1111/sdi.12059 [DOI] [PubMed] [Google Scholar]
20.Rajagopalan MS, Khanna VK, Leiter Y, Stott M, Showalter TN, Dicker AP, et al. Patient-oriented cancer information on the internet: a comparison of wikipedia and a professionally maintained database. J Oncol Pract. 2011;7(5):319–23. 10.1200/JOP.2010.000209 [DOI] [PMC free article] [PubMed] [Google Scholar]
21.Leithner A, Maurer-Ertl W, Glehr M, Friesenbichler J, Leithner K, Windhager R. Wikipedia and osteosarcoma: a trustworthy patients’ information? J Am Med Inform Assoc. 2010. July;17(4):373–4. 10.1136/jamia.2010.004507 [DOI] [PMC free article] [PubMed] [Google Scholar]
22.Skancke KH, Svendsen K. Wikipedia and medicines: who edits medicine articles on the English Wikipedia? Int J Clin Pharm. 2017. February;39(1):254. [Google Scholar]
23.Holtz P, Fetahu B, Kimmerle J. Effects of Contributor Experience on the Quality of Health-Related Wikipedia Articles. J Med Internet Res. 2018. May;20(5):e171 10.2196/jmir.9683 [DOI] [PMC free article] [PubMed] [Google Scholar]
24.Cozza V, Petrocchi M, Spognardi A. A Matter of Words: NLP for Quality Evaluation of Wikipedia Medical Articles In: Bozzon A and CudreMauroux P and Pautasso C, editor. WEB ENGINEERING (ICWE 2016). 2016. p. 448–56. (Lecture Notes in Computer Science; vol. 9671). [Google Scholar]
25.Watad A, Bragazzi NL, Brigo F, Sharif K, Amital H, McGonagle D, et al. Readability of Wikipedia Pages on Autoimmune Disorders: Systematic Quantitative Assessment. J Med Internet Res. 2017. July;19(7):e260 10.2196/jmir.8225 [DOI] [PMC free article] [PubMed] [Google Scholar]
26.Clauson KA, Polen HH, Boulos MNK, Dzenowagis JH. Scope, completeness, and accuracy of drug information in Wikipedia. Ann Pharmacother. 2008;42(12):1814–21. 10.1345/aph.1L474 [DOI] [PubMed] [Google Scholar]
27.Hunter JA, Lee T, Persaud N. A comparison of the content and primary literature support for online medication information provided by Lexicomp and Wikipedia. J Med Libr Assoc. 2018. July;106(3):352–60. 10.5195/jmla.2018.256 [DOI] [PMC free article] [PubMed] [Google Scholar]
28.Koppen L, Phillips J, Papageorgiou R. Analysis of reference sources used in drug-related Wikipedia articles. J Med Libr Assoc. 2015. July;103(3):140–4. 10.3163/1536-5050.103.3.007 [DOI] [PMC free article] [PubMed] [Google Scholar]
29.Lavsa S.M., Corman S.L., Culley C.M., Pummer T.L. Reliability of Wikipedia as a medication information source for pharmacy students. Curr Pharm Teach Learn. 2011;3(2):154–8. [Google Scholar]
30.Phillips J, Lam C, Palmisano L. Analysis of the accuracy and readability of herbal supplement information on Wikipedia. J Am Pharm Assoc. 2014. August;(4):406. [DOI] [PubMed] [Google Scholar]
31.Reilly T, Jackson W, Berger V, Candelario D. Accuracy and completeness of drug information in Wikipedia medication monographs. J Am Pharm Assoc. 2017. April;57(2):inform. [DOI] [PubMed] [Google Scholar]
32.Kim JY, Gudewicz TM, Dighe AS, Gilbertson JR. The pathology informatics curriculum wiki: Harnessing the power of user-generated content. J Pathol Inform. 2010;1(101528849). [DOI] [PMC free article] [PubMed] [Google Scholar]
33.Devgan L, Powe N, Blakey B, Makary M. Wiki-Surgery? Internal validity of Wikipedia as a medical and surgical reference. J Am Coll Surg. 2007. September;205(3, S):S76+.17916524 [Google Scholar]
34.Modiri O, Guha D, Alotaibi NM, Ibrahim GM, Lipsman N, Fallah A. Readability and quality of wikipedia pages on neurosurgical topics. Clin Neurol Neurosurg. 2018;166(df4, 7502039):66–70. 10.1016/j.clineuro.2018.01.021 [DOI] [PubMed] [Google Scholar]
35.Ayes KB, Bardsley CH. Wikipedia Information for Toxicologic Emergencies: How Reliable Is It? Clin Toxicol. 2010. July;48(6):635. [Google Scholar]
36.Ayes K.B., Bardsley C.H., Jantz J.M., Frederick W.A. Wikipedia information for toxicologic emergencies involving household products, plants and envenomations: How reliable is it? Clin Toxicol. 2011;49(6):609. [Google Scholar]
37.Messner M, DiStaso MW, Yan Jin, Meganck S, Sherman S, Norton S. Influencing public opinion from corn syrup to obesity: A longitudinal analysis of the references for nutritional entries on Wikipedia. First Monday. 2014. November;19(11):4–4. [Google Scholar]
38.Sanz-Valero J., Cabrera-Hernandez L., Wanden-Berghe C., Culebras-Fernandez J.M. The popularization of food and nutritional sciences: Wikipedia versus a general encyclopedia. Clin Nutr. 2013;32(SUPPL. 1):S192. [Google Scholar]
39.Sanz-Valero J., Wanden-Berghe C., Guardiola-Wanden-Berghe R. Nutrition and metabolism in wikipedia: Presence and adequacy of English and Spanish terminology. Clin Nutr Suppl. 2012;7(1):198–9. [Google Scholar]
40.Koo M. Complementary and Alternative Medicine on Wikipedia: Opportunities for Improvement. Evid Based Complement Alternat Med. 2014;105186 10.1155/2014/105186 [DOI] [PMC free article] [PubMed] [Google Scholar]
41.Biggs TC, Jayakody N, Best K, King EV. Quality of online otolaryngology health information. J Laryngol Otol. 2018. June;132(6):560–3. 10.1017/S0022215118000774 [DOI] [PubMed] [Google Scholar]
42.Qureishi A., Sharma A. Can Wikipedia replace traditional patient information leaflets? Comparing the internet to official patient information resources in thyroid cancer. Eur Surg—Acta Chir Austriaca. 2012;44(SUPPL. 247):35. [Google Scholar]
43.Simpson A, Le M, Malicka AN. The Accuracy and Readability of Wikipedia Articles on Hearing Loss. J Consum Health Internet. 2018. October;22(4):323–36. [Google Scholar]
44.Reavley NJ, Mackinnon AJ, Morgan AJ, Alvarez-Jimenez M, Hetrick SE, Killackey E, et al. Quality of information sources about mental disorders: a comparison of Wikipedia with centrally controlled web and printed sources. Psychol Med. 2012. Aug;42(8):1753–62. 10.1017/S003329171100287X [DOI] [PubMed] [Google Scholar]
45.STANKUS T, SPIEGEL SE. Wikipedia, Scholarpedia, and References to Journals in the Brain and Behavioral Sciences: A Comparison of Cited Sources and Recommended Readings in Matching Free Online Encyclopedia Entries. Sci Technol Libr. 2010. July;29(3):258–65. [Google Scholar]
46.Martin-Carreras T, Kahn CEJ. Integrating Wikipedia Articles and Images into an Information Resource for Radiology Patients. J Digit Imaging. 2019;32(3):349–53. 10.1007/s10278-018-0133-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
47.Vydiswaran V.G., Mei Q., Hanauer D.A., Zheng K. Mining consumer health vocabulary from community-generated text. AMIA Annu Symp Proc AMIA Symp AMIA Symp. 2014;2014((Vydiswaran) School of Information, University of Michigan, Ann Arbor, MI):1150–9. [PMC free article] [PubMed] [Google Scholar]
48.Herbert VG, Frings A, Rehatschek H, Richard G, Leithner A. Wikipedia—challenges and new horizons in enhancing medical education. BMC Med Educ. 2015;15(101088679):32 10.1186/s12909-015-0309-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
49.Murray H., Walker M., Maggio L., Dawson J. Wikipedia medical page editing as a platform to teach evidence-based medicine. BMJ Evid-Based Med. 2018;23(Supplement 1):A12–3. [Google Scholar]
50.Azzam A., Bresler D., Leon A., Maggio L., Whitaker E., Heilman J., et al. Why Medical Schools Should Embrace Wikipedia: Final-Year Medical Student Contributions to Wikipedia Articles for Academic Credit at One School. Acad Med J Assoc Am Med Coll. 2017;92(2):194–200. [DOI] [PMC free article] [PubMed] [Google Scholar]
51.Judd T, Kennedy G. Expediency-Based Practice? Medical Students’ Reliance on Google and Wikipedia for Biomedical Inquiries. Br J Educ Technol. 2011. March;42(2):351–60. [Google Scholar]
52.Lieberthal RD, Leon J. Engaging Health Professionals in Health Economics: A Human Capital Informed Approach for Adults Learning Online. J Econ Educ. 2015. January 2;46(1):45–55. [Google Scholar]
53.Schweitzer NJ. Wikipedia and psychology: Coverage of concepts and its use by undergraduate students. Teach Psychol. 2008. June;35(2):81–5. [Google Scholar]
54.Azer SA. Evaluation of gastroenterology and hepatology articles on Wikipedia: Are they suitable as learning resources for medical students? Eur J Gastroenterol Hepatol. 2014. February;26(2):155–63. 10.1097/MEG.0000000000000003 [DOI] [PubMed] [Google Scholar]
55.Azer SA, AlSwaidan NM, Alshwairikh LA, AlShammari JM. Accuracy and readability of cardiovascular entries on Wikipedia: are they reliable learning resources for medical students? Bmj Open. 2015;5(10):e008187 10.1136/bmjopen-2015-008187 [DOI] [PMC free article] [PubMed] [Google Scholar]
56.Antivalle M., Battellino M., Ditto M.C., Varisco V., Chevallard M., Rigamonti F., et al. Evaluation of wikipedia rheumatology articles as a learning resource for medical students. Ann Rheum Dis [Internet]. 2014;73(SUPPL. 2). Available from: http://heart.bmj.com/content/73/Suppl_2/801.3.abstract?sid=32ccb1a9-8868-4a08-a8c8-f62abf78e7c3 [Google Scholar]
57.Haigh CA. Wikipedia as an evidence source for nursing and healthcare students. Nurse Educ Today. 2011. February;31(2):135–9. 10.1016/j.nedt.2010.05.004 [DOI] [PubMed] [Google Scholar]
58.Yacob M., Jetty P., Lotfi S. The wikipedia medical student: Comparing the quality of vascular surgery topics across two commonly used educational resources. Int J Surg. 2014;12(SUPPL. 3):S73.25159547 [Google Scholar]
59.Azer SA. Is Wikipedia a Reliable Learning Resource for Medical Students? Evaluating Respiratory Topics. Adv Physiol Educ. 2015. March;39(1):5–14. 10.1152/advan.00110.2014 [DOI] [PubMed] [Google Scholar]
60.Wikipedia Zero—Wikimedia Foundation Governance Wiki [Internet]. [cited 2020 Jan 9]. Available from: https://foundation.wikimedia.org/wiki/Wikipedia_Zero
61.Park E., Masupe T., Joseph J., Ho-Foster A., Chavez A., Jammalamadugu S., et al. Information needs of Botswana health care workers and perceptions of wikipedia. Int J Med Inf. 2016;95((Park) School of Medicine, Boston University, Boston, MA, United States):8–16. [DOI] [PubMed] [Google Scholar]
62.Morata T., Lum M. Partnerships to expand occupational safety and health content in wikipedia. Occup Environ Med. 2018;75(Supplement 2):A71–2. [Google Scholar]
63.Bould MD, Hladkowicz ES, Pigford A-AE, Ufholz L-A, Postonogova T, Shin E, et al. References that anyone can edit: review of Wikipedia citations in peer reviewed health science literature. Bmj-Br Med J. 2014. March 5;348:g1585. [DOI] [PMC free article] [PubMed] [Google Scholar]
64.Nowrouzi B., Gohar B., Nowrouzi-Kia B., Garbaczewska M., Brewster K. An examination of scope, completeness, credibility, and readability of health, medical, and nutritional information on the internet: A comparative study of Wikipedia, WebMD, and the Mayo Clinic websites. Can J Diabetes. 2015;39(SUPPL. 1):S71. [Google Scholar]
65.Apollonio D.E., Broyde K., Azzam A., De Guia M., Heilman J., Brock T. Pharmacy students can improve access to quality medicines information by editing Wikipedia articles. BMC Med Educ. 2018;18(1):265 10.1186/s12909-018-1375-z [DOI] [PMC free article] [PubMed] [Google Scholar]
66.Burdo JR. Wikipedia neuroscience stub editing in an introductory undergraduate neuroscience course. J Undergrad Neurosci Educ JUNE Publ FUN Fac Undergrad Neurosci. 2012;11(1):A1–5. [PMC free article] [PubMed] [Google Scholar]
67.Infeld DL, Adams WC. Using the Internet for Gerontology Education: Assessing and Improving Wikipedia. Educ Gerontol. 2013. October;39(10):707–16. [Google Scholar]
68.Shane-Simpson C, Che E, Brooks PJ. Giving Psychology Away: Implementation of Wikipedia Editing in an Introductory Human Development Course. Psychol Learn Teach-PLAT. 2016. November;15(3, SI):268–93. [Google Scholar]
69.Walker M., Murray H., Dawson J., Maggio L. Wikipedia culture and usage: A survey of first year medical students to determine barriers and facilitators. BMJ Evid-Based Med. 2018;23(Supplement 1):A25. [Google Scholar]
70.Huss JW 3rd, Lindenbaum P, Martone M, Roberts D, Pizarro A, Valafar F, et al. The Gene Wiki: community intelligence applied to human gene annotation. Nucleic Acids Res. 2010;38(Database issue):D633–9. 10.1093/nar/gkp760 [DOI] [PMC free article] [PubMed] [Google Scholar]
71.Good B.M., Clarke E.L., Loguercio S., Su A.I. Building a biomedical semantic network in Wikipedia with Semantic Wiki Links. Database J Biol Databases Curation. 2012;2012((Good) The Scripps Research Institute, La Jolla, CA 92037, USA.). [DOI] [PMC free article] [PubMed] [Google Scholar]
72.Good BM, Howe DG, Lin SM, Kibbe WA, Su AI. Mining the Gene Wiki for functional genomic knowledge. BMC Genomics. 2011;12(100965258):603. [DOI] [PMC free article] [PubMed] [Google Scholar]
73.Rossler B., Holldack H., Schebesta K. Influence of wikipedia and other web resources on acute and critical care decisions. A web-based survey. Intensive Care Med Exp. 2015;3(Supplement 1):A867. [Google Scholar]
74.Scaffidi MA, Khan R, Wang C, Keren D, Tsui C, Garg A, et al. Comparison of the Impact of Wikipedia, UpToDate, and a Digital Textbook on Short-Term Knowledge Acquisition Among Medical Students: Randomized Controlled Trial of Three Web-Based Resources. JMIR Med Educ. 2017;(2):e20 10.2196/mededu.8188 [DOI] [PMC free article] [PubMed] [Google Scholar]
75.Brigo F, Erro R. Why do people google movement disorders? An infodemiological study of information seeking behaviors. Neurol Sci. 2016. May;37(5):781–7. 10.1007/s10072-016-2501-5 [DOI] [PubMed] [Google Scholar]
76.Brigo F., Igwe S.C., Otte W.M., Tezzon F., Lochner P. Information seeking behavior for neurological disorders: An infodemiological study of wikipedia article traffic. Beghi E., Filippini G., editors. Neuroepidemiology. 2014;43(3–4):158. [Google Scholar]
77.Brigo F, Igwe SC, Nardone R, Lochner P, Tezzon F, Otte WM. Wikipedia and neurological disorders. J Clin Neurosci. 2015. July;22(7):1170–2. 10.1016/j.jocn.2015.02.006 [DOI] [PubMed] [Google Scholar]
78.Kint M., Batty P., Peters M., Hart D. Wikipedia dominance in simulated online searching for hemophiliaspecific patient information. Haemophilia. 2014;20(SUPPL. 3):178. [Google Scholar]
79.Evans P, Krauthammer M. Exploring the use of social media to measure journal article impact. AMIA Annu Symp Proc AMIA Symp. 2011;2011(101209213):374–81. [PMC free article] [PubMed] [Google Scholar]
80.Jemielniak D., Masukume G., Wilamowski M. The Most Influential Medical Journals According to Wikipedia: Quantitative Analysis. J Med Internet Res. 2019;21(1):e11429 10.2196/11429 [DOI] [PMC free article] [PubMed] [Google Scholar]
81.Kissner M, Troia L, Bexfield A, Mahbuba R, Coover R. Evaluating the relationship between publications and Wikipedia citations for a cohort of oral anticoagulants. Curr Med Res Opin. 2019. April 11;35(2, SI):15. [Google Scholar]
82.Maggio LA, Willinsky JM, Steinberg RM, Mietchen D, Wass JL, Dong T. Wikipedia as a gateway to biomedical research: The relative distribution and use of citations in the English Wikipedia. PloS One. 2017;12(12):e0190046 10.1371/journal.pone.0190046 [DOI] [PMC free article] [PubMed] [Google Scholar]
83.Generous N, Fairchild G, Deshpande A, Del Valle SY, Priedhorsky R. Global Disease Monitoring and Forecasting with Wikipedia. Plos Comput Biol. 2014. November;10(11):e1003892 10.1371/journal.pcbi.1003892 [DOI] [PMC free article] [PubMed] [Google Scholar]
84.Dzogang F, Lansdall-Welfare T, Cristianini N. Seasonal Fluctuations in Collective Mood Revealed by Wikipedia Searches and Twitter Posts. In: Domeniconi C, Gullo F, Bonchi F, DomingoFerrer J, BaezaYates R, Zhou ZH, et al., editors. 2016 Ieee 16th International Conference on Data Mining Workshops (icdmw). 2016. p. 931–7.
85.McIver DJ, Brownstein JS. Wikipedia Usage Estimates Prevalence of Influenza-Like Illness in the United States in Near Real-Time. Plos Comput Biol. 2014. April;10(4):e1003581 10.1371/journal.pcbi.1003581 [DOI] [PMC free article] [PubMed] [Google Scholar]
86.Tausczik Y, Faasse K, Pennebaker JW, Petrie KJ. Public Anxiety and Information Seeking Following the H1N1 Outbreak: Blogs, Newspaper Articles, and Wikipedia Visits. Health Commun. 2012;27(2):179–85. 10.1080/10410236.2011.571759 [DOI] [PubMed] [Google Scholar]
87.Zimmer C., Leuba S.I., Yaesoubi R., Cohen T. Use of daily Internet search query data improves real-time projections of influenza epidemics. J R Soc Interface. 2018;15(147):20180220 10.1098/rsif.2018.0220 [DOI] [PMC free article] [PubMed] [Google Scholar]
88.Priedhorsky R, Osthus D, Daughton AR, Moran KR, Generous N, Fairchild G, et al. Measuring Global Disease with Wikipedia: Success, Failure, and a Research Agenda. CSCW Proc Conf Comput-Support Coop Work Conf Comput-Support Coop Work. 2017;2017(101604142):1812–34. [DOI] [PMC free article] [PubMed] [Google Scholar]
89.Sharpe JD, Hopkins RS, Cook RL, Striley CW. Evaluating Google, Twitter, and Wikipedia as Tools for Influenza Surveillance Using Bayesian Change Point Analysis: A Comparative Analysis. JMIR Public Health Surveill. 2016;2(2):e161 10.2196/publichealth.5901 [DOI] [PMC free article] [PubMed] [Google Scholar]
90.Qiu R, Hadzikadic M, Yu S, Yao L. Estimating disease burden using Internet data. Health Informatics J. 2018;(100883604):1460458218810743. [DOI] [PubMed] [Google Scholar]
91.Piwowar H, Priem J, Larivière V, Alperin JP, Matthias L, Norlander B, et al. The state of OA: a large-scale analysis of the prevalence and impact of Open Access articles. PeerJ [Internet]. 2018. February 13 [cited 2019 Jul 5];6. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5815332/ [DOI] [PMC free article] [PubMed] [Google Scholar]
92.Hillard J.R., Laird-Fick H.S., Chennamaneni V., Kavuturu S., Nassiri R.M. Helicobacter and wikipedia. Helicobacter. 2015;20(Supplement 1):115. [Google Scholar]
93.Altmann U. Representation of medical informatics in the wikipedia and its perspectives. Stud Health Technol Inform. 2005;116(ck1, 9214582):755–60. [PubMed] [Google Scholar]
94.Hasty RT, Garbalosa RC, Barbato VA, Valdes PJJ, Powers DW, Hernandez E, et al. Wikipedia vs peer-reviewed medical literature for information about the 10 most costly medical conditions. J Am Osteopath Assoc. 2014;114(5):368–73. 10.7556/jaoa.2014.035 [DOI] [PubMed] [Google Scholar]
95.Temple NJ, Fraser J. How Accurate Are Wikipedia Articles in Health, Nutrition, and Medicine? Can J Inf Libr Sci-Rev Can Sci Inf Bibl. 2014. March;38(1):37–52. [Google Scholar]
96.Pfundner A, Schönberg T, Horn J, Boyce RD, Samwald M. Utilizing the Wikidata System to Improve the Quality of Medical Content in Wikipedia in Diverse Languages: A Pilot Study. J Med Internet Res. 2015;17(5):e110 10.2196/jmir.4163 [DOI] [PMC free article] [PubMed] [Google Scholar]
97.Weiner S.S., Horbacewicz J., Rasberry L., Bensinger-Brody Y. Improving the Quality of Consumer Health Information on Wikipedia: Case Series. J Med Internet Res. 2019;21(3):e12450 10.2196/12450 [DOI] [PMC free article] [PubMed] [Google Scholar]
98.Zhang Y., Lin Y.-H. Writing a Wikipedia Article on Cultural Competence in Health Care. Med Ref Serv Q. 2016;35(2):175–86. 10.1080/02763869.2016.1152143 [DOI] [PubMed] [Google Scholar]
99.Bawden D. Users, user studies and human information behaviour: A three‐decade perspective on Tom Wilson’s “On user studies and information needs.” J Doc. 2006. January 1;62(6):671–9. [Google Scholar]

PLoS One. doi: 10.1371/journal.pone.0228786.r001

Decision Letter 0

Stefano Triberti

12 Dec 2019

PONE-D-19-29040

Situating Wikipedia as an online health information resource in various contexts: A scoping review

PLOS ONE

Dear Ms. Smith,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

Two Reviewers evaluated the manuscript and provided generally favorable opinions. I agree with their evaluation for what regards the content of the contribution, and that a scoping review on this topic is of notable interest. However, I would like to stress the importance of adapting the manuscript to PLoS ONE standards. Specifically, this journal does not consider literature reviews, but it considers scoping reviews given that they are well reported, address a clearly defined research question, have been conducted to a high standard, and are reproducible. Consistently with Reviewer 1 in particular I think the methodological part of this manuscript could be improved.

Revise abstract: background is missing and objective is not clear
Revise or improve/specify objectives: while the aim of a scoping review is to “map a field”, this journal requires objectives/research questions to be as specific as possible to make the review reproducible
Redo tables as suggested: Also, some references in the tables include journals, others not – keep consistent
More details and relevant references are needed regarding the exclusion of sources (lines 81-85); it seems reasonable not to include all possible health-related terms, but again make sure enough information is provided to make the review reproducible. The same can be said for the “thematic analysis” (lines 133-139) which lacks methodological details

Finally, Reviewer 2 said his comments are optional, but I suggest you to consider them, especially for what regards correctness about Wikipedia history and activities

==============================

We would appreciate receiving your revised manuscript by Jan 26 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Stefano Triberti, Ph.D.

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please upload a copy of Figure 1, to which you refer in your text on page 6. If the figure is no longer to be included as part of the submission please remove all reference to it within the text. (you appear to have included your figure 1 file as a supporting information file and not a figure)

3. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: N/A

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This scoping review studies the peer-reviewed, published health and information science literature about Wikipedia.

The authors’ described exhaustively Wikipedia, its role and its characteristics, in the introduction, which is well written and organized.

Please, correct the citation (6) at line 50 which is reported two times.

The methods are well described; however, I think the authors should explain why they decided to proceed with a scoping review and the methodology to follow for this type of review.

I think that Results are limited; the authors should better explain in which and how many contributions they did find the specific results. The table alone is insufficient, more detailed description in the text should accompany the analysis.

Additionally, the PRISMA figure is not explained in the test: authors should describe the processes that led them to exclude articles from the first exclusion process (they must start from the initial number of articles, 3630, and explain how they arrived at the final number, 89).

Tables should be completely redone. Usually tables in reviews (systematic or scoping) include each individual study (one study per row) and list their characteristics (e.g., population, research design, results relevant for review, etc.) to help the reader identifying relevant information for each study; in the current form, it is really not clear what Authors wanted to explain with such tables. What is the relationship between multiple studies on the same row, for example? Why is it important to divide the studies basing on population especially? Etc.I advise authors to analyze similar reviews from already published literature and to find an alternative for the design of the tables which could be clearer and more informative for the readers.

I think that most of the information relevant for the results were incorrectly placed in the discussion part: information on the retrieved papers should be put in results, while discussion and conclusion should feature critical resume and synthetic reflections about the main take-home messages for the reader. I suggest authors to revise these sections and be clearer in the distinction between a reporting results vs. a discussion section.

Reviewer #2: read the attached PDF

even though I attached a PDF, the form required that I also post some comments here

Excellent draft - if you changed nothing then it is fit for publication. Suggestions below are optional for you to use.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Lane Rasberry

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

Attachment

Submitted filename: review PLOS Manuscript Number PONE-D-19-29040.pdf

Click here for additional data file.^{(87.7KB, pdf)}

PLoS One. 2020 Feb 18;15(2):e0228786. doi: 10.1371/journal.pone.0228786.r002

Author response to Decision Letter 0

15 Jan 2020

Response to editor:

EDITOR: I would like to stress the importance of adapting the manuscript to PLoS ONE standards. Specifically, this journal does not consider literature reviews, but it considers scoping reviews given that they are well reported, address a clearly defined research question, have been conducted to a high standard, and are reproducible. Consistently with Reviewer 1 in particular I think the methodological part of this manuscript could be improved.

DS: I have revised the manuscript to meet the stylistic and rigor requirements of PLoS ONE to the best that I have understood them based on your feedback and reviewer’s comments. I have made significant improvements to the methods section and added my full search strategy (appendix A). I hope you will find make the study reproducible.

• EDITOR: Revise abstract: background is missing and objective is not clear

o DS: Added research questions to objective. Added background.

• EDITOR: Revise or improve/specify objectives: while the aim of a scoping review is to “map a field”, this journal requires objectives/research questions to be as specific as possible to make the review reproducible

o DS: Added research questions to objective in Abstract. RQs are also in line 70 and 71 of introduction.

• EDITOR: Redo tables as suggested: Also, some references in the tables include journals, others not – keep consistent

o DS: Addressed (see response to Reviewer #1)

• EDITOR: More details and relevant references are needed regarding the exclusion of sources (lines 81-85); it seems reasonable not to include all possible health-related terms, but again make sure enough information is provided to make the review reproducible. The same can be said for the “thematic analysis” (lines 133-139) which lacks methodological details

o DS: provided more detail in flow chart (precisely how many articles were excluded for each criterion for exclusion), fleshed out methods section to include rationale for scoping review as the chosen method and a stronger rationale for not including health terms.

o DS: Added more detail to define “thematic analysis” and describe methods of conducting the thematic analysis to improve reproducibility of study.

Finally, Reviewer 2 said his comments are optional, but I suggest you to consider them, especially for what regards correctness about Wikipedia history and activities

DS: Complete. Responses to individuals comments below (see response to Reviewer #2)

RESPONSE TO REVIEWER 1:

R1: Please, correct the citation (6) at line 50 which is reported two times.

DS: Corrected

R1: The methods are well described; however, I think the authors should explain why they decided to proceed with a scoping review and the methodology to follow for this type of review.

DS: Added search strategy for each database search. Added lines 74-81 to provide context for purpose of scoping review.

R1: I think that Results are limited; the authors should better explain in which and how many contributions they did find the specific results. The table alone is insufficient, more detailed description in the text should accompany the analysis.

DS: I combine the Results and discussion into one section (as in the body formatting guidelines) and was better able to illustrate specifically from where each category emerged. The detailed description in the text that previously sat in the Discussion section now complements the results in a section called Results and Discussion

R1: Additionally, the PRISMA figure is not explained in the test: authors should describe the processes that led them to exclude articles from the first exclusion process (they must start from the initial number of articles, 3630, and explain how they arrived at the final number, 89).

DS: I added a section to the Methods section that describes how I went from my starting number of results to the final 89 articles. NOTE: the numbers changed as I went back to revisit my records and realized I did manual screening of a single database search and this was left out of my originally submitted counts. This has now been accounted for in the text and adequately described.

R1: Tables should be completely redone. Usually tables in reviews (systematic or scoping) include each individual study (one study per row) and list their characteristics (e.g., population, research design, results relevant for review, etc.) to help the reader identifying relevant information for each study; in the current form, it is really not clear what Authors wanted to explain with such tables. What is the relationship between multiple studies on the same row, for example? Why is it important to divide the studies basing on population especially? Etc.I advise authors to analyze similar reviews from already published literature and to find an alternative for the design of the tables which could be clearer and more informative for the readers.

DS: Completely revised tables. I see now how troubling the layout was. Have also renamed title of each of the three tables to be more descriptive about the content therein. Each table is divided into categories and the relevant articles from each category are listed, one study per row. Each study in the table corresponds with a category discussed within the body of the discussion. The purpose of including these tables is to provide readers with a quick reference to consult when they’d like to see all articles that fall within a particular category/theme I identify and discuss in the body of the article. The decision to have some categories labeled by population is because in the body of my discussion I specifically address trends in the literature where some populations are considered more than others. For example, for quality of Wikipedia’s health content, the dominant conversation is around whether the quality makes it suitable for patients, but it is important to provide a list of which articles have evaluated Wikipedia’s quality with patients in mind, or another population, such as medical students or health professionals.

Since this is a scoping review I chose not to place emphasis on recording study design or other characteristics. The main goal of this review is to get a sense of what conversations are happening in the literature and to draw out any patterns/trends/gaps/or concerns I can find. I made notes when methodology was dubious and report these concerns, when appropriate, in the discussion section.

DS: As mentioned above, I moved the discussion section to the results and combined them into one section to provide richer detail in a more streamlined way. The conclusion will include reflections and the main take-home messages.

RESPONSES TO REVIEWER 2:

37 “Imagine a world in which every single person has free access to the sum of all medical 38 knowledge” (1). This is what WikiProject Medicine, a community of editors who develop 39 the health care content in Wikipedia (2),

Some clarification here - omit the link to Wordpress, your citation #1. This is not maintained and is the same as the #2 citation

https://meta.wikimedia.org/wiki/Wiki_Project_Med

DS: Fixed.

41-42

than any other health information 42 resource online

"than any other single health information resource"

I realize that we in Wikipedia do not track non-digital media because it is not possible, but Wikipedia is a continually living 20 year global resource. It competes against things like personal doctor visits.

I think you went to the Google / Facebook / Microsoft event at the WikiConference. I have less understanding of Google, but I think they said that 15% of user queries are health related. I feel increasingly comfortable calling Wikipedia most popular without qualifiers like "online", "new media", etc.

DS: I removed many of the qualifiers of “online” but not all. I have strategically placed terms like “internet” and “online” in the abstract so that the article can be retrieved in a search for “online OR interent” AND health information.

However, a paradigm45 shifting study conducted by Nature found that Wikipedia and Encyclopedia

Britannica

This was in 2005, we have high school students born after this who use Wikipedia routinely. Since 2005 there has been one human generation of development passed and several technological generations. For example, in 2005 no one thought Facebook would become popular. . There has been no big challenge of Wikipedia since then. There have been various small challenges.

You know the literature. My point in this is just that the Nature study is old and the bigger news is that no one ever challenged it.

DS: Yes, I included this study so that other readers are aware I know it exists. A common exercise in academia, I’m afraid. You’re right to point out that the bigger news is that the finding has not been challenged and so I have added a sentence to that effect. Line 47-48

59 Today Wikipedia is the 7th most frequently accessed web site in the United States and 60 Canada(7).

Since 2005 Wikipedia has been an Alexa top 10 website. There is no source to cite for this. Evidence becomes stronger after 2007 and stronger again after 2012, but no one has published this simple reporting. If you feel bold enough you could say "top ten website since at least 2014" and cite Heilman 2015 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4376174/

the strength of the claim in the source is weak but the statement is accurate

DS: I cited the Alex web site rankings: https://www.alexa.com/topsites/countries/US & https://www.alexa.com/topsites/countries/CA. There are new stats, so I updated the manuscript with the latest rankings for US and Canada

Wikicite has 90 papers, you found 89, and I wonder what the difference is.

DS: It’s possible new articles have been published since I ran my searches 7 months ago. In fact I know there have been at least one or two new articles about editing Wikipedia or evaluation quality of Wikipedia, I just coudln’t add them to this review retroactively and if I did, they wouldn’t change my findings anyhow.

98 treatment of Wikipedia's health content was insufficient to merit inclusion. These articles 6 99 will be reviewed separately in an additional study of the literature that specifically explores 100 online health information seeking behaviour and Wikipedia’s role within it.

I get anxious seeing promises like this made because it can be hard to identify the later paper if it exists. I sort of wish this promise could be omitted.

DS: You’re right. It’s a bold statement. I have removed. But for the record – I did end up doing this work. If it gets published I’ll be sure to cite this review so readers can draw the connections on their own.

110 In 2018, a study based on a sample of 100,000 articles indexed in CrossRef, a database that 111 indexes al

This bit about OA is interesting but this is the start of the results section, and I was hoping for a strong start about Wikipedia. The OA information could come later as it is more incidental.

DS: Yes, it’s jarring that I don’t launch into results about Wikipedia straight away. I have moved the OA content and have moved it to the end of the Results and discussion section.

117 openness and the broad dissemination of information. It is encouraging to see evidence of 118 similar values between those investigating Wikipedia and the organization itself.

Can you clarify what "the organization" is? Perhaps rephrase?

DS: Yes – changed “the organization” to “Wikipedia” since it is not an organization, but a project of the Wikimedia Foundation

124 literature about Wikipedia is illustrated by a noticeable peak in 2014 (Fig 3), the same year 125 the only two known systematic reviews of Wikipedia scholarship were published.

Wait - can you cite the reviews to which you are referring?

DS: Done.

125 the only two known systematic reviews of Wikipedia scholarship were published. In 2015, 126 Nature reported its findings that suggested Wikipedia was more or less equal in accuracy 127 compared to Encyclopedia Britannica

Seems like an error! Not 2015, 2005!

https://www.nature.com/articles/438900a

I am not aware of a 2015 paper!

DS: Yes this was a typo I became aware of after submitting the manuscript. Thanks for the reminder. I may have forgotten to fix it otherwise.

123 -129

perhaps, but is this a confusion about the date of the nature paper? If so then revise or strike this.

DS: fixed… a result of my typo and working on this too late at night!

134 1. Articles with a general focus on situating Wikipedia as a health information resource 135 with no specific context as a major focus (e.g for consumers, professionals, or 136 students);

It seems extraneous to qualify this when this is the only category of this sort. Why not just say "Articles with a general focus on situating Wikipedia as a health information resource", then later explain this additional detail that you sought no specific audience targeting?

DS: This is reasonable to me. I’ve adjusted. I do explain this later so you’re right to point out the extraneity of it

156 dominate the conversation. 53% (n=47) of the articles included in this study measure

Is "measure" the correct term? I would have expected "report" or "make claims upon", but I think most studies do not do a measurement in a quantitative way.

DS: Changed to “assess”

160 currency, comprehensiveness, completeness of coverage, number of references, and 161 currency.

currency twice

also I know that Wikipedia emphasizes number of references, but I am not aware of other sources strongly valuing citations in medical information like Wikipedia does.

DS: Removed duplicate word. Some studies in this review evaluate Wikipedia’s quality by assessing the number of references its pages on a specific topic have. This is why it is included in the list of characteristics that may be used to evaluate the quality of Wikipedia.

181 resource demonstrate this by references its frequency of use and some advocate for the 182 medical community to contribute to Wikipedia’s medical content. E

something unclear here - "references" to "referencing", perhaps

182 medical community to contribute to Wikipedia’s medical content. Evidence has been 183 collected from the Pew Research Centre

This confuses me a bit. Active voice could help -

DS: Yes- revised the above two statements with more clear, direct language.

218 same topics (17). This is study is seminal positioning Wikipedia’s prominence in the 219 existing complement of online health information resources.

rephrase

This study positions Wikipedia as prominent among the existing...

DS: done.

227 gastroenterology (18), nephrology (19), and some cancers (20–22). These assessments 228 assess

"assessments assess" to "studies assess" or similar

DS: “assessments” changed to “studies”

229 findings in relation to the public consumer or patient. Of those that include results – some 230 conference proceedings do not – they generally agree that Wikipedia is suitable for patients 231 (18,19,22) and a 2010 study find

Cut this into multiple sentences. The hyphen conjuctions etc is too much.

DS: done.

232 the preferred one (21). Unfortunately, these assessments of suitability for patients are no 233 longer likely to be relevant with publication dates ranging between 2008 and 2013.

The situation is mixed. Many articles require updating and some do not.

DS: Yes – some articles require updating and some do not. Howeever, these particular assessments are mainly concerned with whether or not the content is suitable for patients. The do not necessarily provide evidence to suggest that an outcome of their findings will be to update the content. Some call on the medical community to contribute, others don’t even mention the possibility of making changes to the articles they assess. Also – the studies of suitability are also unlikely to be relevant today because they were published years ago. We can’t evaluate Wikipedia articles based on studies 6+ years old.

236 (2017) investigation into who edits Wikipedia’s medicines content "medicines" to medical, as this not about drugs specifically

DS: This is the language used in the article and it is referring to pharmaceutical medications. Revised language to be more clear: “Wikipedia’s content on medicines”

248 Two distinctive studies describe leveraging Wikipedia’s content in order to improve 249 patient care and this is perhaps the most exciting finding of this review.

I do not follow why you think these are the most exciting. Can you explain more?

DS: Added: “due to their real world application to medical practice” I found these studies exciting because they provided examples of broad uses of Wikipedia for patient care beyond physicians using Wikipedia articles to jog their memory.

251 now integrated with Wikipedia to pull radiology images from the WikiMedia Commons into

It is stylized "Wikimedia Commons"

DS: Revised

252 their database (46). Another study describes how it used Wikipedia to mine consumer 253 health vocabulary by finding synonyms for medical terminology

Excavating the mother lode

https://www.sciencedirect.com/science/article/pii/S0306457316303004

This talks about medicine and the people behind this are wiki medicine people. Not sure if you detected this one as it does not have Wikipedia or medicine in the title. The paper was bold enough to identify Wikipedia as the mother lode but regrettably did not say Wikipedia in the title. They really are sitting on a big idea.

That was out of scope for you but confirms the application.

DS: I did see that that has Wikipedia in the title and this article. It was retrieved in my Web of Science search. It was excluded in the title/abstract round because it did not mention health or medicine. These categories threw me off: “information retrieval, natural language processing, and ontology building”and made it sound like a paper focused on computer science applications. This review cited a study from Friedlen and McDonald (2010) that I had found but determined irrelevant due to its IT/CompSci focus. You’re right that the review article out of scope but is a great confirmation of application of this practice.

262 the findings cannot be generalized (48). Interestingly this study also finds that of the 263 participants, 65% did not know how to revise Wikipedia when they encountered an error.

prefer "did not know that Wikipedia invites them to edit text to revise errors which they encounter"

DS: Not changed. This is a direct quote from the study: “Of those, 861 (65%) students did not know how to revise articles and 199 (15%) let the false information unaltered, despite knowing how to correct articles” Changing the sentence according to your suggestion above would not be an accurate representation of the findings in the cited paper. The study did not report on whether the participants knew that they could edit, just that they did or did not know how to do it.

277 studies, and the findings vary from one study to another. Some conclude Wikipedia is 278 suitable for medical students(53,57), but most conclude it is not(53–56,58,59).

yeah, I wish more studies would compare Wikipedia's suitability to any other specific thing instead of comparing it to the ideal of perfection.

DS: Right – the reality is there is no “gold standard” and Wikipedia cannot be compared to perfection or the “ideal information resource” which inherently does not exist. All types of information resources have their own unique strengths and flaws.

That project, Wikipedia Zero, is sunsetted. Regardless of publication and evaluation, we in the Wikipedia editing and outreach community never became aware of any cases of use which resulted in anyone contacting anyone in our social network. It was kind of an expensive outsider project which never had Wikipedia community collaboration in an apparent way.

The community effort, named as a flagship Wikimedia Foundaiton accomplishment in their 2018 annual report, is Internet in a Box. I and lots of WikiProject Medicine people contribute to this. https://meta.wikimedia.org/wiki/Internet-in-a-Box

I wish you could omit this text just because that Wikipedia Zero project is over, it never matured, it had more presence in publication than in Wikipedia community engagement, and it was kind of a project on paper which had thin ties to any reality. I do not want to criticize it because it could be revived, good people contributed to it, and is still a good idea but we do not have much to show for this.

DS: I am aware of the Internet-in-a-Box and its applications. The reference to Wikipedia Zero here is entirely relative to the specific study I’m reporting on, where the value of Wikipedia to this community of Botswanan medical workers was dependent on the fact that they could access it without internet. That despite awareness of Wikipedia’s flaws, they still saw it as valuable simply because it was consistently available. So – rather than omit this section, I have added some commentary, and citations, that draw to the light the fact that Wikipedia Zero does not exist any longer.

314 that while Wikipedia does well to remain current (27,42,44), the readability of its medical 315 content is too low to accommodate the very people it stands to benefit most. The lowest 316 reading level reported is around ninth grade (26,31,41)

clarity here please

you use the term "low" in two different senses. One sense is "low readability", which I think means "high complexity", and the other sense is "low reading level", which means "low complexity".

perhaps

that while Wikipedia does well to remain current (27,42,44), its medical content uses technical terms which result in readability which is too low accommodate the very people it stands to benefit most. The most accessible reading level reported is around ninth grade (26,31,41)

DS: Thanks. Changed this to: “ while Wikipedia does well to remain current (27,42,44), its medical content uses technical terms that result in readability levels too low to accommodate the very people it stands to benefit most. The readability of the easiest articles has been reported to be around ninth grade (26,31,41)”

325 methodological approaches assess Wikipedia in comparison to a resource with which 326 significant differences should be expected, such as subscription-based reference tools 327 designed for practitioners or experts (28,31,36,37), or websites managed by leading 328 organizations on the topic (42)

I think it is fair to compare Wikipedia with leading resources, regardless of whether they claim to have good subscription funding or target expects. The dubious methodologies include things like setting comparisons where Wikipedia does not try to be competitive, like taking points off Wikipedia for not including dose information or professional practice information. Wikipedia competes as reference information and should be compared on that basis.

DS: Yes- this is what I am getting at in this sentence. That comparisons should not be drawn between Wikipedia and a resource that is designed for something entirely different than Wikipedia. Since Wikipedia is meant to summarize medical information, we cannot expect it to be as robust as a textbook on how to perform vascular operating procedures, or as you say, dosage information. I have revised the sentence to be more clear about my intent.

328 More appropriate comparisons are drawn between other 329 open, free tertiary resources (27) or, alternatively, a validated instrument, such as 330 DISCERN, is used to quantitatively assess Wikipedia articles (34,35,43,63)

two kinds of comparisons - reference information quality is one issue, and accessibility is another issue. For quality I want Wikipedia compared against the best competing resource for any topic. Along with that comparison, there can also be a comparison of accessibility.

Wikipedia could then also be compared against the best available free resource, but I see little value in that because I expect that nothing would come close to Wikipedia's quality. Also any other available free resource is almost certainly negligible traffic and therefore inaccessible.

DS: I agree, but as this is a review I can only report on what the literature has reported and at this point, many of the published studies that compare Wikipedia to another resource seem to missing the point – that Wikipedia is meant to be an encyclopedia. It is a tertiary source. Therefore, it is inappropriate to draw comparisons between the “quality” of a an encyclopedia compared to a textbook or clinical tool, which are an entirely different type of information resource. For example, I wouldn’t go to Dynamed or UpToDate to get an overview of a medical condition for the same reasons I wouldn’t seek out dosage guidelines on Wikipedia. Each type of research has its own role to play in the landscape of medical information and the comparisons published to date don’t seem to understand that nuance.

347 Any attempt to generalize findings in order to generate a holistic understanding of the 348 quality of the entirety Wikipedia’s medical content is an insurmountable task.

specific the methodology which you imagine

I admit that we do not have a holistic understanding but also no one has tried. I think if we had modest funding to do an evaluation then we could do well to get a good general understanding. There are several possible strategies to attempt and realistically there has hardly been any attempt.

I am much more hopeful about this.

DS: What I meant here is that some studies included in this review evaluated the quality of a sub set of Wikipedia articles on a given topic (e.g. autoimmune disorders) and then generalized those findings to make claims about the quality of Wikipedia’s medical content more broadly or at least more broadly than the articles included in the study. Attempts to generalize findings to generate a holistic view of the quality of Wikipedia’s medical content is futile. The articles are changing all the time and the quality of articles is not standard across Wikipedia, Some are poor, some are remarkably well done. So it’s impossible to say whether or not Wikipedia, generally, is “good” or not. I hope the revision makes this more clear:

“Attempts to generalize findings from a single study to generate an evaluation of the quality of the entire corpus of medical content in Wikipedia content are futile at best. It is impossible to generalize that its medical content, generally, is of either good or poor quality. Wikipedia’s articles are individual pieces of a larger whole, creating a mosaic of information where each piece contributes to the summarization of medical knowledge but where some pieces are more complete than others.”

351 more complete than others. Wikipedia is also fluid. While it is prudent to describe 352 Wikipedia as a tertiary information resource, it is not equivalent to a published 353 encyclopedia. Not only is it incomplete, it is also dynamic: evolving, and expanding; thus

It has been 20 years. Wikipedia is the golden standard for what defines a published encyclopedia. Saying that it is "not equivalent to a published encyclopedia" is odd in the context that Wikipedia is known as an encyclopedia and gets more human hours of consultation than all other encyclopedias put together. At some point it becomes necessary to state what one expects of an encyclopedia and what Wikipedia is.

DS: Thanks for pointing out the need for me to clarify this. By “traditional encyclopedia” what I really meant is that while it IS an encyclopedia, it is not traditional in the sense that is not clandestinely laboured over for years by an elite group of hand-picked experts and then published when it is deemed to be complete. Have made the change to this:

“While it is prudent to describe Wikipedia as a tertiary information resource, it is not equivalent to what is considered to be a traditional encyclopedia. That is, its content is not clandestinely produced by an elite group of experts and only made available once it is deemed as complete published encyclopedia. Wikipedia is fluid. Not only is it incomplete, it is dynamic: evolving, and expanding; thus, rendering the practice of evaluating Wikipedia’s content, in any context, extraneous”

Also remember that yes, Wikipedia is 20 years old, but its prevalence in the academic sector is still young. Anecdotally, I still have yet to meet a university student who WAS NOT told by their high school teachers to avoid Wikipedia because it is not reliable. So it’s been around, yes. We (Wikipedians) know it’s the new model of encyclopedic knowledge. But academia is still catching up.

357 Wikipedia’s content, it is a challenge to understand their value. Studies that measure the 358 quality of Wikipedia’s health information seem to be asking the wrong question. Further, 359 recommendations for or against the use of Wikipedia in any context do not take into 360 consideration Wikipedia’s utility.

Be direct, what is the wrong question, and what is the right question

DS: “Studies that measure the quality of Wikipedia’s health information seem to be asking the wrong question. A more appropriate question would be “how can Wikipedia’s health information be improved?” – Added.

368 The first known for-credit course at a medical school for editing Wikipedia was offered by 369 University of California, San Francisco in 2013 (50),

"The first documented record of a for credit course at a medical school"

I led a Wikipedia editing series at the University of Washington medical school in 2011. When I did that there were a few others doing similar things. When that 2013 program began I trained Amin Azzam based on the culture of practice for medical editing in schools which already existed for this.

I guess that I am realizing that I need to write up some history because many people at many institutions contributed greatly, and I want the origin story to be Wikipedia-style normal community collaboration and not one person's new idea.

DS: Changed langage to “documented.” You’re right. There’s also been a long history of editing in undergraduate education, but not necessarily as a full course. It would be neat to create a wiki timeline of Wikipedia editing in medical education that all educators who have used Wikipedia could contribute to.

369 University of California, San Francisco in 2013 (50), but the first reported use of Wikipedia 370 in a course, as a single assignment, is from 2011 and it required students to edit 371 neuroscience stub articles (71)

You might mean "medical course". There are probably earlier records of medical school course editing on Wikipedia, but not in academic publishing. Likewise there are earlier records of class editing. United States Public Policy in 2010 was the first big classroom initiative. https://en.wikipedia.org/wiki/Wikipedia:WikiProject_United_States_Public_Policy

even this followed earlier classroom experiments.

DS: changed course to “medical course”

372 education courses has crossed a number of health disciplines. Pharmacy students have 373 been asked to write content in Wikipedia as an alternative to composing pharmaceutical 374 drug monographs (70) and a gerontology instructor identified gaps in Wikipedia’s content 375 on aging and assigned students the task of using their term papers to edit it (72).

Here is my own study about classroom use. If you are looking for examples of classroom engagement then this program is in its 5th year and of course this paper is Wikipedia-values oriented.

Improving the Quality of Consumer Health Information on Wikipedia: Case Series

https://www.jmir.org/2019/3/e12450/

DS: Thanks – this article was selected for inclusion in the review. I cited it after this sentence 318-319: “only four discuss or propose methods for improving or adding to Wikipedia’s health and medicine content(62,68–70)”

394 to UpToDate and a digital textbook

check your consistent stylization UpToDate

DS: used Find and Replace function to fix this. Thanks.

441 There is a notable trend in the assertion that one major value of 442 Wikipedia is its curated list of published articles, m

not sure what this means, clari fy please

DS: Changed to “These studies also assert that a major value of Wikipedia is its curated list of published articles, many of which are cited in Wikipedia within days, but at most months, of publication(81,82).”

Hope this helps

458 context of research focused on Wikipedia’s potential role as a data source in 459 epidemiological research. Generous, e

you mentioned the Ebola study above. Maybe that is better placed here? The context of that was that the WHO did not have expertise to translate Ebola text into languages of Africa, but Wikipedia did have community of translators, and a platform to accept this language content, and the means of distribution for this information. If you want an epidemiological example then that ebola incident is the one the media picked up.

DS: I prefer to leave the reference to ebola where it is. It is actually a citation from Heilman’s “evolution” article, and not a study on ebola and Wikipedia. The context of the Ebola example was to demonstrate the impact that Wikipedia Zero had during the Ebola outbreak. So, I mentioned Ebola as an interesting example of Wikipedia’s global impact.

470 As promising as the research into the potential utility of Wikipedia in epidemiology may be, 471 one major limitation of using Wikipedia for outbreak prediction is often touted as one of 472 Wikipedia’s strengths: a

This is a misunderstanding. Google will not talk to university researchers. Google is inhuman and like aliens far away. Wikipedia is a human project and of course we have the data to emulate Zeiger's Google Flu Trends project. Typical researchers can also ask for some private IP address data. Wikipedia does not share this data lightly, but whereas Google almost certainly would not share data with less than a ~300 million investment in this, the buy in for access to Wikipedia's data is ~1 million and commitment to act like a human while having conversations in public with the wiki community. Of course we want to do this project, and the barrier is not on the Wikipedia side of this.

DS: Thanks for pointing this out. I was under the incorrect impression that Wikipedia does not collect data about the locations of its users, period. And that data related to pageview statistics is completely anonymized. I appreciate that you’ve pointed out that data can be made available for research at the discretion of Wikipedia (for a price). This is something I’d like to learn more about. I have removed the following:

“As promising as the research into the potential utility of Wikipedia in epidemiology may be, one major limitation of using Wikipedia for outbreak prediction is often touted as one of Wikipedia’s strengths: all global searches of Wikipedia are aggregated so searches that originate in a specific geographical location cannot be parsed from the data. Unless we can isolate pageviews geographically it will be difficult to determine a geographic location for disease outbreak.”

And I have replaced it with:

“Overall, there is promise in Wikipedia for epidemiological data, but more needs to be done to develop a better understanding of its utility in this field.”

489 than other free online health information sources, such as Medline Plus, and its pages 490 regularly rank highly in Google search results,

you could say Bing and DuckDuckGo, or you could say search engines. Google is not quite universal. Russia and Korea use other search engines, and lots of other countries do too. China is moving onto the world stage and probably will compete against Google in the developing world with their search engines eventually. If you generalize this paper has some more life and citability in the future.

DS: Good point. Changed.

512 within the context of health is still in the early stages of its development. Several questions 513 have not yet been asked within the literature that are instrumental in understanding 514 Wikipedia’s role as a health information resource and various contexts: as a patient or 515 consumer health information resource, as a resource for medical or health students, as a 516 resource for professional in health and medicine, and as a data resource for researchers.

can you rephrase? this is long and blocky and I cannot readily see what questions you want asked and answered.

DS: Completely rephrased this to:

“While Wikipedia itself is well into its second decade, the academic study of Wikipedia within the context of health is still in young. However, its role in various contexts is apparent in the existing literature. It can be a patient or consumer health information resource, a resource for medical or health students, a resource for health and medical professionals, and a data source for researchers”

520 explaining the technicalities of genome editing and do not necessarily focus on other hotly 521 debated conversations around the topic, such as ethics (9).

Obviously the issue is that the genetics people import structured data at scale, which is relatively easy, whereas for ethics we require prose which is more difficult.

DS: Yes. Since I am reviewing the literature I don’t feel comfortable making that kind of claim. I don’t know enough about the project to do that.

525 data provides evidence that Wikipedia is accessed frequently, this data is insufficient to 526 understand whether its content is actually used by the consumer.

I know why you say this because everyone says this but it is totally bunk. No one holds other communication channels to this standard. Everyone assumes that doctor's office pamphlets and billboards and online patient guides and apps andWe have no reasonable evidence of how consumers use other sources. Suddenly when Wikipedia appears among other communication resources somehow people question whether users get impact out of Wikipedia while taking for granted that they get impact from other communication channels. It is not reasonable and it indicates a bias against Wikipedia.

You are not citing sources on this so if this is your own musing, I would prefer that this be phrased like this.

There is evidence that Wikipedia is more popular than other channels, but some critics dismiss this by saying that the value of measurable user attention to Wikipedia is less than the value of user attention to other sources. It would be useful to have a system to assign impact value consistently to Wikipedia and other outreach channels so that their usefulness could be compared.

DS: You are correct that this is my own musing and I will rephrase so that it is more palatable. This is basically what I will be doing my dissertation research on because in a field as evidence-based as health and medicine, we don’t have qualitative evidence to describe how Wikipedia is used, where it fits in the learning process of patients, etc. I have rephrased to: “While page view data provides evidence that Wikipedia is accessed frequently, this data alone cannot provide insight into what usage looks like.”

Figure 2

Can you label this more explicity? Perhaps you do this in the text of the article. Just looking at the image I became lost understanding the vertical access labels.

DS: I have revised all labels in the article to be more explicit.

Figure 4

You might have already described this but I missed the place. Was the classification obvious or did you use a non-obvious sorting system?

DS: I have revised the methodology section, which now indicates that I categorized articles inductively

Attachment

Submitted filename: Response to Reviewers.docx

Click here for additional data file.^{(50.5KB, docx)}

PLoS One. doi: 10.1371/journal.pone.0228786.r003

Decision Letter 1

Stefano Triberti

21 Jan 2020

PONE-D-19-29040R1

Situating Wikipedia as a health information resource in various contexts: A scoping review

PLOS ONE

Dear Ms. Smith,

==============================

The Author properly responded to Reviewers' concerns. Minor modifications are still needed:

- abstract has been revised well for what regards content and structure, but now it exceeds the 300 words lenght allowed: https://journals.plos.org/plosone/s/submission-guidelines

- The Author declared in the cover letter being awarded a grant from the Wikimedia Foundation whose funding would cover this publication if accepted. Financial Disclosure section and Competing Interests section should be updated accordingly (in competing interests, I believe Author could specify having obtained the grant -after- completing and submitting this work)

==============================

We would appreciate receiving your revised manuscript by Mar 06 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

We look forward to receiving your revised manuscript.

Kind regards,

Stefano Triberti, Ph.D.

Academic Editor

PLOS ONE

PLoS One. 2020 Feb 18;15(2):e0228786. doi: 10.1371/journal.pone.0228786.r004

Author response to Decision Letter 1

22 Jan 2020

EDITOR: The Author properly responded to Reviewers' concerns. Minor modifications are still needed:

- abstract has been revised well for what regards content and structure, but now it exceeds the 300 words lenght allowed: https://journals.plos.org/plosone/s/submission-guidelines

DS: My mistake. I thought I had double checked this against the word limit. I have revised and reduced abstract to 295 words.

DS: During the resubmission process, I was not granted access to update the Financial Disclosure section. I emailed PLoS ONE and was advised to provide the update to my financial disclosure statement in my cover letter ONLY, which I did. Again, I am unable to access the Financial Disclosure section through the resubmission process. Instead, I have drafted the disclosure I would have included in this section, and have included it in a new cover letter.

I have added the statement suggested by the Editor (above) to the Competing Interests section of the paper.

PLoS One. doi: 10.1371/journal.pone.0228786.r005

Decision Letter 2

Stefano Triberti

24 Jan 2020

Situating Wikipedia as a health information resource in various contexts: A scoping review

PONE-D-19-29040R2

Dear Dr. Smith,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Stefano Triberti, Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

PLoS One. doi: 10.1371/journal.pone.0228786.r006

Acceptance letter

Stefano Triberti

30 Jan 2020

PONE-D-19-29040R2

Situating Wikipedia as a health information resource in various contexts: A scoping review

Dear Dr. Smith:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Stefano Triberti

Academic Editor

PLOS ONE

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

S1 Appendix. Search strategy and results.

(DOCX)

Click here for additional data file.^{(14.3KB, docx)}

S2 Appendix. General articles that aim to situate Wikipedia as a health information resource.

(DOCX)

Click here for additional data file.^{(13.8KB, docx)}

S3 Appendix. Studies of Wikipedia content quality categorized by population of concern.

(DOCX)

Click here for additional data file.^{(21.6KB, docx)}

S4 Appendix. Studies of Wikipedia’s utility in health education, categorized by education type.

(DOCX)

Click here for additional data file.^{(15.8KB, docx)}

S5 Appendix. Studies of Wikipedia’s utility in health-related research, categorized by field.

(DOCX)

Click here for additional data file.^{(19.5KB, docx)}

Attachment

Submitted filename: review PLOS Manuscript Number PONE-D-19-29040.pdf

Click here for additional data file.^{(87.7KB, pdf)}

Attachment

Submitted filename: Response to Reviewers.docx

Click here for additional data file.^{(50.5KB, docx)}

Data Availability Statement

All relevant data are within the manuscript in Appendices B through E, inclusive.

[pone.0228786.ref001] 1.Wiki Project Med—Meta [Internet]. [cited 2019 Sep 25]. Available from: https://meta.wikimedia.org/wiki/Wiki_Project_Med

[pone.0228786.ref002] 2.Heilman JM, West AG. Wikipedia and Medicine: Quantifying Readership, Editors, and the Significance of Natural Language. J Med Internet Res. 2015;17(3):e62 10.2196/jmir.4069 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref003] 3.Hodson R. Wikipedians reach out to academics. Nat News [Internet]. [cited 2019 Jul 29]; Available from: http://www.nature.com/news/wikipedians-reach-out-to-academics-1.18313 [Google Scholar]

[pone.0228786.ref004] 4.Giles J. Internet encyclopaedias go head to head [Internet]. Nature. 2005. [cited 2019 Jul 29]. Available from: https://www.nature.com/articles/438900a [DOI] [PubMed] [Google Scholar]

[pone.0228786.ref005] 5.Faric N, Potts HWW. Motivations for contributing to health-related articles on Wikipedia: an interview study. J Med Internet Res. 2014;16(12):e260 10.2196/jmir.3569 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref006] 6.Alexa—Top Sites in Canada—Alexa [Internet]. [cited 2019 Jul 29]. Available from: https://www.alexa.com/topsites/countries/CA

[pone.0228786.ref007] 7.Alexa—Top Sites in United States—Alexa [Internet]. [cited 2020 Jan 8]. Available from: https://www.alexa.com/topsites/countries/US

[pone.0228786.ref008] 8.Wikimedia Statistics—English Wikipedia—Unique devices [Internet]. [cited 2019 Jul 29]. Available from: https://stats.wikimedia.org/v2/#/en.wikipedia.org/reading/unique-devices/normal|line|1-year|~total|monthly

[pone.0228786.ref009] 9.Calabrese C, Anderton BN, Barnett GA. Online Representations of “Genome Editing” Uncover Opportunities for Encouraging Engagement: A Semantic Network Analysis. Sci Commun. 2019. April;41(2):222–42. [Google Scholar]

[pone.0228786.ref010] 10.Okoli C., Mehdi M, Mesgari M., Nielson F.A., Lanamäki A. Wikipedia in the eyes of its beholders: A systematic review of scholarly research on Wikipedia readers and readership. J Assoc Inf Sci Technol. 2014. December;(12):2381. [Google Scholar]

[pone.0228786.ref011] 11.Mesgari M, Okoli C, Mehdi M, Nielsen FÅ, Lanamäki A. “The sum of all human knowledge”: A systematic review of scholarly research on the content of Wikipedia. J Assoc Inf Sci Technol. 2015. February;66(2):219–45. [Google Scholar]

[pone.0228786.ref012] 12.Scoping studies: towards a methodological framework: International Journal of Social Research Methodology: Vol 8, No 1 [Internet]. [cited 2020 Jan 6]. Available from: https://www.tandfonline.com/doi/abs/10.1080/1364557032000119616

[pone.0228786.ref013] 13.Heilman JM, Kemmann E, Bonert M, Chatterjee A, Ragar B, Beards GM, et al. Wikipedia: a key tool for global public health promotion. J Med Internet Res. 2011;13(1):e14 10.2196/jmir.1589 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref014] 14.Shafee T, Masukume G, Kipersztok L, Das D, Haggstrom M, Heilman J. Evolution of Wikipedia’s medical content: past, present and future. J Epidemiol Community Health. 2017;71(11):1122–9. 10.1136/jech-2016-208601 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref015] 15.James R. WikiProject Medicine: Creating Credibility in Consumer Health. J Hosp Librariansh. 2016. October;16(4):344–51. [Google Scholar]

[pone.0228786.ref016] 16.Law MR, Mintzes B, Morgan SG. The Sources and Popularity of Online Drug Information: An Analysis of Top Search Engine Results and Web Page Views. Ann Pharmacother. 2011. March;45(3):350–6. 10.1345/aph.1P572 [DOI] [PubMed] [Google Scholar]

[pone.0228786.ref017] 17.Laurent MR, Vickers TJ. Seeking Health Information Online: Does Wikipedia Matter? J Am Med Inform Assoc. 2009. August;16(4):471–9. 10.1197/jamia.M3059 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref018] 18.Czarnecka-Kujawa K, Abdalian R, Grover SC. The quality of open access and open source internet material in gastroenterology: Is wikipedia appropriate for knowledge transfer to patients? GASTROENTEROLOGY. 2008. April;134(4, 1):A325–6. [Google Scholar]

[pone.0228786.ref019] 19.Thomas GR, Eng L, de Wolff JF, Grover SC. An Evaluation of Wikipedia as a Resource for Patient Education in Nephrology. Semin Dial. 2013. April;26(2):159–63. 10.1111/sdi.12059 [DOI] [PubMed] [Google Scholar]

[pone.0228786.ref020] 20.Rajagopalan MS, Khanna VK, Leiter Y, Stott M, Showalter TN, Dicker AP, et al. Patient-oriented cancer information on the internet: a comparison of wikipedia and a professionally maintained database. J Oncol Pract. 2011;7(5):319–23. 10.1200/JOP.2010.000209 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref021] 21.Leithner A, Maurer-Ertl W, Glehr M, Friesenbichler J, Leithner K, Windhager R. Wikipedia and osteosarcoma: a trustworthy patients’ information? J Am Med Inform Assoc. 2010. July;17(4):373–4. 10.1136/jamia.2010.004507 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref022] 22.Skancke KH, Svendsen K. Wikipedia and medicines: who edits medicine articles on the English Wikipedia? Int J Clin Pharm. 2017. February;39(1):254. [Google Scholar]

[pone.0228786.ref023] 23.Holtz P, Fetahu B, Kimmerle J. Effects of Contributor Experience on the Quality of Health-Related Wikipedia Articles. J Med Internet Res. 2018. May;20(5):e171 10.2196/jmir.9683 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref024] 24.Cozza V, Petrocchi M, Spognardi A. A Matter of Words: NLP for Quality Evaluation of Wikipedia Medical Articles In: Bozzon A and CudreMauroux P and Pautasso C, editor. WEB ENGINEERING (ICWE 2016). 2016. p. 448–56. (Lecture Notes in Computer Science; vol. 9671). [Google Scholar]

[pone.0228786.ref025] 25.Watad A, Bragazzi NL, Brigo F, Sharif K, Amital H, McGonagle D, et al. Readability of Wikipedia Pages on Autoimmune Disorders: Systematic Quantitative Assessment. J Med Internet Res. 2017. July;19(7):e260 10.2196/jmir.8225 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref026] 26.Clauson KA, Polen HH, Boulos MNK, Dzenowagis JH. Scope, completeness, and accuracy of drug information in Wikipedia. Ann Pharmacother. 2008;42(12):1814–21. 10.1345/aph.1L474 [DOI] [PubMed] [Google Scholar]

[pone.0228786.ref027] 27.Hunter JA, Lee T, Persaud N. A comparison of the content and primary literature support for online medication information provided by Lexicomp and Wikipedia. J Med Libr Assoc. 2018. July;106(3):352–60. 10.5195/jmla.2018.256 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref028] 28.Koppen L, Phillips J, Papageorgiou R. Analysis of reference sources used in drug-related Wikipedia articles. J Med Libr Assoc. 2015. July;103(3):140–4. 10.3163/1536-5050.103.3.007 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref029] 29.Lavsa S.M., Corman S.L., Culley C.M., Pummer T.L. Reliability of Wikipedia as a medication information source for pharmacy students. Curr Pharm Teach Learn. 2011;3(2):154–8. [Google Scholar]

[pone.0228786.ref030] 30.Phillips J, Lam C, Palmisano L. Analysis of the accuracy and readability of herbal supplement information on Wikipedia. J Am Pharm Assoc. 2014. August;(4):406. [DOI] [PubMed] [Google Scholar]

[pone.0228786.ref031] 31.Reilly T, Jackson W, Berger V, Candelario D. Accuracy and completeness of drug information in Wikipedia medication monographs. J Am Pharm Assoc. 2017. April;57(2):inform. [DOI] [PubMed] [Google Scholar]

[pone.0228786.ref032] 32.Kim JY, Gudewicz TM, Dighe AS, Gilbertson JR. The pathology informatics curriculum wiki: Harnessing the power of user-generated content. J Pathol Inform. 2010;1(101528849). [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref033] 33.Devgan L, Powe N, Blakey B, Makary M. Wiki-Surgery? Internal validity of Wikipedia as a medical and surgical reference. J Am Coll Surg. 2007. September;205(3, S):S76+.17916524 [Google Scholar]

[pone.0228786.ref034] 34.Modiri O, Guha D, Alotaibi NM, Ibrahim GM, Lipsman N, Fallah A. Readability and quality of wikipedia pages on neurosurgical topics. Clin Neurol Neurosurg. 2018;166(df4, 7502039):66–70. 10.1016/j.clineuro.2018.01.021 [DOI] [PubMed] [Google Scholar]

[pone.0228786.ref035] 35.Ayes KB, Bardsley CH. Wikipedia Information for Toxicologic Emergencies: How Reliable Is It? Clin Toxicol. 2010. July;48(6):635. [Google Scholar]

[pone.0228786.ref036] 36.Ayes K.B., Bardsley C.H., Jantz J.M., Frederick W.A. Wikipedia information for toxicologic emergencies involving household products, plants and envenomations: How reliable is it? Clin Toxicol. 2011;49(6):609. [Google Scholar]

[pone.0228786.ref037] 37.Messner M, DiStaso MW, Yan Jin, Meganck S, Sherman S, Norton S. Influencing public opinion from corn syrup to obesity: A longitudinal analysis of the references for nutritional entries on Wikipedia. First Monday. 2014. November;19(11):4–4. [Google Scholar]

[pone.0228786.ref038] 38.Sanz-Valero J., Cabrera-Hernandez L., Wanden-Berghe C., Culebras-Fernandez J.M. The popularization of food and nutritional sciences: Wikipedia versus a general encyclopedia. Clin Nutr. 2013;32(SUPPL. 1):S192. [Google Scholar]

[pone.0228786.ref039] 39.Sanz-Valero J., Wanden-Berghe C., Guardiola-Wanden-Berghe R. Nutrition and metabolism in wikipedia: Presence and adequacy of English and Spanish terminology. Clin Nutr Suppl. 2012;7(1):198–9. [Google Scholar]

[pone.0228786.ref040] 40.Koo M. Complementary and Alternative Medicine on Wikipedia: Opportunities for Improvement. Evid Based Complement Alternat Med. 2014;105186 10.1155/2014/105186 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref041] 41.Biggs TC, Jayakody N, Best K, King EV. Quality of online otolaryngology health information. J Laryngol Otol. 2018. June;132(6):560–3. 10.1017/S0022215118000774 [DOI] [PubMed] [Google Scholar]

[pone.0228786.ref042] 42.Qureishi A., Sharma A. Can Wikipedia replace traditional patient information leaflets? Comparing the internet to official patient information resources in thyroid cancer. Eur Surg—Acta Chir Austriaca. 2012;44(SUPPL. 247):35. [Google Scholar]

[pone.0228786.ref043] 43.Simpson A, Le M, Malicka AN. The Accuracy and Readability of Wikipedia Articles on Hearing Loss. J Consum Health Internet. 2018. October;22(4):323–36. [Google Scholar]

[pone.0228786.ref044] 44.Reavley NJ, Mackinnon AJ, Morgan AJ, Alvarez-Jimenez M, Hetrick SE, Killackey E, et al. Quality of information sources about mental disorders: a comparison of Wikipedia with centrally controlled web and printed sources. Psychol Med. 2012. Aug;42(8):1753–62. 10.1017/S003329171100287X [DOI] [PubMed] [Google Scholar]

[pone.0228786.ref045] 45.STANKUS T, SPIEGEL SE. Wikipedia, Scholarpedia, and References to Journals in the Brain and Behavioral Sciences: A Comparison of Cited Sources and Recommended Readings in Matching Free Online Encyclopedia Entries. Sci Technol Libr. 2010. July;29(3):258–65. [Google Scholar]

[pone.0228786.ref046] 46.Martin-Carreras T, Kahn CEJ. Integrating Wikipedia Articles and Images into an Information Resource for Radiology Patients. J Digit Imaging. 2019;32(3):349–53. 10.1007/s10278-018-0133-4 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref047] 47.Vydiswaran V.G., Mei Q., Hanauer D.A., Zheng K. Mining consumer health vocabulary from community-generated text. AMIA Annu Symp Proc AMIA Symp AMIA Symp. 2014;2014((Vydiswaran) School of Information, University of Michigan, Ann Arbor, MI):1150–9. [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref048] 48.Herbert VG, Frings A, Rehatschek H, Richard G, Leithner A. Wikipedia—challenges and new horizons in enhancing medical education. BMC Med Educ. 2015;15(101088679):32 10.1186/s12909-015-0309-2 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref049] 49.Murray H., Walker M., Maggio L., Dawson J. Wikipedia medical page editing as a platform to teach evidence-based medicine. BMJ Evid-Based Med. 2018;23(Supplement 1):A12–3. [Google Scholar]

[pone.0228786.ref050] 50.Azzam A., Bresler D., Leon A., Maggio L., Whitaker E., Heilman J., et al. Why Medical Schools Should Embrace Wikipedia: Final-Year Medical Student Contributions to Wikipedia Articles for Academic Credit at One School. Acad Med J Assoc Am Med Coll. 2017;92(2):194–200. [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref051] 51.Judd T, Kennedy G. Expediency-Based Practice? Medical Students’ Reliance on Google and Wikipedia for Biomedical Inquiries. Br J Educ Technol. 2011. March;42(2):351–60. [Google Scholar]

[pone.0228786.ref052] 52.Lieberthal RD, Leon J. Engaging Health Professionals in Health Economics: A Human Capital Informed Approach for Adults Learning Online. J Econ Educ. 2015. January 2;46(1):45–55. [Google Scholar]

[pone.0228786.ref053] 53.Schweitzer NJ. Wikipedia and psychology: Coverage of concepts and its use by undergraduate students. Teach Psychol. 2008. June;35(2):81–5. [Google Scholar]

[pone.0228786.ref054] 54.Azer SA. Evaluation of gastroenterology and hepatology articles on Wikipedia: Are they suitable as learning resources for medical students? Eur J Gastroenterol Hepatol. 2014. February;26(2):155–63. 10.1097/MEG.0000000000000003 [DOI] [PubMed] [Google Scholar]

[pone.0228786.ref055] 55.Azer SA, AlSwaidan NM, Alshwairikh LA, AlShammari JM. Accuracy and readability of cardiovascular entries on Wikipedia: are they reliable learning resources for medical students? Bmj Open. 2015;5(10):e008187 10.1136/bmjopen-2015-008187 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref056] 56.Antivalle M., Battellino M., Ditto M.C., Varisco V., Chevallard M., Rigamonti F., et al. Evaluation of wikipedia rheumatology articles as a learning resource for medical students. Ann Rheum Dis [Internet]. 2014;73(SUPPL. 2). Available from: http://heart.bmj.com/content/73/Suppl_2/801.3.abstract?sid=32ccb1a9-8868-4a08-a8c8-f62abf78e7c3 [Google Scholar]

[pone.0228786.ref057] 57.Haigh CA. Wikipedia as an evidence source for nursing and healthcare students. Nurse Educ Today. 2011. February;31(2):135–9. 10.1016/j.nedt.2010.05.004 [DOI] [PubMed] [Google Scholar]

[pone.0228786.ref058] 58.Yacob M., Jetty P., Lotfi S. The wikipedia medical student: Comparing the quality of vascular surgery topics across two commonly used educational resources. Int J Surg. 2014;12(SUPPL. 3):S73.25159547 [Google Scholar]

[pone.0228786.ref059] 59.Azer SA. Is Wikipedia a Reliable Learning Resource for Medical Students? Evaluating Respiratory Topics. Adv Physiol Educ. 2015. March;39(1):5–14. 10.1152/advan.00110.2014 [DOI] [PubMed] [Google Scholar]

[pone.0228786.ref060] 60.Wikipedia Zero—Wikimedia Foundation Governance Wiki [Internet]. [cited 2020 Jan 9]. Available from: https://foundation.wikimedia.org/wiki/Wikipedia_Zero

[pone.0228786.ref061] 61.Park E., Masupe T., Joseph J., Ho-Foster A., Chavez A., Jammalamadugu S., et al. Information needs of Botswana health care workers and perceptions of wikipedia. Int J Med Inf. 2016;95((Park) School of Medicine, Boston University, Boston, MA, United States):8–16. [DOI] [PubMed] [Google Scholar]

[pone.0228786.ref062] 62.Morata T., Lum M. Partnerships to expand occupational safety and health content in wikipedia. Occup Environ Med. 2018;75(Supplement 2):A71–2. [Google Scholar]

[pone.0228786.ref063] 63.Bould MD, Hladkowicz ES, Pigford A-AE, Ufholz L-A, Postonogova T, Shin E, et al. References that anyone can edit: review of Wikipedia citations in peer reviewed health science literature. Bmj-Br Med J. 2014. March 5;348:g1585. [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref064] 64.Nowrouzi B., Gohar B., Nowrouzi-Kia B., Garbaczewska M., Brewster K. An examination of scope, completeness, credibility, and readability of health, medical, and nutritional information on the internet: A comparative study of Wikipedia, WebMD, and the Mayo Clinic websites. Can J Diabetes. 2015;39(SUPPL. 1):S71. [Google Scholar]

[pone.0228786.ref065] 65.Apollonio D.E., Broyde K., Azzam A., De Guia M., Heilman J., Brock T. Pharmacy students can improve access to quality medicines information by editing Wikipedia articles. BMC Med Educ. 2018;18(1):265 10.1186/s12909-018-1375-z [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref066] 66.Burdo JR. Wikipedia neuroscience stub editing in an introductory undergraduate neuroscience course. J Undergrad Neurosci Educ JUNE Publ FUN Fac Undergrad Neurosci. 2012;11(1):A1–5. [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref067] 67.Infeld DL, Adams WC. Using the Internet for Gerontology Education: Assessing and Improving Wikipedia. Educ Gerontol. 2013. October;39(10):707–16. [Google Scholar]

[pone.0228786.ref068] 68.Shane-Simpson C, Che E, Brooks PJ. Giving Psychology Away: Implementation of Wikipedia Editing in an Introductory Human Development Course. Psychol Learn Teach-PLAT. 2016. November;15(3, SI):268–93. [Google Scholar]

[pone.0228786.ref069] 69.Walker M., Murray H., Dawson J., Maggio L. Wikipedia culture and usage: A survey of first year medical students to determine barriers and facilitators. BMJ Evid-Based Med. 2018;23(Supplement 1):A25. [Google Scholar]

[pone.0228786.ref070] 70.Huss JW 3rd, Lindenbaum P, Martone M, Roberts D, Pizarro A, Valafar F, et al. The Gene Wiki: community intelligence applied to human gene annotation. Nucleic Acids Res. 2010;38(Database issue):D633–9. 10.1093/nar/gkp760 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref071] 71.Good B.M., Clarke E.L., Loguercio S., Su A.I. Building a biomedical semantic network in Wikipedia with Semantic Wiki Links. Database J Biol Databases Curation. 2012;2012((Good) The Scripps Research Institute, La Jolla, CA 92037, USA.). [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref072] 72.Good BM, Howe DG, Lin SM, Kibbe WA, Su AI. Mining the Gene Wiki for functional genomic knowledge. BMC Genomics. 2011;12(100965258):603. [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref073] 73.Rossler B., Holldack H., Schebesta K. Influence of wikipedia and other web resources on acute and critical care decisions. A web-based survey. Intensive Care Med Exp. 2015;3(Supplement 1):A867. [Google Scholar]

[pone.0228786.ref074] 74.Scaffidi MA, Khan R, Wang C, Keren D, Tsui C, Garg A, et al. Comparison of the Impact of Wikipedia, UpToDate, and a Digital Textbook on Short-Term Knowledge Acquisition Among Medical Students: Randomized Controlled Trial of Three Web-Based Resources. JMIR Med Educ. 2017;(2):e20 10.2196/mededu.8188 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref075] 75.Brigo F, Erro R. Why do people google movement disorders? An infodemiological study of information seeking behaviors. Neurol Sci. 2016. May;37(5):781–7. 10.1007/s10072-016-2501-5 [DOI] [PubMed] [Google Scholar]

[pone.0228786.ref076] 76.Brigo F., Igwe S.C., Otte W.M., Tezzon F., Lochner P. Information seeking behavior for neurological disorders: An infodemiological study of wikipedia article traffic. Beghi E., Filippini G., editors. Neuroepidemiology. 2014;43(3–4):158. [Google Scholar]

[pone.0228786.ref077] 77.Brigo F, Igwe SC, Nardone R, Lochner P, Tezzon F, Otte WM. Wikipedia and neurological disorders. J Clin Neurosci. 2015. July;22(7):1170–2. 10.1016/j.jocn.2015.02.006 [DOI] [PubMed] [Google Scholar]

[pone.0228786.ref078] 78.Kint M., Batty P., Peters M., Hart D. Wikipedia dominance in simulated online searching for hemophiliaspecific patient information. Haemophilia. 2014;20(SUPPL. 3):178. [Google Scholar]

[pone.0228786.ref079] 79.Evans P, Krauthammer M. Exploring the use of social media to measure journal article impact. AMIA Annu Symp Proc AMIA Symp. 2011;2011(101209213):374–81. [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref080] 80.Jemielniak D., Masukume G., Wilamowski M. The Most Influential Medical Journals According to Wikipedia: Quantitative Analysis. J Med Internet Res. 2019;21(1):e11429 10.2196/11429 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref081] 81.Kissner M, Troia L, Bexfield A, Mahbuba R, Coover R. Evaluating the relationship between publications and Wikipedia citations for a cohort of oral anticoagulants. Curr Med Res Opin. 2019. April 11;35(2, SI):15. [Google Scholar]

[pone.0228786.ref082] 82.Maggio LA, Willinsky JM, Steinberg RM, Mietchen D, Wass JL, Dong T. Wikipedia as a gateway to biomedical research: The relative distribution and use of citations in the English Wikipedia. PloS One. 2017;12(12):e0190046 10.1371/journal.pone.0190046 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref083] 83.Generous N, Fairchild G, Deshpande A, Del Valle SY, Priedhorsky R. Global Disease Monitoring and Forecasting with Wikipedia. Plos Comput Biol. 2014. November;10(11):e1003892 10.1371/journal.pcbi.1003892 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref084] 84.Dzogang F, Lansdall-Welfare T, Cristianini N. Seasonal Fluctuations in Collective Mood Revealed by Wikipedia Searches and Twitter Posts. In: Domeniconi C, Gullo F, Bonchi F, DomingoFerrer J, BaezaYates R, Zhou ZH, et al., editors. 2016 Ieee 16th International Conference on Data Mining Workshops (icdmw). 2016. p. 931–7.

[pone.0228786.ref085] 85.McIver DJ, Brownstein JS. Wikipedia Usage Estimates Prevalence of Influenza-Like Illness in the United States in Near Real-Time. Plos Comput Biol. 2014. April;10(4):e1003581 10.1371/journal.pcbi.1003581 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref086] 86.Tausczik Y, Faasse K, Pennebaker JW, Petrie KJ. Public Anxiety and Information Seeking Following the H1N1 Outbreak: Blogs, Newspaper Articles, and Wikipedia Visits. Health Commun. 2012;27(2):179–85. 10.1080/10410236.2011.571759 [DOI] [PubMed] [Google Scholar]

[pone.0228786.ref087] 87.Zimmer C., Leuba S.I., Yaesoubi R., Cohen T. Use of daily Internet search query data improves real-time projections of influenza epidemics. J R Soc Interface. 2018;15(147):20180220 10.1098/rsif.2018.0220 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref088] 88.Priedhorsky R, Osthus D, Daughton AR, Moran KR, Generous N, Fairchild G, et al. Measuring Global Disease with Wikipedia: Success, Failure, and a Research Agenda. CSCW Proc Conf Comput-Support Coop Work Conf Comput-Support Coop Work. 2017;2017(101604142):1812–34. [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref089] 89.Sharpe JD, Hopkins RS, Cook RL, Striley CW. Evaluating Google, Twitter, and Wikipedia as Tools for Influenza Surveillance Using Bayesian Change Point Analysis: A Comparative Analysis. JMIR Public Health Surveill. 2016;2(2):e161 10.2196/publichealth.5901 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref090] 90.Qiu R, Hadzikadic M, Yu S, Yao L. Estimating disease burden using Internet data. Health Informatics J. 2018;(100883604):1460458218810743. [DOI] [PubMed] [Google Scholar]

[pone.0228786.ref091] 91.Piwowar H, Priem J, Larivière V, Alperin JP, Matthias L, Norlander B, et al. The state of OA: a large-scale analysis of the prevalence and impact of Open Access articles. PeerJ [Internet]. 2018. February 13 [cited 2019 Jul 5];6. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5815332/ [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref092] 92.Hillard J.R., Laird-Fick H.S., Chennamaneni V., Kavuturu S., Nassiri R.M. Helicobacter and wikipedia. Helicobacter. 2015;20(Supplement 1):115. [Google Scholar]

[pone.0228786.ref093] 93.Altmann U. Representation of medical informatics in the wikipedia and its perspectives. Stud Health Technol Inform. 2005;116(ck1, 9214582):755–60. [PubMed] [Google Scholar]

[pone.0228786.ref094] 94.Hasty RT, Garbalosa RC, Barbato VA, Valdes PJJ, Powers DW, Hernandez E, et al. Wikipedia vs peer-reviewed medical literature for information about the 10 most costly medical conditions. J Am Osteopath Assoc. 2014;114(5):368–73. 10.7556/jaoa.2014.035 [DOI] [PubMed] [Google Scholar]

[pone.0228786.ref095] 95.Temple NJ, Fraser J. How Accurate Are Wikipedia Articles in Health, Nutrition, and Medicine? Can J Inf Libr Sci-Rev Can Sci Inf Bibl. 2014. March;38(1):37–52. [Google Scholar]

[pone.0228786.ref096] 96.Pfundner A, Schönberg T, Horn J, Boyce RD, Samwald M. Utilizing the Wikidata System to Improve the Quality of Medical Content in Wikipedia in Diverse Languages: A Pilot Study. J Med Internet Res. 2015;17(5):e110 10.2196/jmir.4163 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref097] 97.Weiner S.S., Horbacewicz J., Rasberry L., Bensinger-Brody Y. Improving the Quality of Consumer Health Information on Wikipedia: Case Series. J Med Internet Res. 2019;21(3):e12450 10.2196/12450 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0228786.ref098] 98.Zhang Y., Lin Y.-H. Writing a Wikipedia Article on Cultural Competence in Health Care. Med Ref Serv Q. 2016;35(2):175–86. 10.1080/02763869.2016.1152143 [DOI] [PubMed] [Google Scholar]

[pone.0228786.ref099] 99.Bawden D. Users, user studies and human information behaviour: A three‐decade perspective on Tom Wilson’s “On user studies and information needs.” J Doc. 2006. January 1;62(6):671–9. [Google Scholar]

PERMALINK

Situating Wikipedia as a health information resource in various contexts: A scoping review

Denise A Smith

Roles

Abstract

Background

Objective

Methods

Results

Conclusion

Introduction

Methods

Search strategy

Study selection

Fig 1. PRISMA flow diagram of studies selected for inclusion.

Thematic analysis

Results

Situating Wikipedia as a health information resource

Fig 2. Distribution of relevant studies by category.

Quality of Wikipedia’s health content

For consumers

For students

For professionals

General assessments of quality

Utility in education

Utility in research

Bibliometric and information-seeking behaviour studies

Epidemiological studies

Publication trends

Fig 3. Distribution of publication models between CrossRef and the health literature on Wikipedia.

Fig 4. Distribution of literature by publication year.

Discussion

Limitations

Conclusion

Supporting information

Abbreviations

Data Availability

Funding Statement

References

Decision Letter 0

Stefano Triberti

Roles

Author response to Decision Letter 0

Decision Letter 1

Stefano Triberti

Roles

Author response to Decision Letter 1

Decision Letter 2

Stefano Triberti

Roles

Acceptance letter

Stefano Triberti

Roles

Associated Data

Supplementary Materials

Data Availability Statement

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases